Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove requests requirement, and use urllib built-in modules #34

Merged
merged 1 commit into from
Sep 21, 2021

Conversation

mwtoews
Copy link
Contributor

@mwtoews mwtoews commented Sep 21, 2021

Hi Mike, I'm experiencing issues with the web_service module, e.g.

from hilltoppy import web_service as ws

base_url = 'https://data.hbrc.govt.nz/Envirodata'
hts = 'ContinuousArchive.hts'
site = 'Well.16772 Ngatarawa Rd'
measurement = 'Elevation Above Sea Level[Recorder Water Level]'
ts_df = ws.get_data(base_url, hts, site, measurement)
ts = ts_df.reset_index().set_index("DateTime")["Value"]

raises "ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))".

This error is the same on Windows and Linux using recentish packages from conda-forge.

Is this #29 and/or #30?

A fix (for me at least) is to simply replace Python Requests with the urllib built-in modules. It doesn't seem that the high-level features of Requests is warranted by this module, as it's pretty basic encode and retrieve data from a stream.

I've mostly rewritten build_url to build a dict of data that gets encoded once. The order of the dict should be preserved, even for Python 3.6, since most folks use CPython. (The insertion order is guaranteed from Python 3.7). The previous build_url encoded parts, but not all. This will send perhaps different URLs. So, check carefully with your use-cases. Also interested to know how this change behaves with large data requests. I've only been using this module since yesterday, so I'm actually a novice with this package.

@mullenkamp
Copy link
Owner

Thanks for that. I do wonder what's going wrong with requests...
Let's give it a try, and I'll definitely test the larger datasets.
I've been meaning to rewrite some of the code in the build_url function. I wrote that code fairly early on in my Python coding career ;)

@mullenkamp mullenkamp merged commit cf8723c into mullenkamp:master Sep 21, 2021
@mwtoews mwtoews deleted the rm-requests branch September 21, 2021 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants