fix httpfile serialization bug #973
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I think this PR is fine, and solves one arm of he possibilities in #579 . Another possible thing to consider, is making OpenFile file-like, calling into |
rabernat
commented
Jun 1, 2022
| @@ -350,7 +350,7 @@ def __getattr__(self, item): | |||
|
|
|||
| def __enter__(self): | |||
| self._incontext = True | |||
| return self.f.__enter__() | |||
| return self | |||
There was a problem hiding this comment.
Am I correct in assuming that this will solve
open_file = fsspec.open(local_path) -> <OpenFile '/local/path.nc'>
❌ doesn't work with context manager (TypeError: cannot pickle '_io.BufferedReader' object)
from #579 (comment)
There was a problem hiding this comment.
This solves the
with fsspec.open(localpath) as f:
case. I think that's what you said.
There was a problem hiding this comment.
Update: just re-ran my test and the answer is YES! ✅
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This implements the suggestion in pangeo-forge/pangeo-forge-recipes#370 (comment), which fixes a bug in how
HTTPFileis serialized.It's a bit worrying to me that serialization of
HTTPFileis not covered by tests sufficiently to have already surfaced this bug. Therefore, I would like to use this PR to contribute some improvements to testing as well.