Non-kerchunk backend for HDF5/netcdf4 files. #87
Conversation
There was a problem hiding this comment.
This is looking great so far @sharkinsspatial !
kerchunk backend's specialized encoding translation logic
This part I would really like to either factor out, or at a least really understand what it's doing. See #68
virtualizarr/readers/hdf.py
Outdated
| @@ -0,0 +1,206 @@ | |||
| from typing import List, Mapping, Optional | |||
|
|
|||
| import fsspec | |||
There was a problem hiding this comment.
Does one need fsspec if reading a local file? Is there any other way to read from S3 without fsspec at all?
There was a problem hiding this comment.
Not with a filesystem-like API. You would have to use boto3 or aiobotocore directly.
This is one of the great virtues of fsspec and is not to be under-valued.
virtualizarr/readers/hdf.py
Outdated
| def virtual_vars_from_hdf( | ||
| path: str, | ||
| drop_variables: Optional[List[str]] = None, | ||
| ) -> Mapping[str, xr.Variable]: |
There was a problem hiding this comment.
I like this an a way to interface with the code in open_virtual_dataset
|
This looks cool @sharkinsspatial! My opinion is that it doesn't make sense to just forklift the kerchunk code into virtualizarr. What I would love to see is an extremely tight, strictly typed, unit-tested total refactor of the parsing logic. I think you're headed down the right path, but I encourage you to push as far as you can in that direction. |
TomNicholas
added
enhancement
for more information, see https://pre-commit.ci
|
@rabernat Fully agree with your take above 👆 👍 . I'm trying to work through this incrementally whenever I can find some spare time. In the spirit of thorough test coverage 🎊 looking through your issue pydata/xarray#7388 and the corresponding PR I'm not sure what the proper incantation of variable encoding configuration is to use |
for more information, see https://pre-commit.ci
TomNicholas
had a problem deploying
to
test-release
GitHub Actions
Failure
sharkinsspatial
force-pushed
the
hdf5_reader
branch
from
13f82f9 to
ee6fa0b
Compare
sharkinsspatial
had a problem deploying
to
test-release
GitHub Actions
Failure
sharkinsspatial
force-pushed
the
hdf5_reader
branch
2 times, most recently
from
b2c89df to
06a5ae1
Compare
sharkinsspatial
had a problem deploying
to
test-release
GitHub Actions
Failure
sharkinsspatial
force-pushed
the
hdf5_reader
branch
from
06a5ae1 to
a8cc82f
Compare
sharkinsspatial
had a problem deploying
to
test-release
GitHub Actions
Failure
sharkinsspatial
had a problem deploying
to
test-release
GitHub Actions
Failure
sharkinsspatial
force-pushed
the
hdf5_reader
branch
from
13c51a4 to
a1c1ff1
Compare
sharkinsspatial
had a problem deploying
to
test-release
GitHub Actions
Failure
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. Thanks for integrating Codecov - We've got you covered ☂️ |
sharkinsspatial
force-pushed
the
hdf5_reader
branch
from
a1c1ff1 to
65a6b14
Compare
sharkinsspatial
temporarily deployed
to
test-release
GitHub Actions
Inactive
|
@TomNicholas This should be ready for final review now. |
sharkinsspatial
changed the title
| import hdf5plugin # type: ignore | ||
| except ModuleNotFoundError: | ||
| hdf5plugin = None # type: ignore | ||
| warnings.warn("hdf5plugin is required for HDF reader") |
There was a problem hiding this comment.
Why wouldn't this either error or just not be a problem?
There was a problem hiding this comment.
I added a release note in #307, but I'm still curious about this @sharkinsspatial
This is a rudimentary initial implementation for #78. The core code is ported directly from kerchunk's hdf backend. I have not ported the bulk of the kerchunk backend's specialized encoding translation logic but I'll try to do so incrementally so that we can build complete test coverage for the many edge cases it currently covers.