-
-
Notifications
You must be signed in to change notification settings - Fork 289
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use implicit fill values for zarr v2 #2274
Use implicit fill values for zarr v2 #2274
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few comments. Seems reasonable though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good. My one suggestion would be to consider a special case for object / string dtypes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome, nice work Tom!
out[out_selection] = chunk_spec.fill_value | ||
fill_value = chunk_spec.fill_value | ||
|
||
if fill_value is None: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Something we may consider in the future is to parameterize the CodecPipeline
class with the zarr_format
of the calling Array. This would allow us feel more confident that workarounds like this one only apply to v2 data.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @TomAugspurger!
Closes #2271
I'm not sure if this is a good idea, but this, consolidated metadata, and #2270 lets xarray round trip a dataset. The basic idea is to let
ArrayV2Metadata
represent the v2 metadata a bit more faithfully. Currently, we "eagerly" interpret the lack offill_value
in a V2 document as the default fill value for some dtype (e.g.0
for int64). This PR changes that, to allowArrayV2Metadata.fill_value
to be None if it's unspecified in the metadata. But we do need some fill value when reading data. So that's where we finally substitute in the default.This doesn't really affect Zarr directly, but it is useful for downstream libraries that want to inspect the array metadata before reading and make some decision based on whether or not
fill_value
is specified.TODO: