Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support chunking of returned datasets #42

Open
pont-us opened this issue Feb 24, 2021 · 1 comment
Open

Support chunking of returned datasets #42

pont-us opened this issue Feb 24, 2021 · 1 comment
Labels
enhancement New feature or request

Comments

@pont-us
Copy link
Member

pont-us commented Feb 24, 2021

Currently, the CDS Store does not specify any chunking when opening downloaded NetCDF files. This means that the entire datasets are always loaded into memory (see open_dataset and open_mfdataset documentation). The store should provide some option to chunk datasets. Mirroring NetCDF chunking would be a good start, but see caveats in this discussion.

@forman
Copy link
Member

forman commented Feb 24, 2021

For implementation refer to https://github.com/bcdev/nc2zarr/blob/main/nc2zarr/opener.py#L91, method _prefetch_chunk_sizes().

@pont-us pont-us added the enhancement New feature or request label Sep 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants