You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Aug 29, 2023. It is now read-only.
After a few first tests, I am very enthusiastic about the Zarr, a new data format optimized for distributed and concurrent array I/O. It seems to offer much better I/O performance over NetCDF4, which maybe due to single-threaded HDF5 decompression in Python (not checked).
As it seems to be a 1:1 representation of the NetCDF4 / HDF5 data model, Cate could use it for very efficient workspaces persistence or users could use it for intermediate computation results.
The good news is, that Cate doesn't require any extra dependencies as the zarr package is already a dependency of xarray 0.10.
The text was updated successfully, but these errors were encountered:
And it is lightning fast. Just ingested a Zarr data cube with dims=(time=250, lat=1000, lon=2000) (for another project) and when I time-travel through it, layers are displayed immediately.
xarray 0.10 introduced two new methods, xr.open_zarr() and Dataset.to_zarr.
After a few first tests, I am very enthusiastic about the Zarr, a new data format optimized for distributed and concurrent array I/O. It seems to offer much better I/O performance over NetCDF4, which maybe due to single-threaded HDF5 decompression in Python (not checked).
As it seems to be a 1:1 representation of the NetCDF4 / HDF5 data model, Cate could use it for very efficient workspaces persistence or users could use it for intermediate computation results.
The good news is, that Cate doesn't require any extra dependencies as the zarr package is already a dependency of xarray 0.10.
The text was updated successfully, but these errors were encountered: