Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache pre-existing Zarr arrays in Zarr backend #9861

Merged
merged 48 commits into from
Dec 18, 2024
Merged
Changes from 1 commit
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
9503c71
cache array keys on ZarrStore instances
d-v-b Dec 4, 2024
94b48ac
refactor get_array_keys
d-v-b Dec 6, 2024
2dacb2a
add test for get_array_keys
d-v-b Dec 6, 2024
c7cfacf
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 6, 2024
b3cf2e5
lint
d-v-b Dec 6, 2024
d9bd7fd
Merge branch 'perf/cache-array-keys' of https://github.com/d-v-b/xarr…
d-v-b Dec 6, 2024
8d47dd3
Merge branch 'main' of https://www.github.com/pydata/xarray into perf…
d-v-b Dec 6, 2024
bec8b42
add type annotation for _cache
d-v-b Dec 6, 2024
61fe759
make type hint accurate
d-v-b Dec 6, 2024
0952c93
make type hint pass mypy
d-v-b Dec 6, 2024
96ddcf4
make type hint pass mypy, correctly
d-v-b Dec 6, 2024
60761a8
Update xarray/backends/zarr.py
d-v-b Dec 9, 2024
f82659d
Merge branch 'main' of https://github.com/pydata/xarray into perf/cac…
d-v-b Dec 10, 2024
107c4e7
use roundtrip method instead of explicit alternative
d-v-b Dec 10, 2024
2d3c13a
cache members instead of just array keys
d-v-b Dec 10, 2024
be5e389
adjust test to members caching
d-v-b Dec 10, 2024
1eaf3ea
Merge branch 'perf/cache-array-keys' of https://github.com/d-v-b/xarr…
d-v-b Dec 10, 2024
9d147b9
refactor members cache
d-v-b Dec 10, 2024
bdb20a6
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 10, 2024
6578caa
explictly revert changes to test_write_store
d-v-b Dec 10, 2024
9596bcf
Merge branch 'perf/cache-array-keys' of https://github.com/d-v-b/xarr…
d-v-b Dec 10, 2024
df4274f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 10, 2024
5bf7aff
indent correctly
d-v-b Dec 10, 2024
df51855
Merge branch 'perf/cache-array-keys' of https://github.com/d-v-b/xarr…
d-v-b Dec 10, 2024
5a818a5
update instrumented tests, remove cache update functionality, and set…
d-v-b Dec 10, 2024
c18f81b
update instrumented tests, remove cache update functionality, and set…
d-v-b Dec 10, 2024
7b13099
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 10, 2024
8925020
lint
d-v-b Dec 10, 2024
5efbe30
Merge branch 'perf/cache-array-keys' of https://github.com/d-v-b/xarr…
d-v-b Dec 10, 2024
192fc8b
update tests
d-v-b Dec 10, 2024
2ce5b2e
make check_requests more permissive
d-v-b Dec 10, 2024
87a40c7
update instrumented tests for zarr==2.18
d-v-b Dec 10, 2024
7e79b79
Merge branch 'main' into perf/cache-array-keys
dcherian Dec 11, 2024
d987529
Update xarray/backends/zarr.py
d-v-b Dec 14, 2024
169b736
Update xarray/backends/zarr.py
d-v-b Dec 14, 2024
e20d11d
Update xarray/backends/zarr.py
d-v-b Dec 14, 2024
a1c25f2
Update xarray/backends/zarr.py
d-v-b Dec 14, 2024
f8d78cb
Update xarray/backends/zarr.py
d-v-b Dec 16, 2024
650937e
doc: add whats new content
d-v-b Dec 17, 2024
fcf2d64
fixup
d-v-b Dec 17, 2024
bf64fd2
Merge branch 'main' of https://www.github.com/pydata/xarray into perf…
d-v-b Dec 17, 2024
dcecf43
update whats new
d-v-b Dec 17, 2024
d0bb5a2
fixup
d-v-b Dec 17, 2024
65cadcb
remove vestigial cache_array_keys
d-v-b Dec 17, 2024
e8cd3c8
make cache_members default to True
d-v-b Dec 17, 2024
8d31f0e
tweak
dcherian Dec 17, 2024
3d15d3d
Remove from public API
dcherian Dec 17, 2024
f1ef905
Merge branch 'main' into perf/cache-array-keys
dcherian Dec 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
remove vestigial cache_array_keys
d-v-b committed Dec 17, 2024
commit 65cadcba8f3ffd67c349c1b4bab913745749a8ea
4 changes: 2 additions & 2 deletions xarray/backends/zarr.py
Original file line number Diff line number Diff line change
@@ -1560,7 +1560,7 @@ def open_dataset(
store=None,
engine=None,
use_zarr_fill_value_as_mask=None,
cache_array_keys: bool = True,
cache_members: bool = True,
) -> Dataset:
filename_or_obj = _normalize_path(filename_or_obj)
if not store:
@@ -1576,7 +1576,7 @@ def open_dataset(
zarr_version=zarr_version,
use_zarr_fill_value_as_mask=None,
zarr_format=zarr_format,
cache_members=cache_array_keys,
cache_members=cache_members,
)

store_entrypoint = StoreBackendEntrypoint()