Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/concurrent members #2519

Open
wants to merge 34 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
8407c64
feat: add wrapperstore
d-v-b Nov 8, 2024
5e9ffb8
feat: add latencystore
d-v-b Nov 8, 2024
5d7abf4
rename noisysetter -> noisygetter
d-v-b Nov 8, 2024
c486351
rename _wrapped to _store
d-v-b Nov 8, 2024
f97b27c
loggingstore inherits from wrapperstore
d-v-b Nov 8, 2024
ffca710
initial commit
d-v-b Nov 8, 2024
d33cb7d
working members traversal
d-v-b Nov 8, 2024
5ba51af
Merge branch 'main' into feat/latency-store
d-v-b Nov 12, 2024
8f87977
bolt concurrent members implementation onto async group
d-v-b Nov 18, 2024
87e0b83
update scratch file
d-v-b Nov 18, 2024
502ad5e
use metadata / node builders for v3 node creation
d-v-b Nov 20, 2024
53c8738
Merge branch 'main' of github.com:zarr-developers/zarr-python into fe…
d-v-b Nov 26, 2024
70a4ff5
Merge branch 'feat/latency-store' of github.com:d-v-b/zarr-python int…
d-v-b Nov 26, 2024
d10d805
fix key/name handling in recursion
d-v-b Nov 26, 2024
4c624e1
add latency-based test
d-v-b Nov 26, 2024
f23ee85
add latency-based concurrency tests for group.members
d-v-b Nov 26, 2024
cba42f3
improve comments for test
d-v-b Nov 26, 2024
9691102
add concurrency limit
d-v-b Nov 26, 2024
d790379
add test for concurrency limiting
d-v-b Nov 26, 2024
aadbece
docstrings
d-v-b Nov 26, 2024
1156859
Merge branch 'main' of github.com:zarr-developers/zarr-python into fe…
d-v-b Nov 26, 2024
e19238d
remove function that was only calling itself
d-v-b Nov 26, 2024
db74205
docstrings
d-v-b Nov 26, 2024
46ba0cb
relax timing requirement for concurrency test
d-v-b Nov 27, 2024
180ca9b
Merge branch 'main' into feat/concurrent-members
d-v-b Dec 11, 2024
a48efa8
Update src/zarr/core/group.py
d-v-b Dec 12, 2024
e53f847
Merge branch 'main' into feat/concurrent-members
d-v-b Dec 12, 2024
824ff03
Merge branch 'main' of github.com:zarr-developers/zarr-python into fe…
d-v-b Dec 18, 2024
9a17dfd
Merge branch 'feat/concurrent-members' of github.com:d-v-b/zarr-pytho…
d-v-b Dec 18, 2024
ebfc200
exists_ok -> overwrite
d-v-b Dec 18, 2024
39ec6b5
simplify group_members_perf test, just require that the duration is l…
d-v-b Dec 18, 2024
f25bda3
update test docstring
d-v-b Dec 18, 2024
3f8de76
Merge branch 'main' into feat/concurrent-members
d-v-b Dec 19, 2024
d920a9d
Merge branch 'main' of https://github.com/zarr-developers/zarr-python…
d-v-b Jan 1, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions src/zarr/api/asynchronous.py
Original file line number Diff line number Diff line change
Expand Up @@ -187,14 +187,12 @@ async def consolidate_metadata(
group.store_path.store._check_writable()

members_metadata = {k: v.metadata async for k, v in group.members(max_depth=None)}

# While consolidating, we want to be explicit about when child groups
# are empty by inserting an empty dict for consolidated_metadata.metadata
for k, v in members_metadata.items():
if isinstance(v, GroupMetadata) and v.consolidated_metadata is None:
v = dataclasses.replace(v, consolidated_metadata=ConsolidatedMetadata(metadata={}))
members_metadata[k] = v

ConsolidatedMetadata._flat_to_nested(members_metadata)

consolidated_metadata = ConsolidatedMetadata(metadata=members_metadata)
Expand Down
3 changes: 2 additions & 1 deletion src/zarr/core/array.py
Original file line number Diff line number Diff line change
Expand Up @@ -1660,10 +1660,11 @@ def path(self) -> str:

@property
def name(self) -> str:
"""Array name following h5py convention."""
return self._async_array.name

@property
def basename(self) -> str | None:
def basename(self) -> str:
"""Final component of name."""
return self._async_array.basename

Expand Down
Loading
Loading