-
-
Notifications
You must be signed in to change notification settings - Fork 290
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[v3] Re-opening an existing RemoteStore V3 group causes the first child to be deleted #2359
Comments
I understand what is going on here. I believe this is the intended behavior for our new stores but am open to the idea that we should change it! In Zarr-Python 2, stores were just a mapping interface on top of a storage backend. No access mode was applied at the store level - except for stores that required an access mode ( In Zarr-Python 3, we moved the access mode handling to the store so that we could have a single place where we could enforce access. The simplest way to think of this is that the store is kind of like a file object, opening it in In your example, opened the store in Questions:
Edit: here's a short example using a memory store with what I would consider expected behavior. Note that the only place where data = {}
store = zarr.storage.MemoryStore(data, mode='w')
# create a group with two children
group = zarr.open_group(store=store, zarr_format=3)
print(group.members())
# ()
group.create("A", shape=1)
print(group.members())
# (('A', <Array memory://5393639360/A shape=(1,) dtype=float64>),)
group.create("B", shape=1)
print(group.members())
# (('B', <Array memory://5393639360/B shape=(1,) dtype=float64>), ('A', <Array memory://5393639360/A shape=(1,) dtype=float64>))
# re-open the group in a mode
store2 = zarr.storage.MemoryStore(data, mode='a')
group = zarr.open_group(store=store2, zarr_format=3)
print(group.members())
(('B', <Array memory://5393696256/B shape=(1,) dtype=float64>), ('A', <Array memory://5393696256/A shape=(1,) dtype=float64>))
# re-open the group in w mode
store2 = zarr.storage.MemoryStore(data, mode='w')
group = zarr.open_group(store=store2, zarr_format=3)
print(group.members())
# () |
I think this is a really bad default behavior and will lead to catastrophic scenarios of data loss. Simply opening a store in 'w' mode should absolutely not delete the entire contents of the store. IMO, simply opening or initializing a store should have no side effects regardless of the mode, other than perhaps checking for the existence of the store. On the other hand, the various Array and Group open / create APIs also have a |
I think this is right. @jhamman could you restate (or link to) the original reasoning for moving the |
I don't want to come across like I have a particular stake in the outcome here, but I'll try to describe the argument for how things are today. We can definitely change the semantics and/or order of operations for when actions occure. Zarr-Python 2
Zarr-Python 3
Some additional notes:
|
I spent some time over the weekend thinking through this situation and I've come up with a new proposal.
@zarr-developers/python-core-devs - what do you think about this plan? |
I like it. Especially that the operations are local to the prefix being updated, and are delayed until the operation is invoked. |
I like having less modes and side effects! |
This is a weird and unexpected behavior
Array
A
has disappeared! That's bad!This might have something to do with #2338. Really not sure. I haven't gotten to the bottom of it, but I thought I would post a report anyway.
The text was updated successfully, but these errors were encountered: