-
-
Notifications
You must be signed in to change notification settings - Fork 290
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
object-store
-based Store implementation
#1661
base: main
Are you sure you want to change the base?
Conversation
Amazing @kylebarron! I'll spend some time playing with this today. |
With roeap/object-store-python#9 it should be possible to fetch multiple ranges within a file concurrently with range coalescing (using That PR also adds a |
src/zarr/v3/store/object_store.py
Outdated
async def get_partial_values( | ||
self, key_ranges: List[Tuple[str, Tuple[int, int]]] | ||
) -> List[bytes]: | ||
# TODO: use rust-based concurrency inside object-store |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How I did it in rfsspec: https://github.com/martindurant/rfsspec/blob/main/src/lib.rs#L141
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
object-store has a built-in function for this: get_ranges
. With the caveat that it only manages multiple ranges in a single file.
get_ranges also automatically handles request merging for nearby ranges in a file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes I know, but mine already did the whole thing, so I am showing how I did that.
Great work @kylebarron! |
I suggest we see whether it makes any improvements first, so it's author's choice for now. |
While @rabernat has seen some impressive perf improvements in some settings when making many requests with Rust's tokio runtime, which would possibly also trickle down to a Python binding, the biggest advantage I see is improved ease of use in installation. A common hurdle I've seen is handling dependency management, especially around boto3, aioboto3, etc dependencies. Versions need to be compatible at runtime with any other libraries the user also has in their environment. And Python doesn't allow multiple versions of the same dependency at the same time in one environment. With a Python library wrapping a statically-linked Rust binary, you can remove all Python dependencies and remove this class of hardship. The underlying Rust object-store crate is stable and under open governance via the Apache Arrow project. We'll just have to wait on some discussion in object-store-python for exactly where that should live. I don't have an opinion myself on where this should live, but it should be on the order of 100 lines of code wherever it is (unless the v3 store api changes dramatically) |
👍
I want to keep an open mind about what the core stores provided by Zarr-Python are. My current thinking is that we should just do a |
This is no longer an issue, s3fs has much more relaxed deps than it used to. Furthermore, it's very likely to be already part of an installation environment. |
I agree with that. I think it is beneficial to keep the number of dependencies of core zarr-python small. But, I am open for discussion.
Sure! That is certainly useful. |
This is awesome work, thank you all!!! |
Co-authored-by: Deepak Cherian <[email protected]>
The I'd like to update this PR soonish to use that library instead. |
If the zarr group prefers object-store-rs, we can move it into the zarr-developers org, if you like. I would like to be involved in developing it, particularly if it can grow more explicit fsspec compatible functionality. |
I have a few questions because the
I like that |
This came up in the discussion at https://github.com/zarr-developers/zarr-python/pull/2426/files/5e0ffe80d039d9261517d96ce87220ce8d48e4f2#diff-bb6bb03f87fe9491ef78156256160d798369749b4b35c06d4f275425bdb6c4ad. By default, it's passed as Does it look compatible with what you need? |
Update obstore tests
Co-authored-by: Davis Bennett <[email protected]>
I'm making some progress on the failing tests in kylebarron#3 (currently up to 37/67 passing from 3/67). I need some help understanding the expected behavior of this test: zarr-python/src/zarr/testing/store.py Lines 297 to 315 in 122760f
The expected keys for store.list_dir('foo') are |
src/zarr/storage/object_store.py
Outdated
async def set_if_not_exists(self, key: str, value: Buffer) -> None: | ||
buf = value.to_bytes() | ||
await obs.put_async(self.store, key, buf, mode="create") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
async def set_if_not_exists(self, key: str, value: Buffer) -> None: | |
buf = value.to_bytes() | |
await obs.put_async(self.store, key, buf, mode="create") | |
async def set_if_not_exists(self, key: str, value: Buffer) -> None: | |
buf = value.to_bytes() | |
try: | |
await obs.put_async(self.store, key, buf, mode="create") | |
except AlreadyExistsError: | |
pass |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks Joe, pre-commit recommended a slightly different version (kylebarron@86951b8) but this fixed one of the failing tests
Leaving an update that we're down to 6 failing tests in kylebarron#3. I probably won't have time to work on the tests again for several days at least. Test 1The store will need to be make serializable for Dask to work Test 2-4Obstore has the end byte on a range request as exclusive whereas Zarr-Python tests have this as inclusive. I want to confirm the intended behavior before proposing any more changes.
Test 5Obstore doesn't like
Test 6See #1661 (comment) for an explanation
|
I decided to try out an S3Store in https://gist.github.com/maxrjones/0318a87d6179faf4eacbe01beb6d3eb6 (all previous tests were run with local or memory stores). I ran into an issue where Just for clarification, I know that list accepts as 'prefix' argument but the prefix I think needs to be configured at the store level since it should apply to all operations including get, set, etc. |
Thanks for your help @maxrjones!
I might suggest postponing this until we validate that obstore does provide a performance improvement. I'm not sure how to add pickle support.
Obstore defines range requests as
I don't think there should be two different ways to specify "the beginning of the buffer". |
this also holds for |
Get most of the tests to pass
Sorry for all the churn here, but I realized the actual difference seems to be that ByteRangeRequest is actually [start, length] rather than [start, end]. I guess this makes the inclusive/exclusive end debate moot, but was still super surprising to me. See #2437 for any further discussions on byte range representations. |
An update for anyone following this PR following a sync with @kylebarron today - the implementations obstore.store.LocalStore and obstore.store.MemoryStore backed Zarr stores seem good to go, but AzureStore, GCSStore, and S3Stores will require more work to make the Zarr storage backend work relative to a specified root prefix (the object_store rust library only supports relativity to the bucket). Kyle and I plan to pair on finishing up this PR in the new year. I hope the work early next year will unblock performance testing of Zarr V3 + obstore relative to Zarr V2 and Zarr V3 + icechunk, and would like to collaborate with Earthmover + others on those performance tests. |
A Zarr store based on
obstore
, which is a Python library that uses the Rustobject_store
crate under the hood.object-store is a rust crate for interoperating with remote object stores like S3, GCS, Azure, etc. See the highlights section of its docs.
obstore
maps async Rust functions to async Python functions, and is able to streamGET
andLIST
requests, which all make it a good candidate for use with the Zarr v3 Store protocol.You should be able to test this branch with the latest pre-release version of
obstore
:TODO: