Default to RemoteStore for fsspec URIs #2198

TomAugspurger · 2024-09-17T16:16:55Z

This allows for specifying a RemoteStore by an fsspec URI + storage options, as a convenience (and backwards compatibility shim) for creating a RemoteStore:

In [1]: import zarr.store

In [2]: g = zarr.open_group(store="az://daymet-zarr/annual/hi.zarr", zarr_format=2, storage_options={"account_name": "daymeteuwest"})

In [3]: list(g.arrays())
Object at .zmetadata is not recognized as a component of a Zarr hierarchy.
Out[3]: 
[<Array az://daymet-zarr/annual/hi.zarr/lambert_conformal_conic shape=() dtype=int16>,
 <Array az://daymet-zarr/annual/hi.zarr/lat shape=(584, 284) dtype=float32>,
...

Closes #2195

TomAugspurger · 2024-09-17T16:18:27Z

src/zarr/store/common.py

    elif isinstance(store_like, str):
-        return StorePath(await LocalStore.open(root=Path(store_like), mode=mode or "r"))
+        try:
+            fs, path = fsspec.url_to_fs(store_like, **storage_options)


Nitpick: this constructs the actual filesystem class instance, but we have to go back to a URL (and reconstruct the filesystem instance) for the RemoteStore interface. It'd be nice to avoid that.

TomAugspurger · 2024-09-17T16:20:47Z

src/zarr/store/common.py

+        try:
+            fs, path = fsspec.url_to_fs(store_like, **storage_options)
+        except Exception:
+            # not sure what to do here, but I don't want this to fail...


This is bad, but in my experience fsspec.url_to_fs can run pretty much arbitrary code, depending on what packages you have installed and what URI you pass. I really don't want this to fail and cause issues for people who just want a local path. maybe we'd always get an fsspec LocalFileSystem for a plain, non-fsspec URI string, but I'm not sure.

TomAugspurger · 2024-09-17T16:27:06Z

src/zarr/store/common.py

+    else:
+        raise TypeError
+
+    if path is not None:


Oops, I forgot to revert this. It's not used yet.

We call make_store_path multiple times in a single call by the user (e.g. save_array calls make_store_path and passes the result to AsyncArray.create which also calls make_store_path). I was thinking about consolidating all the store / path stuff in a single spot, but it ended up being a bit much and I wasn't sure what the tolerance was for expanding the API of methods like AsyncArray.create even more. Do we want users to do

AsyncArray.create(store=StorePath(RemoteStore(..., **storage_options), path))

or

AsyncArray.create(store="s3://bucket/path.zarr", storage_options=...)

dstansby

I don't have much opinion on the actual change, but is it worth adding warnings everywhere if storage_options is passed by the user (ie not None), but they haven't passed an fsspec URL so they aren't used?

src/zarr/api/asynchronous.py

jhamman · 2024-09-18T15:12:52Z

src/zarr/store/common.py

    elif isinstance(store_like, Path):
-        return StorePath(await LocalStore.open(root=store_like, mode=mode or "r"))
+        result = StorePath(await LocalStore.open(root=store_like, mode=mode or "r"))


we can pass **storage_options to LocalStore as well

could be missing something, but I don't think that'll work. LocalStore.open will call LocalStore.__init__, which just takes root and mode, which are passed as regular args here.

I see what you mean. I was thinking auto_mkdir would be passed through but if that's not the case, let's not get distracted here.

jhamman · 2024-09-18T15:17:18Z

src/zarr/store/common.py

    elif isinstance(store_like, str):
-        return StorePath(await LocalStore.open(root=Path(store_like), mode=mode or "r"))
+        storage_options = storage_options or {}
+        fs, _ = fsspec.url_to_fs(store_like, **storage_options)


A few thoughts here:

It would be nice to avoid creating the FileSystem if possible. A lighter approach could attempt to parse the string and look for :// (and not file://).

If we do have to create filesystem, it would be nice to reuse it when creating the RemoteStore. Something like RemoteStore.from_filesystem(...) could be a nice pattern.

See below for the way we used to do this. May be worth replicating parts of this

zarr-python/zarr/storage.py

Line 170 in e1d98cd

if "://" in store or "::" in store:

Just pushed that refactor. It's a bit bigger than I'd like, but the basic version is that RemoteStores's __init__ matches what it uses internally: you can only given it an async filesystem.

There's a from_url and from_path that handles the other stuff that was previously in __init__.

I removed RemoteStore._url. I think that that's safe to remove, but I need to actually think through how it interacted with store.path

…re' into fix/auto-remote-store

* v3: chore: update pre-commit hooks (zarr-developers#2222) fix: validate v3 dtypes when loading/creating v3 metadata (zarr-developers#2209) fix typo in store integration test (zarr-developers#2223) Basic Zarr-python 2.x compatibility changes (zarr-developers#2098) Make Group.arrays, groups compatible with v2 (zarr-developers#2213) Typing fixes to test_indexing (zarr-developers#2193) Default to RemoteStore for fsspec URIs (zarr-developers#2198) Make MemoryStore serialiazable (zarr-developers#2204) [v3] Implement Group methods for empty, full, ones, and zeros (zarr-developers#2210) implement `store.list_prefix` and `store._set_many` (zarr-developers#2064) Fixed codec for v2 data with no fill value (zarr-developers#2207)

Default to RemoteStore for fsspec URIs

f81364b

TomAugspurger commented Sep 17, 2024

View reviewed changes

TomAugspurger added 2 commits September 17, 2024 11:29

fixup

64b9371

fixup

72fb559

dstansby reviewed Sep 17, 2024

View reviewed changes

src/zarr/api/asynchronous.py Show resolved Hide resolved

jhamman reviewed Sep 18, 2024

View reviewed changes

jhamman approved these changes Sep 18, 2024

View reviewed changes

jhamman reviewed Sep 18, 2024

View reviewed changes

TomAugspurger added 5 commits September 18, 2024 11:27

wip

205807e

Merge remote-tracking branch 'upstream/v3' into fix/auto-remote-store

69b6cda

fixup

0ea04af

Added check for invalid.

c8535af

fixup

fe089fc

jhamman approved these changes Sep 19, 2024

View reviewed changes

jhamman and others added 5 commits September 19, 2024 08:53

Merge branch 'v3' into fix/auto-remote-store

4db6386

Merge remote-tracking branch 'upstream/v3' into fix/auto-remote-store

940084c

Merge remote-tracking branch 'refs/remotes/origin/fix/auto-remote-sto…

41b2be5

…re' into fix/auto-remote-store

fixup

8bbc508

Merge branch 'v3' into fix/auto-remote-store

1610371

jhamman merged commit f538662 into zarr-developers:v3 Sep 19, 2024
26 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Default to RemoteStore for fsspec URIs #2198

Default to RemoteStore for fsspec URIs #2198

TomAugspurger commented Sep 17, 2024

TomAugspurger Sep 17, 2024

TomAugspurger Sep 17, 2024

TomAugspurger Sep 17, 2024

dstansby left a comment

jhamman Sep 18, 2024

TomAugspurger Sep 18, 2024

jhamman Sep 19, 2024

jhamman Sep 18, 2024

jhamman Sep 18, 2024

TomAugspurger Sep 18, 2024

Default to RemoteStore for fsspec URIs #2198

Default to RemoteStore for fsspec URIs #2198

Conversation

TomAugspurger commented Sep 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dstansby left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment