Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Passing Path objects to esm_datastore #540

Closed
aulemahal opened this issue Oct 19, 2022 · 0 comments · Fixed by #541
Closed

Passing Path objects to esm_datastore #540

aulemahal opened this issue Oct 19, 2022 · 0 comments · Fixed by #541

Comments

@aulemahal
Copy link
Contributor

aulemahal commented Oct 19, 2022

This is very minor, but I feel we should be able to pass pathlib.Path objects to intake_esm.esm_datastore.

Ex:

from pathlib import Path
from intake_esm import esm_datastore
p = Path('/path/to/catalog.json')

esm_datastore(p)

fails with

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 esm_datastore(p)

File .conda/espoprops/lib/python3.10/site-packages/intake_esm/core.py:94, in esm_datastore.__init__(self, obj, progressbar, sep, registry, read_csv_kwargs, storage_options, intake_kwargs)
     92     self.esmcat = ESMCatalogModel.from_dict(obj)
     93 else:
---> 94     self.esmcat = ESMCatalogModel.load(
     95         obj, storage_options=self.storage_options, read_csv_kwargs=read_csv_kwargs
     96     )
     98 self.derivedcat = registry or default_registry
     99 self._entries = {}

File .conda/espoprops/lib/python3.10/site-packages/intake_esm/cat.py:220, in ESMCatalogModel.load(cls, json_file, storage_options, read_csv_kwargs)
    218 storage_options = storage_options if storage_options is not None else {}
    219 read_csv_kwargs = read_csv_kwargs or {}
--> 220 _mapper = fsspec.get_mapper(json_file, **storage_options)
    222 with fsspec.open(json_file, **storage_options) as fobj:
    223     data = json.loads(fobj.read())

File .conda/espoprops/lib/python3.10/site-packages/fsspec/mapping.py:230, in get_mapper(url, check, create, missing_exceptions, alternate_root, **kwargs)
    199 """Create key-value interface for given URL and options
    200 
    201 The URL will be of the form "protocol://location" and point to the root
   (...)
    227 ``FSMap`` instance, the dict-like key-value store.
    228 """
    229 # Removing protocol here - could defer to each open() on the backend
--> 230 fs, urlpath = url_to_fs(url, **kwargs)
    231 root = alternate_root if alternate_root is not None else urlpath
    232 return FSMap(root, fs, check, create, missing_exceptions=missing_exceptions)

File .conda/espoprops/lib/python3.10/site-packages/fsspec/core.py:392, in url_to_fs(url, **kwargs)
    372 def url_to_fs(url, **kwargs):
    373     """
    374     Turn fully-qualified and potentially chained URL into filesystem instance
    375 
   (...)
    390         The file-systems-specific URL for ``url``.
    391     """
--> 392     chain = _un_chain(url, kwargs)
    393     if len(chain) > 1:
    394         inkwargs = {}

File .conda/espoprops/lib/python3.10/site-packages/fsspec/core.py:346, in _un_chain(path, kwargs)
    342     return out
    343 x = re.compile(".*[^a-z]+.*")  # test for non protocol-like single word
    344 bits = (
    345     [p if "://" in p or x.match(p) else p + "://" for p in path.split("::")]
--> 346     if "::" in path
    347     else [path]
    348 )
    349 if len(bits) < 2:
    350     return []

TypeError: argument of type 'PosixPath' is not iterable

So in fact, the error comes from fsspec. I nevertheless feel that intake_esm would be the best place to cast the path to string.

EDIT: The annotations of the functions actually mention pathlib.Path so this was supposed to work but was broken by the switch to fsspec I believe.

aulemahal added a commit to aulemahal/intake-esm that referenced this issue Oct 19, 2022
Cast paths to string before passing to fsspec.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant