Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Working with index files and xr.open_mfdataset in a location without write permission #275

Open
matzech opened this issue Jan 10, 2022 · 3 comments

Comments

@matzech
Copy link

matzech commented Jan 10, 2022

I work with multiple grib files in a location without write permission. In locations with writing permission, I usually work with open_mfdataset from xarray like

ds = xr.open_mfdataset("201506/*.grib", engine="cfgrib")

As I would like to work with index files, I tried to follow the approach proposed in #126 , but the short_hash is actually the same for the different files when applied like:

ds = xr.open_mfdataset("201506/*.grib", engine="cfgrib", backend_kwargs={"indexpath":'mygrib.{short_hash}.idx'}
Is there another solution?

@alexamici
Copy link
Contributor

alexamici commented Jan 11, 2022

@matzech you may pass {"indexpath": ""} to disable writing the index, or you may try something along the line of (untested):

ds = xr.open_mfdataset("*.grib", engine="cfgrib", backend_kwargs={"indexpath": 'writable_folder{path}.{short_hash}.idx'})

@onnyyonn
Copy link

Is it possible to modify the {path} so that it contains only the basename? Also, what keys are available other than {path} and {short_hash}?

@FRidh
Copy link

FRidh commented Mar 6, 2023

As suggested in pydata/xarray#6512 I think we should use a temporary file/folder as default or disable it by default. It cannot be relied on that the folder the file is in is writeable nor that the index file is accurate. The index file is purely an optimization. In my opinion robustness is more important than performance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants