Allow open_virtual_dataset to read existing Kerchunk references

[norlandrhagen]

Allow open_virtual_dataset to read existing Kerchunk references as virtual datasets

• Closes Open on-disk kerchunk references as a virtual dataset #118
• Tests added
• Tests passing
• Changes are documented in docs/releases.rst
• New functionality has documentation

To Do:

• [JSON] trailing \\ in variable names. Fixed. json was adding trailing slashes on ref write.
• [JSON] coordinates not recognized. Fixed. This was related to json write.
• [JSON] coordinates set as data variables. Fixed. This was related to json write.
• [PARQUET] extract references from 1. LazyReferenceMapper or 2. Reconstruct reference from ✨Zarrquet ✨ format.
• [FUTURE PR] Convert inlined vars into numpy arrays. Note: After talking with @sharkinsspatial, it seems like the HDF reader he is working on won't inline at all. Should we: 1. Raise an error if any inline data exists or 2. Add logic to convert any inlined bytes to numpy arrays to maintain more compatibility with other Kerchunk readers?

Notes:

• Mypy checks in CI are currently disabled! [Tracking] Scheduled CI is failing on main #249
• The utility function _fsspec_openfile_from_filepath forced the fsspec filesystem to open a filepath (the fault of past me). I replaced it with a simple class that has a .open_file method that can be used as needed. The Kerchunk LazyReferenceMapper required a fsspec filesystem.

[keewis]

did you see #186? That was a second attempt after #119 to implement this, and contains some documentation that you might be able to reuse?

[norlandrhagen]

@keewis thanks for the rec. Having some docs already is nice.

[TomNicholas]

Great work @norlandrhagen !

Convert inlined vars into numpy arrays. Note: After talking with @sharkinsspatial, it seems like the HDF reader he is working on won't inline at all. Should we:

1. Raise an error if any inline data exists or
2. Add logic to convert any inlined bytes to numpy arrays to maintain more compatibility with other Kerchunk readers?

I feel like this mixes a few issues together. We do need to be able to read back inlined kerchunk references (though it's fine to add that feature in a follow-up PR), partly because even with Sean's PR we're still going to want to use the other kerchunk readers sometimes. I think it makes sense for @sharkinsspatial 's HDF reader not to create inlined refs, so long as we have another way to create inlined refs (i.e. using the normal xarray backend for that filetype).

Mypy checks in CI are currently disabled!

Should be fixed by #252

The utility function _fsspec_openfile_from_filepath forced the fsspec filesystem to open a filepath

Not totally sure I understand this but abstracting away fsspec details sounds good.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.37%. Comparing base (53a609f) to head (8d53227).
Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #251      +/-   ##
==========================================
+ Coverage   91.20%   91.37%   +0.17%     
==========================================
  Files          32       32              
  Lines        2057     2098      +41     
==========================================
+ Hits         1876     1917      +41     
  Misses        181      181              

Flag	Coverage Δ
unittests	91.37% <100.00%> (+0.17%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[norlandrhagen]

Thanks for the feedback @keewis and @TomNicholas. I think everything is addressed. The auto-detection is pretty rough, but hopefully if you're using this, you know the difference between parquet and json 🤷

[TomNicholas]

Thanks @norlandrhagen - happy to merge!