You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The current implementation included in fsspec for a HDFS filesystem relies on pyarrow.hdfs.HadoopFileSystem, which is deprecated in favor of pyarrow.fs.HadoopFileSystem.
In general, this will require implementing an fsspec-compatible wrapper for the new arrow filesystems (so a wrapper "the other way around" as the one for fsspec-filesystems we have in pyarrow). Such a general wrapper could then also be used for the hdfs wrapper.
See #295 (comment) for context.
The current implementation included in
fsspecfor a HDFS filesystem relies onpyarrow.hdfs.HadoopFileSystem, which is deprecated in favor ofpyarrow.fs.HadoopFileSystem.In general, this will require implementing an fsspec-compatible wrapper for the new arrow filesystems (so a wrapper "the other way around" as the one for fsspec-filesystems we have in pyarrow). Such a general wrapper could then also be used for the hdfs wrapper.
Some time ago I started such a wrapper, which could be used as a starting point: https://github.com/intake/filesystem_spec/compare/master...jorisvandenbossche:pyarrow-wrapper?expand=1
The text was updated successfully, but these errors were encountered: