DeltaStorageHandler is not serializable

[chitralverma]

Environment

Delta-rs version: latest

Binding:
Python

Environment:

• Cloud provider: all
• OS: mac ventura, m1
• Other:

---

Bug

What happened:
I am trying to add delta connector to Polars, but which doing a lazy scan polars tries to serialize function and objects, see here. This line breaks with the below exception as DeltaFileSystemHandler is not serializable I suppose,

TypeError: cannot pickle 'builtins.DeltaFileSystemHandler' object

It will be great if this could be serializable, otherwise on Polars side we will have to rely on fsspec or pyarrow.fs, bypassing the DeltaStorageHandler implementation altogether.

What you expected to happen:
Exception should not occur.

How to reproduce it:
You can use the following minimal code to reproduce this.

import deltalake
from deltalake.fs import DeltaStorageHandler
dsh = DeltaStorageHandler("./delta-table")

import pickle
pickle.dumps(dsh)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot pickle 'builtins.DeltaFileSystemHandler' object

More details:

[roeap]

Hi @chitralverma - had a quick look into this it seems this is not natively supported by pyo3 modules. That being said, there are some pointers in an #pyo3/100 that I can explore to make this work.

Really looking forward to a polars integration - thanks! 👍

[chitralverma]

Hi @chitralverma - had a quick look into this it seems this is not natively supported by pyo3 modules. That being said, there are some pointers in an #pyo3/100 that I can explore to make this work.

Really looking forward to a polars integration - thanks! 👍

Thanks a lot

[ritchie46]

Hey @roeap. We also utilize pyo3 in polars and implemented pickle by adding a __getstate__ and __setstate__ method to the class.

See here for our implementation: https://github.com/pola-rs/polars/blob/4c79ef293975b48aaceefdf7d03611822affc7f5/py-polars/src/lazy/dsl.rs#L80

You can use the serde method of choice.

[roeap]

@ritchie46 @chitralverma - Thanks so much for your input - thanks to that this was quite straight forward and I have a working prototype!

By now I feel much more confident I can open a PR very soon 😆.