Arrow as zero-serialization target

[infogulch]

Hi! I've just found iggy, and I noticed the roadmap mentions zero-copy deserialization. Have you considered using apache arrow as a message serialization format? Arrow may provide interoperability benefits, considering it can be read and written from many languages without copying.

[hubcio]

hello! to be honest, we didn't consider it. right now are intensely experimenting with multiple approaches. we tested rkyv, besides that we also cooked our own zero-copy format. Thank you for this suggestion, I will get back to you in some time with some more thoghts; i'm sure @numinnex will have more comments 🦀 .

[infogulch]

👋

I appreciate that in an experimentation phase you want to keep your options open. Arrow seems to be a practical choice since it is a pretty complete standard, supports a wide array of languages ¹, it would likely be a drop-in implementation, and it is also an apache project. (Maybe another option is spiraldb/vortex which is in a nascent stage but has some neat features.)

I just hoped to get arrow on your radar before you guys settled on something since I hadn't seen it mentioned anywhere in the repo yet.

🚀

Footnotes

1. https://arrow.apache.org/docs/status.html ↩