Read Avro Data File

[potter420]

Describe your feature request

Polars or Py-polars can have some function to read avro data files.

I wrote some small packages to read avro into arrow struct array format based on arrow2 and avro-rs packages. Also have some python bindings. It only deal with flattened data for now, but should be able to handle any kind of nested data soon.
I'm planning to release it in the next few days.

Do you see any possibilities of integrating it with polars?

Thank you.

[jorgecarleitao]

Hey! developer of arrow2 here; Super exciting! do you have a link or public repo over this?

[potter420]

I organizing my code to publish it soon, my private repo have rusoto as dependencies to read from S3, so I would remove that.
I have to say I'm pleased with the performance. The 5x faster than the equivalent code in python (may be poorly written) and concurrency out of the box with rayon. I'm cherry picking the commit that you have added all the FFI types to achieve maximum compatibility with pyarrow.

[potter420]

@jorgecarleitao Here is my repos, feel free to use/reuse any parts of the code.
https://github.com/potter420/arrow_avro_rs

Thanks for creating arrow2. I'm having a blast with it.

[ritchie46]

Nice @potter420. This functionality would definitely be valuable behind a feature flag.

[ghuls]

Reading and writing Avro files is supported now.