feat(rust): Add support for arrow binary type

[ozgrakkurt]

closes #4903

@ritchie46 can you check if this makes sense when you have time?

I redirected arrow dep to my fork temporarily

[ozgrakkurt]

@ritchie46 tested this using code I already had and it works for querying parquet files, filtering reading data etc. Just can't get the tests to compile. Also probably need to add some tests, can you point where I should add tests?

[ritchie46]

@ozgrakkurt thanks for the initial push on this.

Can you feature gate this functionality? I only want this compiled if dtype-binary is activated.

Tests can be placed under polars/tests/it/.

[ozgrakkurt]

@ritchie46 could you do the feature gating if it is possible? Not sure how exactly to do it

[ozgrakkurt]

@ritchie46 can you check again?

[ozgrakkurt]

also closes #4877

[ritchie46]

Thanks a lot for this @ozgrakkurt. I know this is a lot of work. I have left a few remarks on places where I think we can remove the code and let the Series simply return null. This often does this already automatically, so we can probably just remove the code.

I am only not sure we should implement Ord for binary? E.g. min, max, sort? Are byte slices Ord in rust? If so, we can follow.

[ritchie46]

These can be removed. The Series can just return null for the Binary dtype

[ozgrakkurt]

I get this compilation error when I remove this:

error[E0277]: the trait bound `datatypes::BinaryType: datatypes::PolarsIntegerType` is not satisfied
   --> polars/polars-core/src/series/implementations/binary.rs:326:45
    |
326 |         QuantileAggSeries::median_as_series(&self.0)
    |         ----------------------------------- ^^^^^^^ the trait `datatypes::PolarsIntegerType` is not implemented for `datatypes::BinaryType`
    |         |
    |         required by a bound introduced by this call
    |
    = help: the following other types implement trait `datatypes::PolarsIntegerType`:
              datatypes::Int16Type
              datatypes::Int32Type
              datatypes::Int64Type
              datatypes::Int8Type
              datatypes::UInt16Type
              datatypes::UInt32Type
              datatypes::UInt64Type
              datatypes::UInt8Type
note: required for `chunked_array::ChunkedArray<datatypes::BinaryType>` to implement `chunked_array::ops::aggregate::QuantileAggSeries`
   --> polars/polars-core/src/chunked_array/ops/aggregate.rs:715:9
    |
715 | impl<T> QuantileAggSeries for ChunkedArray<T>
    |         ^^^^^^^^^^^^^^^^^     ^^^^^^^^^^^^^^^

error[E0277]: the trait bound `datatypes::BinaryType: datatypes::PolarsIntegerType` is not satisfied
   --> polars/polars-core/src/series/implementations/binary.rs:339:47
    |
339 |         QuantileAggSeries::quantile_as_series(&self.0, quantile, interpol)
    |         ------------------------------------- ^^^^^^^ the trait `datatypes::PolarsIntegerType` is not implemented for `datatypes::BinaryType`
    |         |
    |         required by a bound introduced by this call
    |
    = help: the following other types implement trait `datatypes::PolarsIntegerType`:
              datatypes::Int16Type
              datatypes::Int32Type
              datatypes::Int64Type
              datatypes::Int8Type
              datatypes::UInt16Type
              datatypes::UInt32Type
              datatypes::UInt64Type
              datatypes::UInt8Type
note: required for `chunked_array::ChunkedArray<datatypes::BinaryType>` to implement `chunked_array::ops::aggregate::QuantileAggSeries`
   --> polars/polars-core/src/chunked_array/ops/aggregate.rs:715:9
    |
715 | impl<T> QuantileAggSeries for ChunkedArray<T>
    |         ^^^^^^^^^^^^^^^^^     ^^^^^^^^^^^^^^^

[ritchie46]

These can be removed. The Series can just return null for the Binary dtype

[ritchie46]

These can be removed. The Series can just return null for the Binary dtype

[ritchie46]

Should binary implement sort?

[ozgrakkurt]

yes imo, since string has it. Sorting binary is useful as well

[ozgrakkurt]

Thanks a lot for this @ozgrakkurt. I know this is a lot of work. I have left a few remarks on places where I think we can remove the code and let the Series simply return null. This often does this already automatically, so we can probably just remove the code.

I am only not sure we should implement Ord for binary? E.g. min, max, sort? Are byte slices Ord in rust? If so, we can follow.

Yes if T is Ord any slice of T is ord as well I think. Those operations are useful in binary as well

[ritchie46]

Working on it!

[ritchie46]

Something went wrong. Reopend on #5122