fix: get_prune_stats returns homogenous ArrayRef

[cmackenzie1]

Description

Switch the get_prune_stats functions to use None to represent null
instead of ScalarValue::Null as ArrayRef must be of all the same
type.

Related Issue(s)

• closes Datafusion: unreachable code reached when parsing statistics with missing columns #1374

Documentation

https://github.com/apache/arrow-datafusion/blob/dd5e1dbbfd20539b40ae65acb8883f7e392cba92/datafusion/core/src/physical_optimizer/pruning.rs#L54-L72

[cmackenzie1]

Looks like this change causes this test to begin failing and I am unsure of what the correct behavior should be here.

---- operations::delete::tests::test_delete_on_mixed_columns stdout ----
thread 'operations::delete::tests::test_delete_on_mixed_columns' panicked at 'Could not determine null type: Generic("Unsupported data type for Delta Lake Dictionary(UInt16, Utf8)")', rust/src/operations/transaction/state.rs:205:55
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    operations::delete::tests::test_delete_on_mixed_columns

test result: FAILED. 94 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.42s

[cmackenzie1]

Thoughts on this diff? My concern is there is no explicit restriction on the types, not sure if that matters though

diff --git a/rust/src/delta_datafusion.rs b/rust/src/delta_datafusion.rs
index 43aed52..6ce3f1e 100644
--- a/rust/src/delta_datafusion.rs
+++ b/rust/src/delta_datafusion.rs
@@ -590,11 +590,14 @@ pub(crate) fn get_null_of_arrow_type(t: &ArrowDataType) -> DeltaResult<ScalarVal
                 TimeUnit::Nanosecond => ScalarValue::TimestampNanosecond(None, tz),
             })
         }
+        ArrowDataType::Dictionary(k, v) => Ok(ScalarValue::Dictionary(
+            k.clone(),
+            Box::new(get_null_of_arrow_type(v).unwrap()),
+        )),
         //Unsupported types...
         ArrowDataType::Float16
         | ArrowDataType::Decimal256(_, _)
         | ArrowDataType::Union(_, _)
-        | ArrowDataType::Dictionary(_, _)
         | ArrowDataType::LargeList(_)
         | ArrowDataType::Struct(_)
         | ArrowDataType::List(_)

[rtyler]

From my perspective returning the appropriate nullsy type for the column (which is how I understand this) makes sense. It does feel like some Arrow type system wackiness that there's a Null type and also a nullsy boolean, etc 😛