[BUG] Parquet reader cannot use filters in tandem with column projection

[wence-]

Describe the bug

Reading a column-projected parquet file (that has $N$ on disk columns) and simultaneously applying a filter expression to the read columns is not possible, unless the columns we request to read are the first $[0, \dots, K), K \le N$.

The problem appears to be that the filter expression must (eventually) refer to columns by their index, however one part of the code (that uses row-group filters) expects these indices to match the indices of the columns in the un-projected metadata, whereas another part needs the indices to match the requested projected column indices. It is not possible to arrange this to be the case for either a reordering or a non-dense subset of columns.

Steps/Code to reproduce bug

diff --git a/cpp/tests/io/parquet_reader_test.cpp b/cpp/tests/io/parquet_reader_test.cpp
index abbd0c97f0..1113d6c578 100644
--- a/cpp/tests/io/parquet_reader_test.cpp
+++ b/cpp/tests/io/parquet_reader_test.cpp
@@ -1406,6 +1406,28 @@ TEST_F(ParquetReaderTest, FilterIdentity)
   CUDF_TEST_EXPECT_TABLES_EQUAL(*result.tbl, *result2.tbl);
 }
 
+TEST_F(ParquetReaderTest, FilterWithColumnProjection)
+{
+  auto [src, filepath] = create_parquet_with_stats("FilterWithColumnProjection.parquet");
+  auto val = cudf::numeric_scalar<uint32_t>{10};
+  auto lit = cudf::ast::literal{val};
+  auto col_ref = cudf::ast::column_name_reference{"col_uint32"};
+  auto col_index = cudf::ast::column_reference{0};
+  auto read_expr = cudf::ast::operation(cudf::ast::ast_operator::LESS, col_ref, lit);
+  auto filter_expr = cudf::ast::operation(cudf::ast::ast_operator::LESS, col_index, lit);
+
+  auto predicate = cudf::compute_column(src, filter_expr);
+  auto projected_table = cudf::table_view{{src.get_column(2), src.get_column(0)}};
+  auto expected = cudf::apply_boolean_mask(projected_table, *predicate);
+
+  auto read_opts =
+    cudf::io::parquet_reader_options::builder(cudf::io::source_info{filepath})
+     .columns({"col_double", "col_uint32"})
+     .filter(read_expr);
+  auto result = cudf::io::read_parquet(read_opts);
+  CUDF_TEST_EXPECT_TABLES_EQUAL(*result.tbl, *expected);
+}
+
 TEST_F(ParquetReaderTest, FilterReferenceExpression)
 {
   auto [src, filepath] = create_parquet_with_stats("FilterReferenceExpression.parquet");

Build the cudf tests and then run:

$ ./PARQUET_TEST --gtest_filter=ParquetReaderTest.FilterWithColumnProjection
Note: Google Test filter = ParquetReaderTest.FilterWithColumnProjection
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from ParquetReaderTest
[ RUN      ] ParquetReaderTest.FilterWithColumnProjection
unknown file: Failure
C++ exception with description "CUDF failure at:/home/coder/cudf/cpp/src/io/parquet/predicate_pushdown.cpp:111: Invalid type and stats combination" thrown in the test body.

[  FAILED  ] ParquetReaderTest.FilterWithColumnProjection (13 ms)
[----------] 1 test from ParquetReaderTest (13 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (13 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] ParquetReaderTest.FilterWithColumnProjection

 1 FAILED TEST

Expected behavior

This should work.

[karthikeyann]

Looking into it.
Issue is with using output schema as schema of parquet while reading statistics data. parquet schema needs to be extracted regardless of output schema.