[BUG] libcudf fails to load old Parquet array encoding properly

[jlowe]

Describe the bug
Older Parquet files may have arrays encoded in different ways, including a group followed immediately by a repeated primitive rather than the typical group->group list->primitive. libcudf is failing to load this older encoding properly, loading the data as LIST of LIST of INT32 rather than LIST of INT32.

Steps/Code to reproduce bug
Attached is a sample file which can be loaded with the following test program:

#include <iostream>

#include <cudf/io/parquet.hpp>
#include <cudf/lists/lists_column_view.hpp>
#include <cudf/types.hpp>

int main(int argc, char** argv) {
  auto table = cudf::io::read_parquet(
    cudf::io::parquet_reader_options::builder(cudf::io::source_info("pq891392009.parquet")));
  if (table.tbl->num_columns() != 1) {
    std::cerr << "Incorrect column count" << std::endl;
  }
  auto c0 = table.tbl->get_column(0);
  if (c0.type().id() != cudf::type_id::LIST) {
    std::cerr << "Top level column type mismatch, expected LIST" << std::endl;
    return 1;
  }
  auto lc = cudf::lists_column_view(c0);
  auto child = lc.child();
  if (child.type().id() != cudf::type_id::INT32) {
    std::cerr << "child column type mismatch, expected INT32" << std::endl;
    return 1;
  }
  return 0;
}

pq891392009.parquet.gz

Expected behavior
libcudf should load the file as a LIST of INT32 child column. Both the parquet-cli and Spark CPU can load this file properly.

From parquet-cli:

$ java -cp 'target/*:target/dependency/*' org.apache.parquet.cli.Main meta /tmp/dbg/pq891392009.parquet

File path:  /tmp/dbg/pq891392009.parquet
Created by: RAPIDS Spark Plugin
Properties: (none)
Schema:
message spark_schema {
  required group f (LIST) {
    repeated int32 array;
  }
}


Row group 0:  count: 1  43.00 B records  start: 4  total: 43 B
--------------------------------------------------------------------------------
         type      encodings count     avg size   nulls   min / max
f.array  INT32     _   _     2         21.50 B    0       "0" / "1"

$ java -cp 'target/*:target/dependency/*' org.apache.parquet.cli.Main cat /tmp/dbg/pq891392009.parquet
{"f": [0, 1]}

and from Spark CPU:

scala> spark.read.parquet("/tmp/dbg/pq891392009.parquet").printSchema
root                                                                            
 |-- f: array (nullable = true)
 |    |-- element: integer (containsNull = true)


scala> spark.read.parquet("/tmp/dbg/pq891392009.parquet").show
+------+
|     f|
+------+
|[0, 1]|
+------+

[GregoryKimball]

Thank you @jlowe for raising this. Would you please share a bit more about the origin of this older Parquet file?

[jlowe]

The file is derived from a disabled unit test in the NVIDIA/spark-rapids repository, but it's behavior in the plugin is similar to an issue we ran into with a user who was working with older Parquet files. When I started fixing the Spark plugin issues raised by the unit test, I eventually ran into these issues on the cudf side.