[BUG] Parquet list schema interpretation bug.

[nvdbaranec]

The following file loads incorrectly using the cudf parquet reader:

https://github.com/apache/parquet-testing/blob/master/data/repeated_no_annotation.parquet

Internally there are 2 columns:

• int
• list<struct<int64, string>>

The reader appears to interpret the latter as struct<struct<int64, string>. Ultimately this results in unpredictable/broken decoding.

The actual schema in the file is below ("phoneNumbers" is the column in question)

message spark_schema {
  required int32 id;
  optional group phoneNumbers {
    repeated group phone {
      required int64 number;
      optional binary kind (STRING);
    }
  }
}

[GregoryKimball]

Thank you @nvdbaranec for raising this issue. It seems like another strange file case, where if we rewrite the data by arrow or cudf, cudf produces the correct table on read.

df = pd.read_parquet('repeated_no_annotation.parquet')
df.to_parquet('pd_write.pq')
df2 = cudf.read_parquet('pd_write.pq')
assert cudf.DataFrame.from_pandas(df).to_pandas().equals(df2.to_pandas())
# rewrite with pandas - OK!

df = pd.read_parquet('repeated_no_annotation.parquet')
df2 = cudf.DataFrame.from_pandas(df)
df2.to_parquet('cudf_write.pq')
df3 = cudf.read_parquet('cudf_write.pq')
assert df2.to_pandas().equals(df3.to_pandas())
# rewrite with cudf - OK!

Note, cuDF-python doesn't support equals on dataframes with complex columns

[hyperbolic2346]

Changing this to properly read this as a list reveals another layer to this onion. The header shows the number of rows as 0, but the row groups inside have a valid row count. Need to determine how to properly reconcile when they disagree.

[nvdbaranec]

If this is a list column, the number of rows in the page data is just the number of values in the data stream - the number of rows (in the cudf sense) can only be determined by examining the repetition levels. What code is getting deceived here? Some sort of early out condition?

[jlowe]

Note that Apache Spark thinks the schema of this file is not an INT, LIST<STRUCT<INT64,STRING>> but rather an INT, STRUCT<LIST<STRUCT<INT64,STRING>>>.

scala> spark.read.parquet("repeated_no_annotation.parquet").printSchema
root
 |-- id: integer (nullable = true)
 |-- phoneNumbers: struct (nullable = true)
 |    |-- phone: array (nullable = true)
 |    |    |-- element: struct (containsNull = true)
 |    |    |    |-- number: long (nullable = true)
 |    |    |    |-- kind: string (nullable = true)

[hyperbolic2346]

Can you show the table that spark creates? Does it have one row?

[jlowe]

There are 6 rows. The curly braces indicate a struct, the square brackets indicate an array (list).

scala> spark.read.parquet("repeated_no_annotation.parquet").show(truncate=false)
+---+----------------------------------------------------------------+          
|id |phoneNumbers                                                    |
+---+----------------------------------------------------------------+
|1  |null                                                            |
|2  |null                                                            |
|3  |{[]}                                                            |
|4  |{[{5555555555, null}]}                                          |
|5  |{[{1111111111, home}]}                                          |
|6  |{[{1111111111, home}, {2222222222, null}, {3333333333, mobile}]}|
+---+----------------------------------------------------------------+

Here's the same data collected back to the Spark driver with each row printed per line. In this case square bracket indicates a structure level. Top-level structures are the entire row.

scala> spark.read.parquet("repeated_no_annotation.parquet").collect.foreach(println)
[1,null]
[2,null]
[3,[WrappedArray()]]
[4,[WrappedArray([5555555555,null])]]
[5,[WrappedArray([1111111111,home])]]
[6,[WrappedArray([1111111111,home], [2222222222,null], [3333333333,mobile])]]