[TEST] Compatibility tests for data formats

[mythrocks]

Here is a list of tests to confirm data format compatibility with Apache Spark, in the Spark RAPIDS plugin. This list is a work in progress:

Orc:

• CPU fallback for user-defined types in ORC read/write: OrcQuerySuite.scala#L108 #8730 CPU fallback for user-defined types in ORC read/write: OrcQuerySuite.scala#L108
• Add tests for column names with dots #8704 SPARK-21791: Support for ORC columns with names containing dot('.'). Refer to OrcQuerySuite.scala#L695.
• Schema evolution: Test ORC reads after table schema changes #8692
  • Old partitions created before columns are appended to the table. New columns should have nulls in the old partitions.
  • Old partitions created before columns are added in the middle of the table. New columns should have nulls in the old partitions. (Not supported by Spark V1 datasource)
  • Old partitions created before columns are dropped at the end of the table. New columns should have nulls in the old partitions. (Not suported by Spark V1 datasource)
  • Old partitions created before columns are dropped in the middle of the table. New columns should have nulls in the old partitions. (Not supported by Spark V1 datasource)
  • Old partitions created before columns are renamed. (Not supported by Spark V1 datasource)
  • Old partitions created before new columns are added with the names of previously dropped columns. New columns should have nulls in the old partitions. (Not supported by Spark V1 datasource)
• SPARK-5309: ORC STRING column uses dictionary compression. OrcQuerySuite.scala#359. (Note: Not sure how this test is verifying dictionary_v2). #8797 SPARK-5309: ORC STRING column uses dictionary compression. OrcQuerySuite.scala#359. (Note: Not sure how this test is verifying dictionary_v2).
• ORC reads at scale with all null values: Like OrcQuerySuite.scala#L173, but with large number of rows. #8731 ORC reads at scale with all null values: Like OrcQuerySuite.scala#L173, but with large number of rows.
• SPARK-16610: Honour orc.compress on writes, when compress is unset: OrcQuerySuite.scala#L189. #8781 SPARK-16610: Honour orc.compress on writes, when compress is unset: OrcQuerySuite.scala#L189.
• compress should be honoured when set (ZLIB, Snappy, None). Refer to OrcQuerySuite.scala#L224. #8782 compress should be honoured when set (ZLIB, Snappy, None). Refer to OrcQuerySuite.scala#L224.
• [TEST] Test dictionary encoding in ORC file for nested types with lots of rows. #8809 Test directory encoding in ORC file for nested types with lots of rows
• SPARK-9170: Upper case ORC column names are not implicitly stored in lowercase. Refer to OrcQuerySuite.scala:371 #8793 SPARK-9170: Upper case ORC column names are not implicitly stored in lowercase. Refer to OrcQuerySuite.scala:371
• ORC reader should not fall back to using the ORC file's write schema when reading an ORC file with an empty read schema #8838
• When reading an invalid ORC file, SQLConf.IGNORE_CORRUPT_FILES should be honored #8840
• Add a test to check if ORC Row-index stride size is honored #8841
• Selecting complex fields: (Sampling from SchemaPruningSuite.)
  • Write a test for selecting a single complex field array and its parent struct array #8712
  • Write a test for partial schema intersection - select missing subfield #8713
  • Write a test for selecting a single complex field and in where clause #8714
  • Write a test for reading nested column on generator output  #8715
• #8856 ORC version V_0_11 and V_0_12 are readable by plugin/CUDF. Refer to TestNewIntegerEncoding#testBasicOld.
• Test predicate pushdown (PPD) with timestamps, decimals, booleans, etc. Refer to OrcQuerySuite.scala#L464. #8823 Test predicate pushdown (PPD) with timestamps, decimals, booleans, etc. Refer to OrcQuerySuite.scala#L464.
• Statistics tests for ORC files written by GPU #8763
• Leverage Apache ORC example files in ORC tests Leverage Apache ORC example files in ORC tests #9215

Parquet:

• Add test for the timestamp error case described in SPARK-10177 #8693

• Add test for Parquet schema interpretation problem described in SPARK-16344 #8694

• Add an xfail test for Parquet reads for LIST<STRUCT<int, string>> #8708

• Add tests for column names with dots #8704 Support for Parquet columns with names containing dot('.')

• Selecting complex fields: (Sampling from SchemaPruningSuite.)

  • Write a test for selecting a single complex field array and its parent struct array #8712
  • Write a test for partial schema intersection - select missing subfield #8713
  • Write a test for selecting a single complex field and in where clause #8714
  • Write a test for reading nested column on generator output  #8715

• Add test to verify Parquet predicate pushdown for fields having dots in the names #9094

• [Test] Test predicate pushdown (PPD) with timestamps, decimals, booleans, etc for Parquet file #9127

• P3: [P3] Parquet reads with CONVERT_METASTORE_PARQUET_WITH_SCHEMA_MERGING=true. #9074 (No reference Spark tests. Not sure how this is tested.)

• Support parquet.block.size to control row group size for parquet [FEA] Support parquet.block.size to control row group size for parquet #9126

• Statistics tests for Parquet files written by GPU #8762

• Add test to verify the fallback for UDT for parquet, refer to link1 and link2 Add test to verify the fallback for UDT for parquet #9137

• Test compatibility between pyarrow and GPU

• Test compatibility between fastparquet and GPUTest compatibility between fastparquet and GPU #9550

• P1: Written/read by Spark and read/written by GPU. Generate files in parquet testing cases

• P1: Written/read by Hive and read/written by GPU. Generate files in parquet testing cases

• Verify compression codec in Parquet file #9151

Tasks

Preview Give feedback

1. Schema evolution: Test ORC reads after table schema changes #8692
   task test +2
   
2. Add test for the timestamp error case described in SPARK-10177 #8693
   task test +2
   
3. Add test for Parquet schema interpretation problem described in SPARK-16344 #8694
   task test +2
   
4. Add an xfail test for Parquet reads for LIST<STRUCT<int, string>> #8708
   task test +2
   
5. CPU fallback for user-defined types in ORC read/write: OrcQuerySuite.scala#L108 #8730
   task test +2
   
6. ORC reads at scale with all null values: Like OrcQuerySuite.scala#L173, but with large number of rows. #8731
   test +1
   
7. Test predicate pushdown (PPD) with timestamps, decimals, booleans, etc. Refer to OrcQuerySuite.scala#L464. #8823
   test +1
   
8. Leverage Apache ORC example files in ORC tests #9215
   test +1

[mythrocks]

Argh. Ignore the previous comment. The task descriptions are rendered unreadable if not logged in through Github Enterprise.

Sorry for the confusion. There isn't a problem here.