[FEA] Support reading Avro

[GaryShen2008]

This is the epic issue to trace the tasks for reading Avro.
cuDF has some basic support on reading Avro.

High priority

• Avro: cuDF JNI support
• Primitve types
  • [FEA] Support reading Avro: primitive types #4935 (Numeric, Boolean, String)
  • Check how Spark/cuDF handle null values for primitive types.
• Logical types
  • [FEA] add date/timestamp support for AVRO reader #4982 (cudf_dependency)
  • Support decimal type. NOTE: the decimal in avro can be backed by fixed or bytes (cudf_dependency)
• Complex types
  • Support Array type (cudf_dependency)
  • Support Record type (Spark maps to StructType) (cudf_dependency)
  • Support Map type (cudf_dependency)
• Support avro read options
• Support avro read configurations
• [FEA] Support AVRO 1.2 version format #5018 (cudf_dependency)
• Add integration tests for reading avro files with corrupt files
• [FEA] Support basic schema evolution for AVRO reading #5691

Low priority

• [FEA] Unify the some functions for avro reader #4983
• Add the MULTI-THREADED reading support for avro #5148
• Add the COALESCING reading support for avro #5149
• [BUG] The coalesced AVRO file may contain different sync markers if the sync marker varies in the avro files being coalesced. #5312
• [FEA]Sort the single block info by the header for AVRO coalescing reading.  #5347
• Support pushed down filters for Avro
• More tests to cover all the compression types.
• Support the from_avro SQL function running on GPU

Needs Triage

• Primitve types
  • Check how Spark/cuDF handle columns of null type.
  • Support null type (Can support on plugin side in post-handling)
  • Support the Bytes type (cuDF native supports it by mapping to string, differing from the Spark mapping to BinaryType) (cudf_dependency)
• Logical types
  • Check the support of Local timestamp (millisecond and microsecond precision and ) in Spark (If yes, add a new task for adding the support)
  • Check the support of Time (millisecond and microsecond precision and ) in Spark (If yes, add a new task for adding the support)
  • Check the support of UUID and Duration in Spark (If yes, add a new task for adding the support)
• Complex types
  • Support Enum type, (cuDF native supports it by mapping to String or INT32, but Spark maps only to StringType.) (cudf_dependency)
  • Support Fixed type (cudf_dependency)
  • Support Union type (cudf_dependency)

[sperlingxx]

Hi @firestarman @GaryShen2008, shall we deem #5691 as a sub-issue of this epic?

[firestarman]

Yes please.