[FEA] Support outputTimestampType being INT96

[tgravescs]

Is your feature request related to a problem? Please describe.

a customer reported jobs failing with the following error:

Caused by: java.lang.IllegalArgumentException: requirement failed: INT96 column contains one or more values that can overflow and will result in data corruption. Please set spark.rapids.sql.format.parquet.writer.int96.enabled to false so we can fallback on CPU for writing parquet but still take advantage of parquet read on the GPU.

Spark currently defaults spark.sql.parquet.outputTimestampType to INT96 so I think many users just end up here. My guess is the timestamp in this case is bogus but its still an interruption to their current pipeline.

Investigate if we can add support to this in the plugin and cudf.

[revans2]

The only way I can think of to make this work is to have CUDF support us passing a DECIMAL128 with a scale of 0 to them holding the properly scaled value in it.

[jlowe]

There might be another way to support this that doesn't involve sending the full 96 bits or more to libcudf. Spark isn't tracking timestamps as anything more than 64 bits in microseconds, so if we could send libcudf a TIMESTAMP_MICROS column and specify in write call that we want that column encoded as an INT96, maybe libcudf could perform the micros-to-nanos conversion without overflowing the 64-bit value first trying to convert micros to nanos before writing it as an INT96 value. We would need to discuss with the libcudf team if this is an option.

[gerashegalov]

Repro notebook for the issue https://github.com/gerashegalov/rapids-shell/blob/master/src/jupyter/int96.ipynb

[hyperbolic2346]

if we could send libcudf a TIMESTAMP_MICROS column and specify in write call that we want that column encoded as an INT96, maybe libcudf could perform the micros-to-nanos conversion without overflowing the 64-bit value first trying to convert micros to nanos before writing it as an INT96 value.

This should work now out of the box. The code in cudf that does this conversion is here.

[gerashegalov]

fixed by #8824