[BUG] Incorrect values when parsing dates from timestamps stored in CSV files

[andygrove]

Describe the bug
If I specify a schema with a DateType when reading from a CSV file containing timestamps, I get corrupt data when the plugin is enabled.

Steps/Code to reproduce bug

Create tests/src/test/resources/timestamps.csv CSV file:

2019-01-03T12:34:56.123456,1
2019-01-03T12:34:56.123456,1
2019-01-03T12:34:56.123456,1
2019-01-05T12:34:56.123456,2
2019-01-05T12:34:56.123456,3
2019-01-06T12:34:56.123456,6

Add this method to tests/src/test/scala/com/nvidia/spark/rapids/SparkQueryCompareTestSuite.scala:

  def timestampsAsDatesCsvDf= {
    fromCsvDf("timestamps.csv", StructType(Array(
      StructField("dates", DateType, false),
      StructField("ints", IntegerType, false)
    )))(_)
  }

Add these tests to tests/src/test/scala/com/nvidia/spark/rapids/CsvScanSuite.scala:

  testSparkResultsAreEqual(
    "Test CSV parse dates",
    datesCsvDf,
    conf=new SparkConf()) {
    df => df.withColumn("next_day", date_add(col("dates"), lit(1)))
  }

  testSparkResultsAreEqual(
    "Test CSV parse timestamps as dates",
    timestampsAsDatesCsvDf,
    conf=new SparkConf()) {
    df => df.withColumn("next_day", date_add(col("dates"), lit(1)))
  }

The first test passes but the second test fails with:

CPU: WrappedArray([2019-01-03,1,2019-01-04], [2019-01-03,1,2019-01-04], [2019-01-03,1,2019-01-04], [2019-01-05,2,2019-01-06], [2019-01-05,3,2019-01-06], [2019-01-06,6,2019-01-07])

GPU: WrappedArray([0718-10-22,1,0718-10-23], [0718-10-22,1,0718-10-23], [0718-10-22,1,0718-10-23], [2280-05-20,2,2280-05-21], [2280-05-20,3,2280-05-21], [3779-09-03,6,3779-09-04])

Expected behavior
The tests should both pass because the output should be the same with or without the plugin enabled.

Environment details (please complete the following information)
Running tests in IDE.

Additional context
N/A

[sameerz]

Please update the documentation at https://github.com/NVIDIA/spark-rapids/blob/branch-0.3/docs/compatibility.md#csv-dates when this is fixed.

[revans2]

I think that this is working now.

scala> spark.read.schema(StructType(Seq(StructField("dates", DateType, false), StructField("ints", IntegerType)))).csv("./test.csv").collect.foreach(System.out.println)
23/08/15 19:39:03 WARN GpuOverrides: 
*Exec <FileSourceScanExec> will run on GPU

[2019-01-03,1]                                                                  
[2019-01-03,1]
[2019-01-03,1]
[2019-01-05,2]
[2019-01-05,3]
[2019-01-06,6]

scala> spark.conf.set("spark.rapids.sql.enabled", false)

scala> spark.read.schema(StructType(Seq(StructField("dates", DateType, false), StructField("ints", IntegerType)))).csv("./test.csv").collect.foreach(System.out.println)
[2019-01-03,1]
[2019-01-03,1]
[2019-01-03,1]
[2019-01-05,2]
[2019-01-05,3]
[2019-01-06,6]

Could we get someone to add a test to verify that it continues to work.