Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Skip fastparquet timestamp tests for non-UTC timezones.
Fixes NVIDIA#9776. The tests in `fastparquet_compatibility_test.py` check for compatibility between Apache Spark, the Spark RAPIDS plugin, and fastparquet. In particular: 1. `test_reading_file_written_by_spark_cpu` checks if timestamp columns written with Apache Spark are read similarly with fastparquet and the plugin. 2. `test_reading_file_written_with_gpu` checks if timestamps written with the plugin are read the same on Apache Spark and fastparquet. If the timezone is not set to "UTC", and the system timezone isn't "UTC" either, the plugin falls back to CPU for read/write of Parquet timestamp columns. This would cause the above tests not to run: the plugin can neither read nor write timestamps on GPU. Further, fastparquet seems to interpret timestamps written from Spark as being in "UTC", regardless of the timezone settings. So on non-UTC timezones, Apache Spark and fastparquet get different results for the same input. For the two reasons above, it is best to only run the three-way timestamp comparison tests in setups with "UTC" timezone. This commit skips the timestamp tests described above, when a non-UTC timezone is detected. Signed-off-by: MithunR <[email protected]>
- Loading branch information