You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
gerashegalov
changed the title
[BUG] test_cast_string_date_valid_format fails on DBR 10.4 (321db) CI
[BUG] test_cast_string_date_valid_format fails
Nov 18, 2023
gerashegalov
changed the title
[BUG] test_cast_string_date_valid_format fails
[BUG] test_cast_string_date_valid_format fails on DATAGEN_SEED=1700250017
Nov 18, 2023
The problem is triggered when the string generator emits a value that has only zeros for the year but also has a valid month and day ordinal. In this case, the offending string was 0000-2-05. Spark handles this properly in the JVM, but when pyspark tries to marshal the date integral value into a Python datetime it throws an error since datetime doesn't support dates before year 1.
Describe the bug
test_cast_string_date_valid_format fails
Local Repro
SPARK_HOME=~/dist/spark-3.3.0-bin-hadoop3 \ SPARK_RAPIDS_TEST_DATAGEN_SEED=1700250017 \ ./integration_tests/run_pyspark_from_build.sh -k test_cast_string_date_valid_format
Environment details (please complete the following information)
AWS DBR 10.4
Additional context
N/A
The text was updated successfully, but these errors were encountered: