Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fallback to CPU for mod only for DB11.3 [Databricks] #7609

Merged
merged 8 commits into from
Feb 1, 2023
Merged

Fallback to CPU for mod only for DB11.3 [Databricks] #7609

merged 8 commits into from
Feb 1, 2023

Conversation

razajafri
Copy link
Collaborator

This PR makes change to the DB11.3 shim to disallow GpuRemainder to run on the GPU for DecimalTypes and skips the tests for testing mod on mixed decimal types

fixes #7595

Signed-off-by: Raza Jafri [email protected]

@sameerz sameerz added the bug Something isn't working label Jan 28, 2023
revans2
revans2 reviewed Jan 30, 2023
View reviewed changes
integration_tests/src/main/python/arithmetic_ops_test.py Outdated
@@ -383,6 +383,7 @@ def test_mod_pmod_by_zero_not_ansi(data_gen):
@pytest.mark.parametrize('rhs', [byte_gen, short_gen, int_gen, long_gen, DecimalGen(6, 3),
DecimalGen(10, -2), DecimalGen(15, 3), DecimalGen(30, 12), DecimalGen(3, -3),
DecimalGen(27, 7), DecimalGen(20, -3)], ids=idfn)
@pytest.mark.skipif(is_databricks113_or_later(), reason='https://github.com/NVIDIA/spark-rapids/issues/7595')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be great if we could at least have a test that verifies that we fall back to the CPU on decimal values for this version of spark.

Even better if we check that it still works for other mixed data types, but I think that is minor if we have mod in general covered for the other non-mixed types.

integration_tests/src/main/python/arithmetic_ops_test.py Outdated
@@ -383,6 +383,7 @@ def test_mod_pmod_by_zero_not_ansi(data_gen):
@pytest.mark.parametrize('rhs', [byte_gen, short_gen, int_gen, long_gen, DecimalGen(6, 3),
DecimalGen(10, -2), DecimalGen(15, 3), DecimalGen(30, 12), DecimalGen(3, -3),
DecimalGen(27, 7), DecimalGen(20, -3)], ids=idfn)
@pytest.mark.skipif(is_databricks113_or_later(), reason='https://github.com/NVIDIA/spark-rapids/issues/7595')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't this check need to also include Spark 3.4?

@razajafri
Copy link
Collaborator Author

build

revans2
revans2 previously approved these changes Jan 31, 2023
View reviewed changes
@revans2
Copy link
Collaborator

revans2 commented Jan 31, 2023 

[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod[Decimal(7,3)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod[Decimal(12,2)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod[Decimal(18,0)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod[Decimal(20,2)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod[Decimal(30,2)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod[Decimal(36,5)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod[Decimal(38,10)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod[Decimal(38,0)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod[Decimal(7,7)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod[Decimal(7,-3)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod[Decimal(36,-5)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod[Decimal(38,-10)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[a % (cast(0 as {}))-Decimal(7,3)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[a % (cast(0 as {}))-Decimal(12,2)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[a % (cast(0 as {}))-Decimal(18,0)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[a % (cast(0 as {}))-Decimal(20,2)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[a % (cast(0 as {}))-Decimal(30,2)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero[a % (cast(0 as {}))-Decimal(36,5)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero_not_ansi[Decimal(7,3)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero_not_ansi[Decimal(12,2)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero_not_ansi[Decimal(18,0)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero_not_ansi[Decimal(20,2)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero_not_ansi[Decimal(30,2)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero_not_ansi[Decimal(36,5)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.716Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero_not_ansi[Decimal(37,0)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.717Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero_not_ansi[Decimal(37,10)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...
[2023-01-31T03:03:50.717Z] FAILED ../../src/main/python/arithmetic_ops_test.py::test_mod_pmod_by_zero_not_ansi[Decimal(7,7)] - pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columna...

Please add mod to the allow list for your fallback test and you need to take a look at test_mod_pmod_by_zero_not_ansi for all of the decimal values.

@razajafri razajafri self-assigned this Jan 31, 2023
revans2
revans2 previously approved these changes Jan 31, 2023
View reviewed changes
@razajafri
Copy link
Collaborator Author

pushed update but need to run sanity tests on db

@revans2
Copy link
Collaborator

revans2 commented Jan 31, 2023

Can you also fix the style problems?

 error file=/home/runner/work/spark-rapids/spark-rapids/sql-plugin/src/main/340+-and-330db/scala/com/nvidia/spark/rapids/shims/DecimalArithmeticOverrides.scala message=File line length exceeds 100 characters line=99
error file=/home/runner/work/spark-rapids/spark-rapids/sql-plugin/src/main/340+-and-330db/scala/com/nvidia/spark/rapids/shims/DecimalArithmeticOverrides.scala message=File line length exceeds 100 characters line=100

@razajafri
Copy link
Collaborator Author

Can you also fix the style problems?

 error file=/home/runner/work/spark-rapids/spark-rapids/sql-plugin/src/main/340+-and-330db/scala/com/nvidia/spark/rapids/shims/DecimalArithmeticOverrides.scala message=File line length exceeds 100 characters line=99
error file=/home/runner/work/spark-rapids/spark-rapids/sql-plugin/src/main/340+-and-330db/scala/com/nvidia/spark/rapids/shims/DecimalArithmeticOverrides.scala message=File line length exceeds 100 characters line=100
``

I have fixed this as well.

@razajafri
Copy link
Collaborator Author

build

@sameerz sameerz merged commit 1739dfd into NVIDIA:branch-23.02 Feb 1, 2023
@razajafri razajafri deleted the SP-7595-test-mod-fallback branch February 1, 2023 07:26
@razajafri razajafri mentioned this pull request Apr 20, 2023
Closed
@NVnavkumar NVnavkumar mentioned this pull request May 16, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@revans2 revans2 revans2 approved these changes

Assignees

@razajafri razajafri

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[BUG] test_mod_mixed decimal test fails on 330db (Databricks 11.3) and TBD 340
3 participants
@razajafri @revans2 @sameerz