Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Tests test_window_aggs_for_rows_collect_list fails when there are null values in columns. #1638

Closed
firestarman opened this issue Feb 1, 2021 · 5 comments · Fixed by #1830
Closed

[BUG] Tests test_window_aggs_for_rows_collect_list fails when there are null values in columns. #1638

firestarman opened this issue Feb 1, 2021 · 5 comments · Fixed by #1830
Assignees
@firestarman
Labels
bug Something isn't working

Comments

@firestarman
Copy link
Collaborator

firestarman commented Feb 1, 2021 •
edited
Loading

Describe the bug
Spark will drop nulls when collecting, but seems GPU does not yet, so exceptions as below come up.

E      Caused by: java.lang.AssertionError:  value at 350 is null
E          	at ai.rapids.cudf.HostColumnVectorCore.assertsForGet(HostColumnVectorCore.java:228)
E         	at ai.rapids.cudf.HostColumnVectorCore.getInt(HostColumnVectorCore.java:254)
E         	at com.nvidia.spark.rapids.RapidsHostColumnVectorCore.getInt(RapidsHostColumnVectorCore.java:109)
E         	at org.apache.spark.sql.vectorized.ColumnarArray.getInt(ColumnarArray.java:128)

Steps/Code to reproduce bug
Run the tests test_window_aggs_for_rows_collect_list with latest cudf and rapids jars.

Expected behavior
Tests should pass.
@firestarman firestarman added bug Something isn't working ? - Needs Triage Need team to review and classify labels Feb 1, 2021
@firestarman firestarman self-assigned this Feb 1, 2021
@firestarman firestarman mentioned this issue Feb 1, 2021
Merged
1 task
@firestarman
Copy link
Collaborator Author

This should be fixed by rapidsai/cudf#7264

@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Feb 9, 2021
@firestarman
Copy link
Collaborator Author

Suppose it will be fixed when updating cudf to 0.19 .

@firestarman
Copy link
Collaborator Author

PR rapidsai/cudf#7264 is not enough, needing support in JNI.

@firestarman
Copy link
Collaborator Author

firestarman commented Feb 26, 2021 •
edited
Loading

Filed the PR rapidsai/cudf#7457 to add support in cuDF JVM .

@firestarman firestarman mentioned this issue Mar 1, 2021
Merged
@firestarman
Copy link
Collaborator Author

firestarman commented Mar 1, 2021 •
edited
Loading

Filed PR #1830 to fix this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@firestarman firestarman

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Enable the tests for collect over window. firestarman/spark-rapids
2 participants
@sameerz @firestarman