Pandas UDF: Sort the data before computing the sum. #1810
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
To make sure the results returned from the UDF for CPU and GPU are the same. The input 'to_process'(orignal type is integral) of the UDF has the same numbers but in different orders for CPU and GPU, then the sum will be different when, 1) 'to_process' contains null values. (Then Pandas will cast the 'to_process' to float64) 2) Some of the numbers are big enough and will lose precision after being casted to float64. Signed-off-by: Firestarman <[email protected]>
|
build |
revans2
approved these changes
Feb 25, 2021
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
) Signed-off-by: Firestarman <[email protected]>
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
) Signed-off-by: Firestarman <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is to fix the issues #757 and #740 by sorting the data before computing the
sumto make sure the results returned from the UDF for CPU and GPU are always the same.The input
to_process(orignal type is integral) of the UDF has the same numbers but in different orders for CPU and GPU, then the sum will be different when,to_processcontains null values (Then Pandas will cast theto_processtofloat64), andfloat64.Seems it is not easy to sort the data in one group by Spark for the test
test_group_aggregate_udf, so the PR chooses to sort the input data in the UDF.For more details please go to #740 (comment)
closes #740
closes #757
Signed-off-by: Firestarman [email protected]