-
Notifications
You must be signed in to change notification settings - Fork 241
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adopt changes from JNI for casting from float to decimal #10917
Conversation
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
The results look really close. The
Related Spark UT still fails, also close:
Performance test to cast 5000000 floats to 10 kinds of decimal types (in ms):
|
Should that case be acceptable as the relative error is just |
I've just updated my JNI code in NVIDIA/spark-rapids-jni#2078, adding some utilities functions by @pmattione-nvidia and it seems to fix all of our failed tests. @thirtiseven please run tests again on your machine. |
It passed Spark UT, personally I think it is good enough for this issue. If set eps = 0 in
Only diffs in last few digits. Some cases in double test:
There some cases like New performance:
|
And again some thoughts about the |
My solution in NVIDIA/spark-rapids-jni#2078 is adopted from the ongoing work of @pmattione-nvidia. Paul is working on even a better solution, so hopefully it will be able to fix the failures above. Let's wait for it. |
Signed-off-by: Nghia Truong <[email protected]>
# Conflicts: # tests/src/test/spark330/scala/org/apache/spark/sql/rapids/utils/RapidsTestSettings.scala
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Alright, since we don't have any better solution in the meantime, I've made NVIDIA/spark-rapids-jni#2078 ready for review. With that:
|
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
This reverts commit c185206. # Conflicts: # tests/src/test/scala/com/nvidia/spark/rapids/CastOpSuite.scala
This reverts commit 93422e8.
Signed-off-by: Nghia Truong <[email protected]>
build |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
This reimplements
castFloatsToDecimal
to just calling the corresponding function fromspark-rapids-jni
, which is designed specifically for casting floating point values to decimal for Spark.Depends on:
Closes #9682, and closes #10809.