[REVIEW] Spark Murmur3 hash functionality #7024
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rwlee
added
3 - Ready for Review
Codecov Report
@@ Coverage Diff @@
## branch-0.18 #7024 +/- ##
===============================================
+ Coverage 82.09% 82.11% +0.02%
===============================================
Files 97 97
Lines 16474 16477 +3
===============================================
+ Hits 13524 13530 +6
+ Misses 2950 2947 -3
Continue to review full report at Codecov.
|
revans2
reviewed
Dec 17, 2020
jrhemstad
reviewed
Dec 17, 2020
jrhemstad
reviewed
Dec 17, 2020
jrhemstad
requested changes
Dec 17, 2020
nvdbaranec
requested changes
Dec 18, 2020
jrhemstad
approved these changes
Dec 18, 2020
nvdbaranec
approved these changes
Dec 18, 2020
revans2
approved these changes
Jan 4, 2021
galipremsagar
approved these changes
Jan 4, 2021
harrism
added
6 - Okay to Auto-Merge
and removed
3 - Ready for Review
rapids-bot bot
pushed a commit
that referenced
this pull request
Mar 24, 2021
#7024 added a Spark variant of Murmur3 hashing, but it is inconsistent with Apache Spark's hash calculations in a few areas: - `-0.0` and `0.0` are not treated the same by Apache Spark for floats and doubles - byte and short integral values are upcast to a 32-bit unsigned int (i.e.: zero-filled) before calculating the hash In addition libcudf allows hashing of timestamp columns but the JNI bindings asserted if timestamp columns were passed in, disabling the ability to hash on timestamps directly. Authors: - Jason Lowe (@jlowe) Approvers: - Nghia Truong (@ttnghia) - Jake Hemstad (@jrhemstad) - Alessandro Bellina (@abellina) - MithunR (@mythrocks) - Robert (Bobby) Evans (@revans2) URL: #7672
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #6863
Expands existing murmur3 hashing functionality to match Spark's murmur3 hashing algorithm by modifying tail processing for unaligned bytes and processing booleans as 32bit integers rather than singular bytes.