Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Decimal 128 Support: MurMur3 and HashPartitioning support. #3878

Closed
revans2 opened this issue Oct 21, 2021 · 1 comment
Closed

[FEA] Decimal 128 Support: MurMur3 and HashPartitioning support. #3878

revans2 opened this issue Oct 21, 2021 · 1 comment
Assignees
@rwlee
Labels
cudf_dependency An issue or PR with this label depends on a new feature in cudf P1 Nice to have for release task Work required that improves the product but is not user facing
Milestone
Jan 10 - Jan 28

Comments

@revans2
Copy link
Collaborator

revans2 commented Oct 21, 2021

Spark hashes Decimal 128 values very differently from decimal 64 and below. The existing MurMur3 hashing in CUDF is not correct for Spark. Once it is fixed we need to update the tests and make sure it is working. Note that there are a number of integration tests in hash aggregate around distinct counts and such that cannot include decimal 128 values in them because this functionality is missing and the shuffle falls back to the CPU. As a part of this it would be nice to enable those tests, or have a follow on issue to enable them.
@revans2 revans2 added feature request New feature or request ? - Needs Triage Need team to review and classify task Work required that improves the product but is not user facing and removed feature request New feature or request labels Oct 21, 2021
This was referenced Oct 21, 2021
Closed
Closed
@Salonijain27 Salonijain27 added P1 Nice to have for release and removed ? - Needs Triage Need team to review and classify labels Oct 26, 2021
@Salonijain27 Salonijain27 added this to the Nov 1 - Nov 12 milestone Oct 31, 2021
@rwlee rwlee mentioned this issue Dec 16, 2021
Merged
rapids-bot bot pushed a commit to rapidsai/cudf that referenced this issue Jan 20, 2022
@rwlee
Spark Decimal128 hashing (#9919)
c00f42b 
cudf work for NVIDIA/spark-rapids#3878

Shortens the hashed data by removing preceding zero values -- ensuring the leave a sign bit -- and flipping the endianness before hashing the value.

Authors:
  - Ryan Lee (https://github.com/rwlee)
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - Nghia Truong (https://github.com/ttnghia)
  - Bradley Dice (https://github.com/bdice)

URL: #9919
@sameerz sameerz added the cudf_dependency An issue or PR with this label depends on a new feature in cudf label Jan 20, 2022
@nartal1
Copy link
Collaborator

nartal1 commented Jan 24, 2022

#4470 adds decimal-128 support for Murmur3 and HashPartitioning. Closing.

@nartal1 nartal1 closed this as completed Jan 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rwlee rwlee

Labels
cudf_dependency An issue or PR with this label depends on a new feature in cudf P1 Nice to have for release task Work required that improves the product but is not user facing
Projects
None yet
Milestone
Jan 10 - Jan 28
Development

No branches or pull requests
5 participants
@revans2 @sameerz @rwlee @Salonijain27 @nartal1