Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Add in float, double, timestamp, and date support to murmur3 #1914

Closed
revans2 opened this issue Mar 11, 2021 · 0 comments · Fixed by #2017
Closed

[BUG] Add in float, double, timestamp, and date support to murmur3 #1914

revans2 opened this issue Mar 11, 2021 · 0 comments · Fixed by #2017
Assignees
Labels
bug Something isn't working

Comments

@revans2
Copy link
Collaborator

revans2 commented Mar 11, 2021

I am filing this as a bug because it is technically a latent bug/feature regression that was found as a part of #1910
With better testing we found that double and float types did not do the correct thing for either NaN values or for -0.0. The NaN values have to be normalized to a single NaN but the -0.0 should not be. In #1910 I updated the murmur3 code to normalize the NaN values, but the hash code itself for what ever reason was treating the -0.0 as 0.0 and outputting the same hash for both. The Spark code extracts the bit representation of these values as either a long or an int and then hashes those. In cudf 0.5 this is now possible with bitCast and we should use it to hopefully get around the issues with float and double. Because hash partitioning now also depends on murmur3 it and its tests also need to be updated.

@revans2 revans2 added bug Something isn't working ? - Needs Triage Need team to review and classify labels Mar 11, 2021
@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Mar 16, 2021
@revans2 revans2 changed the title [BUG] Add in float and double support to murmur3 [BUG] Add in float, double, timestamp, and date support to murmur3 Mar 22, 2021
@revans2 revans2 added this to the Mar 15 - March 26 milestone Mar 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants