You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am filing this as a bug because it is technically a latent bug/feature regression that was found as a part of #1910
With better testing we found that double and float types did not do the correct thing for either NaN values or for -0.0. The NaN values have to be normalized to a single NaN but the -0.0 should not be. In #1910 I updated the murmur3 code to normalize the NaN values, but the hash code itself for what ever reason was treating the -0.0 as 0.0 and outputting the same hash for both. The Spark code extracts the bit representation of these values as either a long or an int and then hashes those. In cudf 0.5 this is now possible with bitCast and we should use it to hopefully get around the issues with float and double. Because hash partitioning now also depends on murmur3 it and its tests also need to be updated.
The text was updated successfully, but these errors were encountered:
revans2
changed the title
[BUG] Add in float and double support to murmur3
[BUG] Add in float, double, timestamp, and date support to murmur3
Mar 22, 2021
I am filing this as a bug because it is technically a latent bug/feature regression that was found as a part of #1910
With better testing we found that double and float types did not do the correct thing for either
NaN
values or for-0.0
. TheNaN
values have to be normalized to a singleNaN
but the-0.0
should not be. In #1910 I updated the murmur3 code to normalize theNaN
values, but the hash code itself for what ever reason was treating the-0.0
as0.0
and outputting the same hash for both. The Spark code extracts the bit representation of these values as either a long or an int and then hashes those. In cudf 0.5 this is now possible withbitCast
and we should use it to hopefully get around the issues with float and double. Because hash partitioning now also depends on murmur3 it and its tests also need to be updated.The text was updated successfully, but these errors were encountered: