Skip to content

Commit

Permalink
testutils/floatcmp: revert to base-10 normalization in FloatsMatch
Browse files Browse the repository at this point in the history
In cockroachdb#106552 we tried changing float normalization to use base 2 instead
of base 10 (in other words, to use `math.Frexp` instead of our
hand-rolled `normalize`). This appears to have broken a logic test, so
revert back to the pre-existing base 10 normalization.

Fixes: cockroachdb#107461

Release note: None
  • Loading branch information
michae2 committed Jul 24, 2023
1 parent 4618dbf commit 19b51e9
Showing 1 changed file with 16 additions and 2 deletions.
18 changes: 16 additions & 2 deletions pkg/testutils/floatcmp/floatcmp.go
Original file line number Diff line number Diff line change
Expand Up @@ -121,9 +121,23 @@ func FloatsMatch(expectedString, actualString string) (bool, error) {
actual = math.Abs(actual)
// Check that 15 significant digits match. We do so by normalizing the
// numbers and then checking one digit at a time.
//
// normalize converts f to base * 10**power representation where base is in
// [1.0, 10.0) range.
normalize := func(f float64) (base float64, power int) {
for f >= 10 {
f = f / 10
power++
}
for f < 1 {
f *= 10
power--
}
return f, power
}
var expPower, actPower int
expected, expPower = math.Frexp(expected)
actual, actPower = math.Frexp(actual)
expected, expPower = normalize(expected)
actual, actPower = normalize(actual)
if expPower != actPower {
return false, nil
}
Expand Down

0 comments on commit 19b51e9

Please sign in to comment.