Floats print with incorrect rounding in compact context

[rdeits]

Similar to #33185 but present in 1.4.2 and 1.5-beta1.

Originally from: https://discourse.julialang.org/t/incorrect-display-of-array-float64-2-numbers/41200/3

MWE:

julia> sprint(show, -498796.2749933266, context=:compact=>true)
"-498796.0"

The .0 is wrong--it ought to be .3.

[rdeits]

This appears to be an edge case for floats with exactly 6 digits before the decimal place:

julia> sprint(show, 12.345678, context=:compact=>true)
"12.3457"

julia> sprint(show, 123.45678, context=:compact=>true)
"123.457"

julia> sprint(show, 1234.5678, context=:compact=>true)
"1234.57"

julia> sprint(show, 12345.678, context=:compact=>true)
"12345.7"

julia> sprint(show, 123456.78, context=:compact=>true)  # Incorrect
"123457.0"

julia> sprint(show, 1234567.8, context=:compact=>true)
"1.23457e6"

[rdeits]

I guess I see what's happening here: the value is correctly rounded to 123457, but because it's still a Float64, it gets printed as 123457.0. That gives the impression that the .0 is a valid significant digit, but it's actually just signifying the type of the result.

[JeffBezanson]

Ah, I guess it should be printed as -498796.?

[rdeits]

Or perhaps the compact display logic should avoid rounding to 0 digits? It doesn't save any space to do so, since the .0 will be appended anyway.

[vtjnash]

I seem to recall fixing this in the old Grisu code (#6608 and #9072), so that in v0.6 we had this output:

julia> [-498796.2749933266 1]
1×2 Array{Float64,2}:
 -4.98796e5  1.0

(commit 1890e57)

[vtjnash]

Ah, we even had a test for this (5d2a0ec), but it was deleted when switching from Grisu->Ryu (869090b)

[JeffBezanson]

Bump @quinnj --- I doubt there is time for this in 1.5 but this seems pretty bad.

[JeffreySarnoff]

rather than the current incorrect

julia> sprint(show, 12345.678, context=:compact=>true)
"12345.7"

julia> sprint(show, 123456.78, context=:compact=>true)  # Incorrect
"123457.0"

julia> sprint(show, 1234567.8, context=:compact=>true)
"1.23457e6"

why not re-adopt what @vtjnash had done

julia> sprint(show, 12345.678, context=:compact=>true)
"12345.7"

julia> sprint(show, 123456.78, context=:compact=>true)  # Correct and nicer than "123457."
"1.23457e5"

julia> sprint(show, 1234567.8, context=:compact=>true)
"1.23457e6"

[JeffreySarnoff]

to obtain

julia> sprint(show, 12345.678, context=:compact=>true)
"12345.7"

julia> sprint(show, 123456.78, context=:compact=>true)
"1.23457e5"

julia> sprint(show, 1234567.8, context=:compact=>true)
"1.23457e6"

fix Line 333 of Ryu/shortest.jl

    if -4 < pt <= (precision == -1 ? (T == Float16 ? 3 : 6) : precision)

by replacing <= with <

    if -4 < pt < (precision == -1 ? (T == Float16 ? 3 : 6) : precision)

[JeffreySarnoff]

Well, it appears there are other impinging considerations. So I closed the PR. Someone more familiar with the other uses of that line ...

[quinnj]

Fix is up: #36819

[quinnj]

Fixed via #36819