display bug near realmin

[stevengj]

julia> realmin(d64"3.4")
I.nfe+2147483244

julia> realmin(Dec64)
Error showing value of type DecFP.Dec64:
ERROR: BoundsError: attempt to access 326-element Array{UInt8,1} at index [327]
Stacktrace:
 [1] ini_dec(::DecFP.Dec64, ::Int64) at /Users/stevenj/.julia/v0.6/DecFP/src/DecFP.jl:167
 [2] show(::IOContext{Base.Terminals.TTYTerminal}, ::DecFP.Dec64) at /Users/stevenj/.julia/v0.6/DecFP/src/DecFP.jl:75
 [3] display(::Base.REPL.REPLDisplay{Base.REPL.LineEditREPL}, ::MIME{Symbol("text/plain")}, ::DecFP.Dec64) at ./REPL.jl:122
 [4] display(::Base.REPL.REPLDisplay{Base.REPL.LineEditREPL}, ::DecFP.Dec64) at ./REPL.jl:125
 [5] display(::DecFP.Dec64) at ./multimedia.jl:194
 [6] eval(::Module, ::Any) at ./boot.jl:235
 [7] print_response(::Base.Terminals.TTYTerminal, ::Any, ::Void, ::Bool, ::Bool, ::Void) at ./REPL.jl:144
 [8] print_response(::Base.REPL.LineEditREPL, ::Any, ::Void, ::Bool, ::Bool) at ./REPL.jl:129
 [9] (::Base.REPL.#do_respond#16{Bool,Base.REPL.##26#36{Base.REPL.LineEditREPL,Base.REPL.REPLHistoryProvider},Base.REPL.LineEditREPL,Base.LineEdit.Prompt})(::Base.LineEdit.MIState, ::Base.AbstractIOBuffer{Array{UInt8,1}}, ::Bool) at ./REPL.jl:646

cc @jmkuhn

[jmkuhn]

This boils down to:

@sprintf("%.6e", Dec64("1e-398"))

I attempt to round with

rounded = round(Dec64("1e-398") * exp10(Dec64(6 - 1 + 398)))

https://github.com/stevengj/DecFP.jl/blob/135efecd3a0340a711a0cbe9afbbc156c6954428/src/DecFP.jl#L226-L234
the exp10 evaluates to Inf so rounded is Inf and _buffer is filled with "+Inf". The search never finds an 'E' in _buffer so the loop attempts to write beyond the end of the DIGITS array.

I don't have a patch for this yet.

[stevengj]

I seem to remember running into #47 here too...

[jmkuhn]

You may be thinking of #54 which ran into #47.