BenchmarkTools breaks with lu! in 1.9-beta2 (MacBook Air, M2)

[ctkelley]

BenchmarkTools is breaking for lu! on 1.9-beta2 (and the nightly)

Here's a MWE.

With 1.8.4

julia> using Random

julia> using BenchmarkTools

julia> Random.seed!(46071);

julia> n=1024;

julia> AH=rand(n,n);

julia> tlu=@belapsed lu($AH);

julia> println("$tlu is ok")
5.90304e-03 is ok

julia> tluh=@belapsed lu!($AH);

julia> println("$tluh is ok")
5.43113e-03 is ok

On 1.9-beta2

julia> using Random

julia> using BenchmarkTools

julia> Random.seed!(46071);

julia> n=1024;

julia> AH=rand(n,n);

julia> tlu=@belapsed lu($AH);

julia> println("$tlu is ok")
5.90304e-03 is ok

julia> tluh=@belapsed lu!($AH);

ERROR: ArgumentError: matrix contains Infs or NaNs
Stacktrace:
  [1] chkfinite
    @ /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/LinearAlgebra/src/lapack.jl:86 [inlined]
  [2] getrf!(A::Matrix{Float64})
    @ LinearAlgebra.LAPACK /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/LinearAlgebra/src/lapack.jl:559
  [3] #lu!#170
    @ /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/LinearAlgebra/src/lu.jl:81 [inlined]
  [4] #lu!#169
    @ /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/LinearAlgebra/src/lu.jl:79 [inlined]
  [5] lu!
    @ /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/LinearAlgebra/src/lu.jl:79 [inlined]
  [6] var"##core#299"(AH#298::Matrix{Float64})
    @ Main ~/.julia/packages/BenchmarkTools/7xSXH/src/execution.jl:489
  [7] var"##sample#300"(::Tuple{Matrix{Float64}}, __params::BenchmarkTools.Parameters)
    @ Main ~/.julia/packages/BenchmarkTools/7xSXH/src/execution.jl:495
  [8] _run(b::BenchmarkTools.Benchmark, p::BenchmarkTools.Parameters; verbose::Bool, pad::String, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ BenchmarkTools ~/.julia/packages/BenchmarkTools/7xSXH/src/execution.jl:105
  [9] _run(b::BenchmarkTools.Benchmark, p::BenchmarkTools.Parameters)
    @ BenchmarkTools ~/.julia/packages/BenchmarkTools/7xSXH/src/execution.jl:92
 [10] #invokelatest#2
    @ ./essentials.jl:816 [inlined]
 [11] invokelatest
    @ ./essentials.jl:813 [inlined]
 [12] #run_result#45
    @ ~/.julia/packages/BenchmarkTools/7xSXH/src/execution.jl:34 [inlined]
 [13] run_result
    @ ~/.julia/packages/BenchmarkTools/7xSXH/src/execution.jl:34 [inlined]
 [14] run(b::BenchmarkTools.Benchmark, p::BenchmarkTools.Parameters; progressid::Nothing, nleaves::Float64, ndone::Float64, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ BenchmarkTools ~/.julia/packages/BenchmarkTools/7xSXH/src/execution.jl:117
 [15] run
    @ ~/.julia/packages/BenchmarkTools/7xSXH/src/execution.jl:117 [inlined]
 [16] run(b::BenchmarkTools.Benchmark)
    @ BenchmarkTools ~/.julia/packages/BenchmarkTools/7xSXH/src/execution.jl:117
 [17] top-level scope
    @ ~/.julia/packages/BenchmarkTools/7xSXH/src/execution.jl:395

julia> println("$tluh is ok")
ERROR: UndefVarError: `tluh` not defined
Stacktrace:
 [1] top-level scope
   @ REPL[14]:1

[KristofferC]

Does the matrix contains infs or nans after you have called lu! on it in a loop? BenchmarkTools doesn't seem relevant here.

[ctkelley]

lu! will change the matrix, but I'd been assuming that

@belapsed lu!($A)

with interpolating A would not. If that is not the case, it is impossible to benchmark any function that overwrites memory.

[ctkelley]

I've fixed the random seed in the MWE, so the matrix should be the same in both cases. 1.8 is fine and 1.9 is not.

[KristofferC]

with interpolating A would not

It will.

it is impossible to benchmark any function that overwrites memory.

You can use setup=(AH=rand(n,n)) + evals=1 to the benchmark macro to provide a fresh matrix for every sample.

[ctkelley]

So why does the example seem to work on 1.8 all the time and fail for 1.9 all the time?

Thanks for the pointer to setup.

[KristofferC]

You are iteratively calling lu! over and over on the same memory, so it doesn't seem surprising that it might blow up at some point. It could be how many iterations the benchmark run.

Why there is a difference, maybe JuliaLang/LinearAlgebra.jl#976.

[ctkelley]

That could be the problem. Anyhow, using setup fixed me up for now. Thanks again.

[ctkelley]

I set the number of BLAS threads in my startup.jl, so that's not the difference