ArviZ - CairoMakie interaction causes Turing compile to fail

[daeh]

Using ArviZ and CairoMakie in the same environment causes Turing compilation to fail.

The issue only arrises with CairoMakie versions >= 0.11.12

Julia release 1.10.4+0.aarch64.apple.darwin14
MacOS 14.5, M2 pro

using Turing

ERROR: LoadError: UndefVarError: `requiresgradient` not defined
Stacktrace:
 [1] getproperty(x::Module, f::Symbol)
   @ Base ./Base.jl:31
 [2] top-level scope
   @ ~/.julia/packages/OptimizationOptimJL/hDX5k/src/OptimizationOptimJL.jl:17
 [3] include
   @ ./Base.jl:495 [inlined]
 [4] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
   @ Base ./loading.jl:2222
 [5] top-level scope
   @ stdin:3
in expression starting at ~/.julia/packages/OptimizationOptimJL/hDX5k/src/OptimizationOptimJL.jl:1
in expression starting at stdin:3
ERROR: LoadError: Failed to precompile OptimizationOptimJL [36348300-93cb-4f02-beb5-3c3902f8871e] to "~/.julia/compiled/v1.10/OptimizationOptimJL/jl_7TD7Gq".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:2468
  [3] compilecache
    @ ./loading.jl:2340 [inlined]
  [4] (::Base.var"#968#969"{Base.PkgId})()
    @ Base ./loading.jl:1974
  [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.10.4+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
  [6] #mkpidlock#6
    @ ~/.julia/juliaup/julia-1.10.4+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
  [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.10.4+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
  [8] #invokelatest#2
    @ ./essentials.jl:894 [inlined]
  [9] invokelatest
    @ ./essentials.jl:889 [inlined]
 [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:2983
 [11] maybe_cachefile_lock
    @ ./loading.jl:2980 [inlined]
 [12] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1970
 [13] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
 [14] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [15] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [16] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1803
 [17] macro expansion
    @ ./loading.jl:1790 [inlined]
 [18] macro expansion
    @ ./lock.jl:267 [inlined]
 [19] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
 [20] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [21] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [22] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
 [23] include(mod::Module, _path::String)
    @ Base ./Base.jl:495
 [24] include(x::String)
    @ Turing ~/.julia/packages/Turing/r3Hmj/src/Turing.jl:1
 [25] top-level scope
    @ ~/.julia/packages/Turing/r3Hmj/src/Turing.jl:55
 [26] include
    @ ./Base.jl:495 [inlined]
 [27] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
    @ Base ./loading.jl:2222
 [28] top-level scope
    @ stdin:3
in expression starting at ~/.julia/packages/Turing/r3Hmj/src/optimisation/Optimisation.jl:1
in expression starting at ~/.julia/packages/Turing/r3Hmj/src/Turing.jl:1
in expression starting at stdin:3
ERROR: LoadError: Failed to precompile Turing [fce5fe82-541a-59a6-adf8-730c64b5f9a0] to "~/.julia/compiled/v1.10/Turing/jl_NDXzvy".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:2468
  [3] compilecache
    @ ./loading.jl:2340 [inlined]
  [4] (::Base.var"#968#969"{Base.PkgId})()
    @ Base ./loading.jl:1974
  [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.10.4+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
  [6] #mkpidlock#6
    @ ~/.julia/juliaup/julia-1.10.4+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
  [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.10.4+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
  [8] #invokelatest#2
    @ ./essentials.jl:894 [inlined]
  [9] invokelatest
    @ ./essentials.jl:889 [inlined]
 [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:2983
 [11] maybe_cachefile_lock
    @ ./loading.jl:2980 [inlined]
 [12] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1970
 [13] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
 [14] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [15] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [16] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1803
 [17] macro expansion
    @ ./loading.jl:1790 [inlined]
 [18] macro expansion
    @ ./lock.jl:267 [inlined]
 [19] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
 [20] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [21] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [22] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746

Failing environment Project.toml

[deps]
ArviZ = "131c737c-5715-5e2e-ad31-c244f01c1dc7"
ArviZPythonPlots = "4a6e88f0-2c8e-11ee-0601-e94153f0eada"
CairoMakie = "13f3f980-e62b-5c42-98c6-ff1f3baf88f0"
Turing = "fce5fe82-541a-59a6-adf8-730c64b5f9a0"

[compat]
CairoMakie = "0.12.0"

Working environment Project.toml

[deps]
ArviZ = "131c737c-5715-5e2e-ad31-c244f01c1dc7"
ArviZPythonPlots = "4a6e88f0-2c8e-11ee-0601-e94153f0eada"
CairoMakie = "13f3f980-e62b-5c42-98c6-ff1f3baf88f0"
Turing = "fce5fe82-541a-59a6-adf8-730c64b5f9a0"

[compat]
CairoMakie = "0.11.11"

The manifests for the failing and working environments:

failing.zip
working.zip

[sethaxen]

The issue is that this environment uses SciMLBase v2.23.1 and OptimizationOptimJL v0.3.2. From the logs, OptimizationOptimJL uses SciMLBase.requiresgradient, which was not added until SciML/SciMLBase.jl#643, which appeared in SciMLBase v2.30.0. So the main issue here is OptimizationOptimJL not setting the correct SciMLBase lower bound. OptimizationOptimJL just reimports SciMLBase from Optimization.jl so doesn't set the bound at all. In this environment you have Optimization v3.24.3, but at least Optimization v3.25.1 is needed.

That being said, it's clear that for this environment there are enough dependency bound conflicts that you can't upgrade Optimization to fix the problem. I'm working on bumping a few compat bounds for ArviZ now, so it's possible when that is finished this will magically be resolved.

[daeh]

Thanks for the thorough explanation! And for your work on this very useful project

[sethaxen]

You're welcome! ArviZ v0.12.0 resolves all outstanding compat issues, and with it your environment seems to precompile just fine. If you run into any other issues, please let me know.

[daeh]

Seems to be working great! Thanks for the fix!