[NDTensors] [BUG] Precompilation issue with 1.10-beta1

[emstoudenmire]

Description of bug

When installing ITensors with Julia 1.10-beta1, there seems to be a conflict or issue between the Base.sort overload in tupletools.jl (line 175) and the sort.jl file defining part of the Julia Base module.

Minimal code demonstrating the bug or unexpected behavior

Minimal runnable code

julia> ]
pkg> add ITensors

Actual output or behavior

NDTensors failed to precompile. The error message and Julia versoininfo() are below.

Output of minimal runnable code

julia> using ITensors
Precompiling ITensors
  ✗ NDTensors
  ✗ ITensors
  0 dependencies successfully precompiled in 6 seconds. 88 already precompiled.

ERROR: The following 1 direct dependency failed to precompile:

ITensors [9136182c-28ba-11e9-034c-db9fb085ebd5]

Failed to precompile ITensors [9136182c-28ba-11e9-034c-db9fb085ebd5] to "/Users/mstoudenmire/.julia/compiled/v1.10/ITensors/jl_BM3z3M".
WARNING: Method definition sort(Tuple) in module Sort at sort.jl:1418 overwritten in module NDTensors at /Users/mstoudenmire/.julia/packages/NDTensors/DkvfK/src/tupletools.jl:175.
ERROR: LoadError: Method overwriting is not permitted during Module precompile.
Stacktrace:
 [1] top-level scope
   @ ~/.julia/packages/NDTensors/DkvfK/src/tupletools.jl:170
 [2] include(mod::Module, _path::String)
   @ Base ./Base.jl:489
 [3] include(x::String)
   @ NDTensors ~/.julia/packages/NDTensors/DkvfK/src/NDTensors.jl:1
 [4] top-level scope
   @ ~/.julia/packages/NDTensors/DkvfK/src/NDTensors.jl:46
 [5] include
   @ Base ./Base.jl:489 [inlined]
 [6] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{…}, dl_load_path::Vector{…}, load_path::Vector{…}, concrete_deps::Vector{…}, source::String)
   @ Base ./loading.jl:2216
 [7] top-level scope
   @ stdin:3
Some type information was truncated. Use `show(err)` to see complete types.
in expression starting at /Users/mstoudenmire/.julia/packages/NDTensors/DkvfK/src/tupletools.jl:170
in expression starting at /Users/mstoudenmire/.julia/packages/NDTensors/DkvfK/src/NDTensors.jl:1
in expression starting at stdin:3
ERROR: LoadError: Failed to precompile NDTensors [23ae76d9-e61a-49c4-8f12-3f1a16adf9cf] to "/Users/mstoudenmire/.julia/compiled/v1.10/NDTensors/jl_qH5r0q".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:2468
  [3] compilecache
    @ Base ./loading.jl:2334 [inlined]
  [4] (::Base.var"#967#968"{Base.PkgId})()
    @ Base ./loading.jl:1968
  [5] mkpidlock(f::Base.var"#967#968"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:92
  [6] #mkpidlock#6
    @ FileWatching.Pidfile /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:87 [inlined]
  [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:110
  [8] #invokelatest#2
    @ Base ./essentials.jl:889 [inlined]
  [9] invokelatest
    @ Base ./essentials.jl:884 [inlined]
 [10] maybe_cachefile_lock(f::Base.var"#967#968"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:2898
 [11] maybe_cachefile_lock
    @ Base ./loading.jl:2895 [inlined]
 [12] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1964
 [13] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1806
 [14] #invoke_in_world#3
    @ Base ./essentials.jl:921 [inlined]
 [15] invoke_in_world
    @ Base ./essentials.jl:918 [inlined]
 [16] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1797
 [17] macro expansion
    @ Base ./loading.jl:1784 [inlined]
 [18] macro expansion
    @ Base ./lock.jl:267 [inlined]
 [19] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1747
 [20] #invoke_in_world#3
    @ Base ./essentials.jl:921 [inlined]
 [21] invoke_in_world
    @ Base ./essentials.jl:918 [inlined]
 [22] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1740
 [23] include
    @ Base ./Base.jl:489 [inlined]
 [24] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{…}, dl_load_path::Vector{…}, load_path::Vector{…}, concrete_deps::Vector{…}, source::Nothing)
    @ Base ./loading.jl:2216
 [25] top-level scope
    @ stdin:3
Some type information was truncated. Use `show(err)` to see complete types.
in expression starting at /Users/mstoudenmire/.julia/packages/ITensors/N9rhA/src/ITensors.jl:1
in expression starting at stdin:3

Stacktrace:
  [1] pkgerror(msg::String)
    @ Pkg.Types /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Pkg/src/Types.jl:69
  [2] precompile(ctx::Pkg.Types.Context, pkgs::Vector{…}; internal_call::Bool, strict::Bool, warn_loaded::Bool, already_instantiated::Bool, timing::Bool, _from_loading::Bool, kwargs::@Kwargs{…})
    @ Pkg.API /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Pkg/src/API.jl:1577
  [3] precompile(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::@Kwargs{_from_loading::Bool})
    @ Pkg.API /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Pkg/src/API.jl:159
  [4] precompile
    @ Pkg.API /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Pkg/src/API.jl:147 [inlined]
  [5] #precompile#114
    @ Pkg.API /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Pkg/src/API.jl:146 [inlined]
  [6] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1957
  [7] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1806
  [8] #invoke_in_world#3
    @ Base ./essentials.jl:921 [inlined]
  [9] invoke_in_world
    @ Base ./essentials.jl:918 [inlined]
 [10] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1797
 [11] macro expansion
    @ Base ./loading.jl:1784 [inlined]
 [12] macro expansion
    @ Base ./lock.jl:267 [inlined]
 [13] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1747
 [14] #invoke_in_world#3
    @ Base ./essentials.jl:921 [inlined]
 [15] invoke_in_world
    @ Base ./essentials.jl:918 [inlined]
 [16] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1740
Some type information was truncated. Use `show(err)` to see complete types.

Version information

• Output from versioninfo():

julia> versioninfo()
Julia Version 1.10.0-beta1
Commit 6616549950e (2023-07-25 17:43 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin22.4.0)
  CPU: 10 × Apple M1 Max
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, westmere)
  Threads: 1 on 10 virtual cores
Environment:
  JULIA_DIR = /Applications/Julia-1.10.app/Contents/Resources/julia

[kmp5VT]

I tried doing this

if VERSION < v"1.10"
  Base.sort(t::Tuple; lt=isless, by=identity, rev::Bool=false) = _sort(t, lt, by, rev)
end

in /ITensors/NDTensors/src/tupletools.jl and compilation still fails but if I comment out the line it compiles fine. Maybe @mtfishman has a solution?

[mtfishman]

That was my first thought for what to try, but I also see that doesn't work. I also tried wrapping it with:

@static if VERSION < v"1.10"
  # ...
end

which can be a bit safer (i.e. it helps with ignoring code that has syntax issues in different versions) but that didn't work either. It seems like the analysis precompilation is doing on the NDTensors package is stricter than the analysis that would be done when loading the package, since that code should get ignored in Julia v1.10.

We probably shouldn't be hijacking the Base.sort method like that anyway, which is considered type piracy. We could make our own NDTensors.sort method. Alternatively, we can have this handled through Compat.jl, I raised an issue about it: JuliaLang/Compat.jl#803 and I'll start making a PR for that. I'll also raise an issue about this behavior of precompilation, since ideally wrapping in @static if VERSION < v"1.10" should work.

[mtfishman]

Will be fixed by #1168. It turns out we need to be more specific with the version constraint, i.e. use if VERSION < v"1.10.0-DEV.1404" instead of if VERSION < v"1.10", since:

julia> v"1.10.0-DEV.1404" < v"1.10"
true