1.11.1: breaking change in extension loading

[aplavin]

Not too often I see packages broken by patch versions of Julia, but it's one of those cases :(
As a specific example, using UnionCollections, Dictionaries, FlexiMaps throws ERROR: LoadError: type Nothing has no field UnionDictionary on this line: https://github.com/JuliaAPlavin/UnionCollections.jl/blob/0000000018ecd297a2e7448671da83c035b3da4c/ext/FlexiMapsDictionariesExt.jl#L8.

Effectively, it does get_extension(MyPackage, :WeakDepA) in the extension for WeakDepA + WeakDepB.
Judging by the documentation, it should work:

• get_extension returns the extension if it's loaded

get_extension(parent::Module, extension::Symbol)
Return the module for extension of parent or return nothing if the extension is not loaded.

• Pkg docs say the following about extension loading:

A package extension is a module in a file (similar to a package) that is automatically loaded when some other set of packages are loaded into the Julia session.

In the specific case with UnionCollections, this is what I expect to happen on that using line:

1. UnionCollections is loaded
2. Dictionaries is loaded
3. UnionCollections -> DictionariesExt is loaded (following Pkg docs)
4. FlexiMaps is loaded
5. UnionCollections -> FlexiMapsDictionariesExt is loaded, it sees DictionariesExt loaded already and successfully imports that (following get_extension docs)

And that's indeed how the observable behavior worked on earlier Julia versions, including 1.11.0. But not on 1.11.1.

[aplavin]

Juliahub search indicates that quite a few packages call get_extension in their extensions, so maybe this bug manifests itself somewhere else as well. Or maybe not, depends on the order I guess, didn't check myself.

[bvdmitri]

Judging by the documentation, it should work:

AFAIK get_extension is internal (its also written in its docstring) and is subject to change and its use is really discouraged since the order load is not really defined.

[KristofferC]

Yeah, this is a bit unfortunate. The issue was that loading extensions in other extensions as was done previously was "unsound" and could easily cause "circular dependencies" leading to bad behavior like:

• Extension cycles are problematic (long precompile times, difficult to understand/resolve) #55557
• Circular dependency detected for Symbolics.jl on Julia 1.11.0-rc2 #55517
• Error precompiling SymbolicsPreallocationToolsExt #53081
• Strange "missing from cache"/"does not support precompilation"/... errors #52132

I didn't see any new regressions from PkgEval from backporting it which is why I assumed it wouldn't have a big impact on the ecosystem but maybe this was hidden by other errors. In hindsight it was probably wrong to backport it.

The use case people want seem to basically be #48734 so we should probably work on fixing that feature. In the mean time the question is if we revert the extension loading change for 1.11.2.

[aplavin]

AFAIK get_extension is internal (its also written in its docstring) and is subject to change

It's mentioned and explained in Pkg docs, https://pkgdocs.julialang.org/v1/creating-packages/. Afaiu, in 1.10 and earlier this means this function is public.

As for loading, the same Pkg docs page writes that

A package extension is a module in a file (similar to a package) that is automatically loaded when some other set of packages are loaded into the Julia session.

Naturally, after loading MyPkg + WeakDepA I expect MyPkg -> WeakDepAExt to be loaded as well.
And actually, seems like it does work – just not during precompilation.

So, I found a workaround: add __precompile__(false) to the extension that does get_extension (@lkdvos – may be relevant for you as well). Don't see any downsides...

I didn't see any new regressions from PkgEval from backporting it which is why I assumed it wouldn't have a big impact on the ecosystem but maybe this was hidden by other errors.

Unfortunately, yet another case of PkgEval not catching an actual issue :( Have to I say, I don't have a good understanding about it in practice. At least UnionCollections tests (example from the first post) tests succeed on 1.10 and 1.11.0, but fail on 1.11.1.

[KristofferC]

I looked in the PkgEval log and UnionCollections is here: https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_hash/0de7b6c_vs_501a4f2/UnionCollections.primary.log. And we can see the extension fails to load but the package tests end up passing anyway...

[aplavin]

Indeed, you are right, I wasn't looking carefully enough! Still not 100% used to seeing "big scary errors" that are practically just warnings :)

[KristofferC]

I guess there is no test that checks that the function in the extension works?

[aplavin]

Indeed, maybe it slipped through the cracks over time... Anyway, I confirm that with __precompile__(false) everything loads fine and functions work afterwards.

[aplavin]

Although, in some scenarios it leads to

│  ┌ Error: Error during loading of extension FlexiMapsDictionariesExt of UnionCollections, use `Base.retry_load_extensions()` to retry.
│  │   exception =
│  │    1-element ExceptionStack:
│  │    Error when precompiling module, potentially caused by a __precompile__(false) declaration in the module.

...

[topolarity]

You might try using a guarded eval instead of __precompile__(false):

# workaround for https://github.com/JuliaLang/julia/issues/56204
function __init__()
    if !Base.generating_output()
        Base.eval(quote
            const DictionariesExt = Base.get_extension(UnionCollections, :DictionariesExt)
            FlexiMaps.mapview(f, d::DictionariesExt.UnionDictionary) = @modify(vs -> mapview(f, vs), DictionariesExt._values(d))

            # TODO disambiguation:
            # FlexiMaps.mapview(p::Union{Symbol,Int,String}, A::UnionDictionary) = mapview(PropertyLens(p), A)
        end)
    end
end

__precompile__(false) assumes that any downstream packages rely on this functionality, so it disables pre-compilation for them too (it's extremely viral). I also think you might have found a bug in it, where it's not being properly applied for extensions.

Using eval in a !Base.generating_output() guard like this has a similar effect of making FlexiMaps.mapview(f, d::DictionariesExt.UnionDictionary) unavailable during pre-compilation, but it avoids disabling pre-compilation for other packages which is generally preferable.

[topolarity]

I think the above workaround works with an appropriate pre-compile guard (the !Base.generating_output())

[andreyz4k]

Another affected package is Transformers.js which relies on Flux's CUDA extension in its own CUDA extension. https://github.com/chengchingwen/Transformers.jl/blob/master/ext/TransformersCUDAExt/TransformersCUDAExt.jl

It compiles just fine but crashes in runtime with ERROR: type Nothing has no field check_use_cuda

[aplavin]

As this 1.11.1 change turned out to be quite breaking (5 packages even mentioned this issue) – any plans to fix/revert it in 1.11.2?
1.11.1 was a patch version that's presumably supposed to have no breaking changes at all...

[KristofferC]

This was very much meant to go into 1.11.0 but due to human error didn't. We could revert it but it will still have to come for 1.12 because the way things were were fundamentally unsound (as can be seen by the linked issues).

[KristofferC]

A workaround would be to use Requires.jl style of conditional loading since that doesn't support precompilation which is the issue here.

[kbarros]

This also seems to break precompilation of the Sunny.jl package.

Sunny defines its plotting extensions in a module PlottingExt that depends on any variant of Makie (e.g. GLMakie or CarioMakie both pull in the Makie package, and therefore trigger loading of PlottingExt). Additionally, Sunny defines an extension module GLMakiePrecompilesExt that precompiles code specific to the GLMakie backend, if it is loaded. Somehow this workflow became broken in 1.11.1, perhaps due to the order in which modules load (GLMakiePrecompilesExt depends on PlottingExt, which maybe requires something like #48734 or #55589 (comment)?)

[aplavin]

We could revert it but it will still have to come for 1.12

Sounds somewhat similar to "it will still have to come for Julia 2.0 anyway, so we are putting it into minor version already" :)

The workaround for this regression is quite nontrivial,

    if !Base.generating_output()
        Base.eval(quote ...

and this only works in limited circumstances IIUC. Also, the solution isn't even explained anywhere aside from this issue. At the very least, could the changelog/NEWS for each version include a section like "known breaking changes" with recommended workarounds?

[topolarity]

If #56368 lands, there will be a proper solution for this in 1.11.2:

• Extensions may now depend on other extensions, if their triggers include all triggers of any
  extension they wish to depend upon (+ at least one other trigger). Ext-to-ext dependencies
  that don't meet this requirement are now blocked from using Base.get_extension during pre-
  compilation, to prevent extension cycles.

For example:

[extensions]
PlottingExt = "Plots"
StatisticsPlottingExt = ["Plots", "Statistics"]

With this, StatisticsPlottingExt is allowed to depend on PlottingExt (it can use get_extension during pre-compilation)

You can even do this across packages. For example, for the Transformers.jl + Flux.jl case above you'd need:

[extensions]
TransformersCUDAExt = ["Flux", "CUDA"]

TransformersCUDAExt triggers on {Flux, CUDA, Transformers} and FluxCUDAExt triggers on {Flux, CUDA}, so get_extension will work

[KristofferC]

Sounds somewhat similar to "it will still have to come for Julia 2.0 anyway, so we are putting it into minor version already"

Not really since the 1.10 behavior is buggy and would leave extensions in a state where their use should be actively discouraged since you at any point (from e.g. someone adding a dependency downstream) could cause your package to break.

[kbarros]

I tested on the backports branch for Julia 1.11.2 using juliaup add pr56228. It appears that this issue will be resolved as described in #56204 (comment). Thank you everyone!

[topolarity]

This should now be resolved in 1.11.2, using the pattern / feature described in #56204 (comment)