Pkg.precompile sometimes doesn't avoid precompilation during load time

[IanButterworth]

There is an edge case in Pkg's precompilation for which we currently don't have a fix. I'm making this issue to make it easier to track given it has been discussed in a few places

One might see

(jl_dDFSry) pkg> precompile

julia> @time (using Plots; display(plot(1:0.1:10, sin.(1:0.1:10))))
[ Info: Precompiling Plots [91a5bcdd-55d7-5caf-9e0b-520d859cae80]

This has improved since julia 1.5 (#1578), but is still known to happen as a result of this kind of sequence:

pkg> add Plots, GR
... # add and fully precompile packages

julia> using Plots

pkg> pin [email protected] # roll back GR
... # precomp will ? for Plots given it's loaded

restart julia

pkg> precompile

julia> using Plots
[ Info: Precompiling Plots [91a5bcdd-55d7-5caf-9e0b-520d859cae80]

The problem is that Base.stale_cachefile can only check whether dependencies of a module have been updated if they are loaded in Main.
Pkg.precompile works around this, given the modules aren't actually loaded into Main, by keeping track itself of whether dependencies are updated.
But if Pkg.precompile isn't able to precompile a package due to it being loaded in Main, and julia is restarted, the new precompile attempt has no idea that dependencies have been recompiled.

In writing this up it occurs to me that the fix could be to add a force_precomp list of packages that's saved to the scratch space, to remember packages that still need precomp (i.e. packages with ?), like the way pkgs_precompile_suspended works.

Previous discussion around this:

• Parallelize Pkg.precompile #2018 (comment)
• precompile misses many packages #1578 (comment)