Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add --pkgimages=existing. #52573

Merged
merged 4 commits into from
Dec 19, 2023
Merged

Add --pkgimages=existing. #52573

merged 4 commits into from
Dec 19, 2023

Conversation

maleadt
Copy link
Member

@maleadt maleadt commented Dec 18, 2023

Equivalent of #50586; implements #51474.

With --pkgimages=existing, it's possible to disable the (often slow) generation of package images, without losing the ability to use existing ones. That's important now that we're moving more and more packages outside of the system image, e.g., running with --pkgimages=no otherwise takes close to 30s here before the Pkg REPL is usable.

The main motivation for this is PkgEval, where generating package images is not very useful, yet disabling generation of them makes each job (which requires Pkg to drive the test process) take a significantly longer time.

For example, --pkgimages=yes vs no:

❯ JULIA_DEBUG=loading ./julia --project=Example.jl --pkgimages=yes
# no precompilation of REPL.jl
┌ Debug: Loading object cache file /Users/tim/Julia/src/julia/build/dev/usr/share/julia/compiled/v1.11/REPL/u0gqU_XmENM.dylib for REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb]
└ @ Base loading.jl:1116

julia> using Example
# short time precompiling + pkgimg generation for Example.jl
┌ Debug: Loading object cache file /Users/tim/.julia/compiled/v1.11/Example/lLvWP_tJaso.dylib for Example [7876af07-990d-54b4-ab0e-23690620f79a]
└ @ Base loading.jl:1116
❯ JULIA_DEBUG=loading ./julia --project=Example.jl --pkgimages=no
┌ Debug: Rejecting cache file /Users/tim/Julia/src/julia/build/dev/usr/share/julia/compiled/v1.11/REPL/u0gqU_XmENM.ji for REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb] since the flags are mismatched
│   current session: use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
└ @ Base loading.jl:3289
# long time precompiling REPL.jl
┌ Debug: Loading cache file /Users/tim/.julia/compiled/v1.11/REPL/u0gqU_CWvWI.ji for REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb]
└ @ Base loading.jl:1119

julia> using Example
# short time precompiling Example
┌ Debug: Loading cache file /Users/tim/.julia/compiled/v1.11/Example/lLvWP_CWvWI.ji for Example [7876af07-990d-54b4-ab0e-23690620f79a]
└ @ Base loading.jl:1119

With the new --pkgimages=existing:

❯ JULIA_DEBUG=loading ./julia --project=Example.jl --pkgimages=existing
# no precompilation of REPL.jl
┌ Debug: Loading object cache file /Users/tim/Julia/src/julia/build/dev/usr/share/julia/compiled/v1.11/REPL/u0gqU_XmENM.dylib for REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb]
└ @ Base loading.jl:1116

julia> using Example
# short time precompiling Example
┌ Debug: Loading cache file /Users/tim/.julia/compiled/v1.11/Example/lLvWP_CWvWI.ji for Example [7876af07-990d-54b4-ab0e-23690620f79a]
└ @ Base loading.jl:1119

@maleadt maleadt added the packages Package management and loading label Dec 18, 2023
Copy link
Member

@vchuravy vchuravy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, The difference to --compile-modules=existing is that we would still create .ji files?

Would be nice to update docs (and maybe news) as well.

@maleadt
Copy link
Member Author

maleadt commented Dec 18, 2023

The difference to --compile-modules=existing is that we would still create .ji files?

Yes, AFAICT creating those doesn't cause a significant overhead, so can be left enabled on PkgEval (i.e., it's just adding some additional granularity).

I'll look into creating docs and updating news once this passes initial review.

@vchuravy
Copy link
Member

Might make sense to add a CLI test as well for the flag parsing.

Prevents generating new pkgimages without disabling loading of existing ones.
@maleadt maleadt force-pushed the tb/pkgimages_existing branch from 496ad2a to a7d1e3a Compare December 19, 2023 09:37
@KristofferC KristofferC merged commit 91d87c6 into master Dec 19, 2023
6 of 7 checks passed
@KristofferC KristofferC deleted the tb/pkgimages_existing branch December 19, 2023 11:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
packages Package management and loading
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants