Upgradable stdlibs

[vchuravy]

A long-term goal for me is to enable upgradable stdlibs. Essentially leveraging pkgimages to move as many stdlibs out of the base system image and allow upgradeability without requiring Julia upgrades.

Currently the only upgradeable stdlib is DelimitedFiles. And maybe soon (tm) Statistics #46501

@KristofferC recently described to me on Slack the steps necessary to make a stdlib upgradeable?

The steps for an upgradable stdlib are:
1. Move the stdlib to an external repo:
2. Remove the stdlib from the sysimage
3. Make the version of the stdlib to the same as the current Julia version
4. Run https://github.com/KristofferC/StdlibRegistryCompatUpdater/blob/d79d66f7734e3d8d71213a4cf6250bea84dafe43/src/StdlibRegistryCompatUpdater.jl#L40 with the stdlib name and UUID to generate a PR to the registry equivalent to JuliaRegistries/General#66039
5. do something like JuliaLang/Pkg.jl#3373

For Step 2. one can look at pkgimage.mk if the stdlib has a pkgbuilder defined for it. (x-ref: #50684)

Would be great if we could crowd-source this process!

Tasks

Beta Give feedback

1. DelimitedFiles.jl remove DelimitedFiles from being an stdlib #45540 put back DelimitedFiles as a non-sysimage "stdlib" #48671
2. Statistics.jl Upgradeable Statistics stdlib #46501
3. Distributed Move Distributed out of Base #51621
4. SparseArrays
5. LazyArtifacts Move LazyArtifacts out of the mono-repo #51549
6. SharedArrays
7. Test Remove Test from the monorepo #51632
8. Pkg.jl Mark Pkg itself as an upgradeable standard library. Pkg.jl#3638
9. LinearAlgebra LinearAlgebra excision from sysimg. LinearAlgebra.jl#1027
10. Random Excise Random from the system image #51432
11. FileWatching Excise FileWatching by moving pidfile to Base #51463
12. Require compat annotations in registry Require [compat] for stdlib dependencies JuliaRegistries/RegistryCI.jl#516
13. ...

How to split a stdlib into a seperate repository

contrib/excise_stdlib.sh Test

[giordano]

I'll point out that it can be very convenient to have upgradable stdlib jlls for Julia dependencies, when they don't break the API/ABI, especially in the case of security fixes. However there are some cases:

• libraries that we shouldn't upgrade because Julia strictly requires a specific version (e.g. llvm)
• libraries that can usually be upgraded somewhat freely and I are needed only at runtime (e.g. mpfr, gmp, openblas (?))
• libraries which are needed to start the Julia processes, so I'm not sure they can be meaningfully upgraded (e.g. zlib)

[PallHaraldsson]

libraries which are needed to start the Julia processes, so I'm not sure they can be meaningfully upgraded (e.g. zlib)

I don't think zlib is such a library, and even if could use system zlib. I have a hard time seeing were it's used, I think only by Libgit2, with is neither a library you can do without:

julia/stdlib/LibGit2/src/error.jl

Line 42 in 0b190b3

Zlib,

I believe I've excised it and Pkg from the sysimage already, it and Pkg should at least not load at startup, and maybe should be upgradable.

[giordano]

I don't think zlib is such a library

Llvm needs zlib, so zlib is very much needed to start Julia.

[vchuravy]

@KristofferC I think we need to move top-down right?

So currently eligible are:

• SparseArrays (maybe @Wimmerer or @ViralBShah want to take that one?)
• LazyArtifacts (@staticfloat?) -- I see that it has a dependency on Pkg so, Mark Pkg itself as an upgradeable standard library. Pkg.jl#3638 might be too soon.
• Test
• SharedArrays

After Pkg & REPL a lot more become eligible.

[ViralBShah]

SparseArrays is already outside of the julia repo, and already not in the system image. So I believe the major work is done. I think the other things, perhaps @KristofferC may be better and quicker with.

[KristofferC]

Yeah, I can do the remaining steps.

[ViralBShah]

The big one to really carry out is LinearAlgebra.

[KronosTheLate]

I just want to chime in and say that the Serialization standard lib is great, until you want to read data in an older julia verion that is written by a newer version. It would be fantastic if this stdlib could become upgradable, so that e.g. RemoteREPL.jl would not require that the Julia version is the same.

[giordano]

How would making the Serialization stdlib upgradable solve the compatibility issues? The fact is that the process of serialisation depends on internal Julia structures which are bound to the specific Julia versions you're using. Making Serialization upgradable is completely orthogonal to that problem.

[KronosTheLate]

I would not want to hijack this issue too much, but I just imagined that if the version of Serialization was not pinned to the Julia version, then someone running 1.9 and someone running 1.6 could both install Serialization 1.x, and know that as long as they both use the same version of Serialization, the communication will work both ways. I guess it should be possible to make Serialization independent of the Julia version, or at least to manually add any specific code needed in order to support older julia versions, perhaps with the help of Compat.

[tecosaur]

Is there much chance of the REPL stdlib excision happening in the near future?

[vchuravy]

Is there much chance of the REPL stdlib excision happening in the near future?

It has been removed from the system image, but not yet moved out of the mono repo

I think we could make it upgradeable, but there might me a dependency edge from PKG to repl that we need to remove (IIRC)