Fixes for julia_version. Expand and consolidate julia_version tests. #4151

IanButterworth · 2025-02-09T20:16:36Z

Trying to clear the block for BinaryBuilder updating to a recent julia version.

Expands HistoricalStdlibVersions tests based on needs outlined in Future Pkg compatibility JuliaPackaging/JLLPrefixes.jl#6 thanks to the tests constructed by @mkitti here
Stops installing HistoricalStdlibVersions manually if not installed, so any such tests would not be run via Base.runtests. It's been a pain to manage the manual install for with the latest version, so just test it on Pkg repo CI
Consolidate all julia_version tests into a single file to easier maintenance
Teaches fixups_from_projectfile! to use STDLIBS_BY_VERSION
BinaryBuilderBase.jl tests pass locally Fixes for julia_version. Expand and consolidate julia_version tests. #4151 (comment)

Fixes a few issues

libcxxwrap_julia_jll: Error During Test at /Users/runner/work/Pkg.jl/Pkg.jl/test/historical_stdlib_version.jl:225
  Got exception outside of a @test
  MethodError: no method matching abspath(::Nothing)
  The function `abspath` exists, but no method is defined for this combination of argument types.
  
  Closest candidates are:
    abspath(::String)
     @ Base path.jl:448
    abspath(::AbstractString)
     @ Base path.jl:614
    abspath(::AbstractString, AbstractString...)
     @ Base path.jl:468
  
  Stacktrace:
    [1] fixups_from_projectfile!(env::Pkg.Types.EnvCache)
      @ Pkg.Operations ~/work/Pkg.jl/Pkg.jl/src/Operations.jl:229
    [2] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; allow_autoprecomp::Bool, preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, target::Symbol)
      @ Pkg.Operations ~/work/Pkg.jl/Pkg.jl/src/Operations.jl:1661
    [3] add
      @ ~/work/Pkg.jl/Pkg.jl/src/Operations.jl:1631 [inlined]
    [4] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, target::Symbol, allow_autoprecomp::Bool, kwargs::@Kwargs{julia_version::VersionNumber, io::IOStream})
      @ Pkg.API ~/work/Pkg.jl/Pkg.jl/src/API.jl:317
    [5] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::IOStream, kwargs::@Kwargs{julia_version::VersionNumber})
      @ Pkg.API ~/work/Pkg.jl/Pkg.jl/src/API.jl:164
    [6] add
      @ ~/work/Pkg.jl/Pkg.jl/src/API.jl:151 [inlined]
    [7] add(; name::String, uuid::Nothing, version::Nothing, url::Nothing, rev::Nothing, path::Nothing, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::@Kwargs{julia_version::VersionNumber})
      @ Pkg.API ~/work/Pkg.jl/Pkg.jl/src/API.jl:181
    [8] add
      @ ~/work/Pkg.jl/Pkg.jl/src/API.jl:170 [inlined]
    [9] macro expansion
      @ ~/work/Pkg.jl/Pkg.jl/test/historical_stdlib_version.jl:228 [inlined]

GMP_jll: Test Failed at /Users/runner/work/Pkg.jl/Pkg.jl/test/historical_stdlib_version.jl:242
  Expression: ((Pkg.dependencies())[GMP_jll_UUID]).version === v"6.2.1+6"
   Evaluated: v"6.3.0+2" === v"6.2.1+6"
Stacktrace:
 [1] macro expansion
   @ ~/hostedtoolcache/julia/nightly/aarch64/share/julia/stdlib/v1.13/Test/src/Test.jl:679 [inlined]
 [2] macro expansion
   @ ~/work/Pkg.jl/Pkg.jl/test/historical_stdlib_version.jl:242 [inlined]
 [3] macro expansion
   @ ~/hostedtoolcache/julia/nightly/aarch64/share/julia/stdlib/v1.13/Test/src/Test.jl:1771 [inlined]
 [4] macro expansion
   @ ~/work/Pkg.jl/Pkg.jl/test/historical_stdlib_version.jl:241 [inlined]
 [5] macro expansion
   @ ~/hostedtoolcache/julia/nightly/aarch64/share/julia/stdlib/v1.13/Test/src/Test.jl:1771 [inlined]
 [6] (::Main.PkgTestsOuter.PkgTestsInner.HistoricalStdlibVersionsTests.var"#19#20")()
   @ Main.PkgTestsOuter.PkgTestsInner.HistoricalStdlibVersionsTests ~/work/Pkg.jl/Pkg.jl/test/historical_stdlib_version.jl:206

GMP_jll: Error During Test at /Users/runner/work/Pkg.jl/Pkg.jl/test/historical_stdlib_version.jl:239
  Got exception outside of a @test
  MethodError: no method matching abspath(::Nothing)
  The function `abspath` exists, but no method is defined for this combination of argument types.
  
  Closest candidates are:
    abspath(::String)
     @ Base path.jl:448
    abspath(::AbstractString)
     @ Base path.jl:614
    abspath(::AbstractString, AbstractString...)
     @ Base path.jl:468
  
  Stacktrace:
    [1] fixups_from_projectfile!(env::Pkg.Types.EnvCache)
      @ Pkg.Operations ~/work/Pkg.jl/Pkg.jl/src/Operations.jl:229
    [2] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; allow_autoprecomp::Bool, preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, target::Symbol)
      @ Pkg.Operations ~/work/Pkg.jl/Pkg.jl/src/Operations.jl:1661
    [3] add
      @ ~/work/Pkg.jl/Pkg.jl/src/Operations.jl:1631 [inlined]
    [4] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, target::Symbol, allow_autoprecomp::Bool, kwargs::@Kwargs{julia_version::VersionNumber, io::IOStream})
      @ Pkg.API ~/work/Pkg.jl/Pkg.jl/src/API.jl:317
    [5] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::IOStream, kwargs::@Kwargs{julia_version::VersionNumber})
      @ Pkg.API ~/work/Pkg.jl/Pkg.jl/src/API.jl:164
    [6] add
      @ ~/work/Pkg.jl/Pkg.jl/src/API.jl:151 [inlined]
    [7] add(; name::String, uuid::Nothing, version::Nothing, url::Nothing, rev::Nothing, path::Nothing, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::@Kwargs{julia_version::VersionNumber})
      @ Pkg.API ~/work/Pkg.jl/Pkg.jl/src/API.jl:181
    [8] add
      @ ~/work/Pkg.jl/Pkg.jl/src/API.jl:170 [inlined]
    [9] macro expansion
      @ ~/work/Pkg.jl/Pkg.jl/test/historical_stdlib_version.jl:245 [inlined]

from JuliaPackaging/JLLPrefixes.jl#6 (comment) Co-Authored-By: Mosè Giordano <[email protected]> Co-Authored-By: Elliot Saba <[email protected]> Co-Authored-By: Mark Kittisopikul <[email protected]>

IanButterworth · 2025-02-11T05:09:23Z

test/historical_stdlib_version.jl

+libcxxwrap_julia_jll_UUID = Base.UUID("3eaa8342-bff7-56a5-9981-c04077f7cee7")
+libblastrampoline_jll_UUID = Base.UUID("8e850b90-86db-534c-a0d3-1478176c7d93")
+
+isolate(loaded_depot=true) do


Tests below this are new

KristofferC · 2025-02-11T13:40:22Z

What's the status of using BB together with this and a new julia version?

IanButterworth · 2025-02-11T13:50:06Z

Untested. @giordano do BB test run locally?

giordano · 2025-02-11T13:58:16Z

Can't do it right now, but relevant tests are in https://github.com/JuliaPackaging/BinaryBuilderBase.jl, in particular the test/dependencies.jl file. That's (mostly) working up to Julia v1.10.

I'm happy to make adjustments in BBB if necessary, as long as there's some guarantee we have something stable to rely on (and tests in this PR should help).

IanButterworth · 2025-02-11T19:36:12Z

@giordano @staticfloat I've requested review mainly for the stuff in test/historical_stdlib_version.jl.

giordano · 2025-02-11T19:38:56Z

Did you try running the test/dependencies.jl tests only? Those are the most relevant ones and if I remember correctly you don't even need the sandbox at all for them.

IanButterworth · 2025-02-11T19:43:30Z

Just cleared space and ran it again and it all passed. (missed that it was a space issue before)

(@v1.13) pkg> test BinaryBuilderBase
     Testing BinaryBuilderBase
      Status `/private/var/folders/1z/jf841bdj73bdj3vk7kc7f_3w0000gn/T/jl_jyNS8s/Project.toml`
  [7f725544] BinaryBuilderBase v1.35.2
  [944b1d66] CodecZlib v0.7.7
  [187b0558] ConstructionBase v1.5.8
  [682c06a0] JSON v0.21.4
  [d8793406] ObjectFile v0.4.3
  [f43a241f] Downloads v1.6.0
  [76f85450] LibGit2 v1.11.0
  [8f399da3] Libdl v1.11.0
  [44cfe95a] Pkg v1.12.0
  [ea8e919c] SHA v0.7.0
  [8dfed614] Test v1.11.0
  [cf7118a7] UUIDs v1.11.0
      Status `/private/var/folders/1z/jf841bdj73bdj3vk7kc7f_3w0000gn/T/jl_jyNS8s/Manifest.toml`
  [7f725544] BinaryBuilderBase v1.35.2
  [944b1d66] CodecZlib v0.7.7
  [187b0558] ConstructionBase v1.5.8
⌃ [6df8b67a] HistoricalStdlibVersions v2.0.1
  [692b3bcd] JLLWrappers v1.7.0
  [682c06a0] JSON v0.21.4
  [d8793406] ObjectFile v0.4.3
  [bac558e1] OrderedCollections v1.8.0
  [6c11c7d4] OutputCollectors v0.1.1
  [69de0a69] Parsers v2.8.1
  [aea7be01] PrecompileTools v1.2.1
  [21216c6a] Preferences v1.4.3
  [92933f4c] ProgressMeter v1.10.2
  [189a3867] Reexport v1.2.2
  [6c6a2e73] Scratch v1.2.1
  [777ac1f9] SimpleBufferStream v1.2.0
  [53d494c1] StructIO v0.3.1
  [3bb67fe8] TranscodingStreams v0.11.3
  [1fd713ca] Attr_jll v2.5.2+0
  [6e34b625] Bzip2_jll v1.0.9+0
  [be1be57a] Gzip_jll v1.13.0+0
  [94ce4f54] Libiconv_jll v1.18.0+0
  [9b64493d] Tar_jll v1.35.0+0
  [ffd25f8a] XZ_jll v5.6.4+1
  [3161d3a3] Zstd_jll v1.5.7+0
  [1bc43ea1] pigz_jll v2.8.0+0
  [88f77b66] unzip_jll v6.0.2+0
  [0dad84c5] ArgTools v1.1.2
  [56f22d72] Artifacts v1.11.0
  [2a0f44e3] Base64 v1.11.0
  [ade2ca70] Dates v1.11.0
  [8ba89e20] Distributed v1.11.0
  [f43a241f] Downloads v1.6.0
  [7b1f6079] FileWatching v1.11.0
  [b77e0a4c] InteractiveUtils v1.11.0
  [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2 v1.11.0
  [8f399da3] Libdl v1.11.0
  [56ddb016] Logging v1.11.0
  [d6f4376e] Markdown v1.11.0
  [a63ad114] Mmap v1.11.0
  [ca575930] NetworkOptions v1.3.0
  [44cfe95a] Pkg v1.12.0
  [de0858da] Printf v1.11.0
  [3fa0cd96] REPL v1.11.0
  [9a3f8284] Random v1.11.0
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization v1.11.0
  [6462fe0b] Sockets v1.11.0
  [f489334b] StyledStrings v1.11.0
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test v1.11.0
  [cf7118a7] UUIDs v1.11.0
  [4ec0a83e] Unicode v1.11.0
  [deac9b47] LibCURL_jll v8.11.1+1
  [e37daf67] LibGit2_jll v1.9.0+0
  [29816b5a] LibSSH2_jll v1.11.3+1
  [14a3606d] MozillaCACerts_jll v2024.12.31
  [458c3c95] OpenSSL_jll v3.0.15+2
  [83775a58] Zlib_jll v1.3.1+2
  [8e850ede] nghttp2_jll v1.64.0+1
  [3f19e933] p7zip_jll v17.5.0+2
        Info Packages marked with ⌃ have new versions available and may be upgradable.
     Testing Running tests...
Test Summary:    | Pass  Total  Time
Compat functions |   19     19  0.3s
Test Summary: | Pass  Total  Time
ArchiveUtils  |   13     13  2.0s
Test Summary: | Pass  Total  Time
Archive Utils |   22     22  0.8s
┌ Warning: The active manifest file is missing a julia version entry. Dependencies may have been resolved with a different julia version.
└ @ /private/var/folders/1z/jf841bdj73bdj3vk7kc7f_3w0000gn/T/jl_7KnNK6/x86_64-linux-gnu/.project/Manifest.toml:0
Test Summary: | Pass  Broken  Total   Time
Dependencies  |  107       4    111  43.9s
Test Summary: | Pass  Total  Time
resolve_jlls  |   10     10  0.8s
Test Summary:       | Pass  Total  Time
Supported Platforms |    4      4  0.4s
Test Summary: | Pass  Total  Time
abi_agnostic  |    1      1  0.1s
Test Summary: | Pass  Total  Time
AnyPlatform   |    8      8  2.1s
Test Summary:     | Pass  Total  Time
Target properties |   27     27  0.0s
Test Summary: | Pass  Total  Time
march flags   |   14     14  0.1s
Test Summary: | Pass  Total  Time
parse         |    5      5  0.0s
Test Summary: | Pass  Total  Time
Prefix        |   12     12  0.1s
Test Summary: | Pass  Total  Time
Packaging     |    7      7  0.1s
Test Summary:   | Pass  Total  Time
Tree symlinking |   26     26  0.2s
Test Summary: | Pass  Total  Time
Compression   |    3      3  0.3s
Test Summary: | Pass  Total  Time
Products      |   67     67  4.5s
Test Summary: | Pass  Total  Time
Dlopen flags  |    7      7  0.0s
Test Summary: | Pass  Total  Time
Sources       |   75     75  5.1s
Test Summary:    | Pass  Total  Time
Expand platforms |   25     25  0.4s
Test Summary:   | Pass  Total  Time
Compiler Shards |   45     45  1.9s
Test Summary:      | Pass  Total  Time
Wrappers utilities |   27     27  0.0s
Test Summary: | Pass  Total  Time
with_logfile  |    3      3  0.1s
 Downloading artifact: GCCBootstrap-x86_64-linux-musl.v6.1.0.x86_64-linux-musl.unpacked
  Downloaded artifact: GCCBootstrap-x86_64-linux-musl.v6.1.0.x86_64-linux-musl.unpacked
Test Summary:    | Pass  Broken  Total     Time
Runner utilities |   67       3     70  1m33.5s
 Downloading artifact: GCCBootstrap-x86_64-linux-gnu.v5.2.0.x86_64-linux-musl.unpacked
  Downloaded artifact: GCCBootstrap-x86_64-linux-gnu.v5.2.0.x86_64-linux-musl.unpacked
 Downloading artifact: RustBase.v1.83.0.x86_64-linux-musl.unpacked
  Downloaded artifact: RustBase.v1.83.0.x86_64-linux-musl.unpacked
 Downloading artifact: RustToolchain-x86_64-linux-gnu.v1.83.0.x86_64-linux-musl.unpacked
  Downloaded artifact: RustToolchain-x86_64-linux-gnu.v1.83.0.x86_64-linux-musl.unpacked
 Downloading artifact: RustToolchain-x86_64-linux-musl.v1.83.0.x86_64-linux-musl.unpacked
  Downloaded artifact: RustToolchain-x86_64-linux-musl.v1.83.0.x86_64-linux-musl.unpacked
 Downloading artifact: Go.v1.23.0.x86_64-linux-musl.unpacked
  Downloaded artifact: Go.v1.23.0.x86_64-linux-musl.unpacked
Test Summary: | Pass  Total     Time
Shards        |    3      3  1m11.8s
     Testing BinaryBuilderBase tests passed

giordano · 2025-02-11T20:32:44Z

The fact that tests pass is very promising (they stopped passing at some point and I lost the energy to follow Pkg changes back then), but there are some things that I'm not sure are covered by BBB tests:

Pkg.add([PackageSpec("GMP_jll")]; julia_version=v"1.6") doesn't install anything #2930: this was the reason why we never had BinaryBuilder fully working on v1.8+
Pkg.add(Pkg.Types.Context(;julia_version), [PackageSpec(...)]) failing since Julia v1.7 #2942: this was already broken in v1.7, but people have been working around it with some nasty hacking of Pkg.Types.get_last_stdlibs
Pkg.add doesn't install the requested version of JLL stdlib #3113
Unable to install unregistered package in Julia >=1.7 (when providing a Context) #3251 is the reason why https://docs.binarybuilder.org/stable/FAQ/#Can-I-publish-a-JLL-package-locally-without-going-through-Yggdrasil?-1 doesn't work (people have been asking about this in Slack every other week)

I'm just skimming through the binarybuilder label here.

giordano · 2025-02-11T20:42:34Z

Pkg.add([PackageSpec("GMP_jll")]; julia_version=v"1.6") doesn't install anything #2930: this was the reason why we never had BinaryBuilder fully working on v1.8+

I need to look at it better, but this issue may have been solved by HistoricalStdlibVersions (which came several months afterwards after that ticket was opened), and it's good this PR adds more tests about it.

IanButterworth · 2025-02-11T20:59:10Z

src/Operations.jl

+            # pkg.exts = p.exts # TODO: STDLIBS_BY_VERSION doesn't record this
+            # pkg.entryfile = p.entryfile # TODO: STDLIBS_BY_VERSION doesn't record this


Opened an issue JuliaPackaging/HistoricalStdlibVersions.jl#32

IanButterworth · 2025-02-11T23:00:04Z

test/historical_stdlib_version.jl

+            @test isdir(src)
+            @test isfile(joinpath(src, "Artifacts.toml"))


These are both failing, so #2930 remains an issue

IanButterworth · 2025-02-12T21:32:25Z

I've marked the #2930 broken tests as such. I'm proposing to merge this as is and I will continue to look into fixing #2930

staticfloat

This is great, thank you for taking this on, Ian!

giordano

🚀

IanButterworth changed the title ~~Expand HistoricalStdlibVersions tests~~ Expand and consolidate HistoricalStdlibVersions tests Feb 10, 2025

IanButterworth force-pushed the ib/more_hsv_tests branch 3 times, most recently from 627aea8 to 775f3a1 Compare February 10, 2025 04:25

IanButterworth and others added 3 commits February 10, 2025 20:11

don't install HSV manually

97cae6e

add HistoricalStdlibVersions tests

9612305

from JuliaPackaging/JLLPrefixes.jl#6 (comment) Co-Authored-By: Mosè Giordano <[email protected]> Co-Authored-By: Elliot Saba <[email protected]> Co-Authored-By: Mark Kittisopikul <[email protected]>

consolidate all julia_version (HistoricalStdlibVersions) tests

c64e2b8

IanButterworth force-pushed the ib/more_hsv_tests branch from 775f3a1 to c64e2b8 Compare February 11, 2025 01:11

IanButterworth added 6 commits February 10, 2025 20:34

make error more informative

b11d19e

fix version check

1a84068

rename testsets

648a02d

use temp envs for each test

74ef757

make fixups_from_projectfile! work with STDLIBS_BY_VERSION if needed

0919cbd

tweak tests

533a3c8

IanButterworth changed the title ~~Expand and consolidate HistoricalStdlibVersions tests~~ Fixes for julia_version. Expand and consolidate julia_version tests. Feb 11, 2025

workaround Pkg.dependencies() mbedtls_jll bug

109770e

IanButterworth marked this pull request as ready for review February 11, 2025 05:06

IanButterworth commented Feb 11, 2025

View reviewed changes

IanButterworth mentioned this pull request Feb 11, 2025

Future Pkg compatibility JuliaPackaging/JLLPrefixes.jl#6

Open

This comment was marked as resolved.

Sign in to view

IanButterworth requested review from staticfloat and giordano February 11, 2025 19:35

IanButterworth commented Feb 11, 2025

View reviewed changes

IanButterworth force-pushed the ib/more_hsv_tests branch from 877eb14 to 339adc5 Compare February 11, 2025 21:38

test that GMP_jll source is installed

04e48fd

IanButterworth force-pushed the ib/more_hsv_tests branch from 339adc5 to 04e48fd Compare February 11, 2025 22:34

giordano added the binarybuilder Related to the use of Pkg in BinaryBuilder label Feb 11, 2025

IanButterworth commented Feb 11, 2025

View reviewed changes

mark tests as broken

f402910

staticfloat approved these changes Feb 13, 2025

View reviewed changes

giordano approved these changes Feb 13, 2025

View reviewed changes

IanButterworth merged commit 0333d55 into JuliaLang:master Feb 13, 2025
8 of 9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes for julia_version. Expand and consolidate julia_version tests. #4151

Fixes for julia_version. Expand and consolidate julia_version tests. #4151

IanButterworth commented Feb 9, 2025 •

edited

Loading

IanButterworth Feb 11, 2025

KristofferC commented Feb 11, 2025

IanButterworth commented Feb 11, 2025

giordano commented Feb 11, 2025

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

IanButterworth commented Feb 11, 2025

giordano commented Feb 11, 2025

IanButterworth commented Feb 11, 2025 •

edited

Loading

giordano commented Feb 11, 2025

giordano commented Feb 11, 2025

IanButterworth Feb 11, 2025

IanButterworth Feb 11, 2025

IanButterworth commented Feb 12, 2025 •

edited

Loading

staticfloat left a comment

giordano left a comment

		# pkg.exts = p.exts # TODO: STDLIBS_BY_VERSION doesn't record this
		# pkg.entryfile = p.entryfile # TODO: STDLIBS_BY_VERSION doesn't record this

		@test isdir(src)
		@test isfile(joinpath(src, "Artifacts.toml"))

Fixes for julia_version. Expand and consolidate julia_version tests. #4151

Fixes for julia_version. Expand and consolidate julia_version tests. #4151

Conversation

IanButterworth commented Feb 9, 2025 • edited Loading

IanButterworth Feb 11, 2025

Choose a reason for hiding this comment

KristofferC commented Feb 11, 2025

IanButterworth commented Feb 11, 2025

giordano commented Feb 11, 2025

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

IanButterworth commented Feb 11, 2025

giordano commented Feb 11, 2025

IanButterworth commented Feb 11, 2025 • edited Loading

giordano commented Feb 11, 2025

giordano commented Feb 11, 2025

IanButterworth Feb 11, 2025

Choose a reason for hiding this comment

IanButterworth Feb 11, 2025

Choose a reason for hiding this comment

IanButterworth commented Feb 12, 2025 • edited Loading

staticfloat left a comment

Choose a reason for hiding this comment

giordano left a comment

Choose a reason for hiding this comment

IanButterworth commented Feb 9, 2025 •

edited

Loading

IanButterworth commented Feb 11, 2025 •

edited

Loading

IanButterworth commented Feb 12, 2025 •

edited

Loading