Source build: missing symbols in p7zip build

[fxcoudert]

After applying the various fixes currently proposed for source builds, a macOS build with make USE_BINARYBUILDER=0 VERBOSE=1 CFLAGS='-Wno-implicit-function-declaration' gets the following error:

/Applications/Xcode.app/Contents/Developer/usr/bin/make -C scratch/p7zip-16.2.0/ DESTDIR="" prefix=/private/tmp/julia/usr bindir=/private/tmp/julia/usr/tools libdir=/private/tmp/julia/usr/lib shlibdir=/private/tmp/julia/usr/lib libexecdir=/private/tmp/julia/usr/libexec datarootdir=/private/tmp/julia/usr/share includedir=/private/tmp/julia/usr/include sysconfdir=/private/tmp/julia/usr/etc O= CC="clang -mmacosx-version-min=11.0" CXX="clang++ -mmacosx-version-min=11.0" 7za
mkdir -p bin
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C CPP/7zip/Bundles/Alone all
clang -mmacosx-version-min=11.0 -Wno-implicit-function-declaration ../../../../C/7zCrc.c
Undefined symbols for architecture arm64:
  "_CrcUpdateT4", referenced from:
      _CrcGenerateTable in 7zCrc-99f363.o
     (maybe you meant: _g_CrcUpdateT4)
  "_CrcUpdateT8", referenced from:
      _CrcGenerateTable in 7zCrc-99f363.o
     (maybe you meant: _g_CrcUpdateT8)
  "_main", referenced from:
     implicit entry/start for main executable
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [7zCrc.o] Error 1
make[2]: *** [7za] Error 2
make[1]: *** [scratch/p7zip-16.2.0/build-compiled] Error 2
make: *** [julia-deps] Error 2

[fxcoudert]

The bug is because CFLAGS='-Wno-implicit-function-declaration' is passed to the top-level julia make. I pass it to work around the bug in libuv build, but it overrides a definition of CFLAGS that is hardcode in the p7zip makefile (which includes -c).

[giordano]

If a flag is needed only to build a specific package maybe we can find a way to pass only to that one. For example to set some flags only for Julia and not the other dependencies we recently did #44867, precisely to avoid overriding other packages' default flags.

[fxcoudert]

Setting CFLAGS for libuv is a workaround for this bug: #45200 (comment)
It should have been fixed by this commit JuliaLang/libuv#21 but apparently the julia master was not updated to pick up the fixed libuv

[fxcoudert]

Some good news, though: once all the bugs currently reported are fixed, I can build julia master from source on macOS ARM (aarch64-apple-darwin21). I will run make testall and report if there are any issues. Are there other self-tests that could/should be performed?

[giordano]

Some good news, though: once all the bugs currently reported are fixed, I can build julia master from source on macOS ARM (aarch64-apple-darwin21).

That's a great news, thanks a lot!

I will run make testall and report if there are any issues. Are there other self-tests that could/should be performed?

No, that should be enough, I think. Note that this platform is currently Tier 3 and it's very likely there will be some test failures (see for example #43321, where you already shared some failures). That's ok according to the definition of Tier 3, although it'd be useful to track them to eventually escalate its support level (pending the possibility of securing more machines for CI, which at the moment doesn't scale up very well for this platform having a single buildmachine). At least there shouldn't be widespread segmentation faults as it happens in the v1.7.x series.

[fxcoudert]

For a full source build of current master, with patches applied:

Test Summary:                             |     Pass  Error  Broken     Total      Time
  Overall                                 | 40582941      1  352662  40935604  36m41.0s

So 1 error only, which sounds good. But 352662 “broken”, which sounds horrible, but seems to be the same number as on Linux CI 🤷

The error is indeed the same I had reported in the past:

``` Error in testset Sockets: Error During Test at /private/tmp/julia/usr/share/julia/stdlib/v1.9/Sockets/test/runtests.jl:419 Got exception outside of a @test timeout Stacktrace: [1] error(s::String) @ Base ./error.jl:35 [2] (::Main.Test65Main_Sockets.var"#wait_with_timeout#36")(recvs::Vector{Task}, TIMEOUT_VAL::Float64) @ Main.Test65Main_Sockets /private/tmp/julia/usr/share/julia/stdlib/v1.9/Sockets/test/runtests.jl:438 [3] (::Main.Test65Main_Sockets.var"#wait_with_timeout#36")(recvs::Vector{Task}) @ Main.Test65Main_Sockets /private/tmp/julia/usr/share/julia/stdlib/v1.9/Sockets/test/runtests.jl:432 [4] macro expansion @ /private/tmp/julia/usr/share/julia/stdlib/v1.9/Sockets/test/runtests.jl:450 [inlined] [5] macro expansion @ /private/tmp/julia/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1360 [inlined] [6] top-level scope @ /private/tmp/julia/usr/share/julia/stdlib/v1.9/Sockets/test/runtests.jl:420 [7] include @ ./Base.jl:428 [inlined] [8] macro expansion @ /private/tmp/julia/test/testdefs.jl:24 [inlined] [9] macro expansion @ /private/tmp/julia/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1360 [inlined] [10] macro expansion @ /private/tmp/julia/test/testdefs.jl:23 [inlined] [11] macro expansion @ ./timing.jl:466 [inlined] [12] runtests(name::String, path::String, isolate::Bool; seed::UInt128) @ Main /private/tmp/julia/test/testdefs.jl:21 [13] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, UInt128, Tuple{Symbol}, NamedTuple{(:seed,), Tuple{UInt128}}}) @ Base ./essentials.jl:801 [14] (::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}})() @ Distributed /private/tmp/julia/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285 [15] run_work_thunk(thunk::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}}, print_error::Bool) @ Distributed /private/tmp/julia/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:70 [16] macro expansion @ /private/tmp/julia/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285 [inlined] [17] (::Distributed.var"#109#111"{Distributed.CallMsg{:call_fetch}, Distributed.MsgHeader, Sockets.TCPSocket})() @ Distributed ./task.jl:494 ERROR: LoadError: Test run finished with errors ```

[giordano]

But 352662 “broken”, which sounds horrible, but seems to be the same number as on Linux CI 🤷

Yes, "broken" tests are those that are known to fail (but it shouldn't). It's probably that high because there are broken tests inside large for loops, which makes it sound kinda worse than it is.