Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[libjulia] add or update 1.12-DEV, 1.11-DEV, 1.10 #8151

Merged
merged 5 commits into from
Feb 23, 2024

Conversation

fingolfin
Copy link
Member

No description provided.

@fingolfin fingolfin requested a review from benlorenz February 22, 2024 13:31
@@ -31,7 +31,7 @@ function Pkg.Types.is_stdlib(uuid::Base.UUID, julia_version::VersionNumber)
return false
end

jllversion=v"1.10.8"
jllversion=v"1.10.9"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be 1.11? 1.12? It's not clear to me given that 1.10.8 already contained 1.11-dev... Huh

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think at some point we gave up with this number and just kept bumping the patch version. Since it is only used as BuildDependency there is no need for compat statements anyway.
Those packages that need a specific version just fix that exact version and that's about it.

L/libjulia/common.jl Outdated Show resolved Hide resolved
@fingolfin
Copy link
Member Author

macOS builds in Julia >= 1.10 failing now.

Weird also mingw builds for Julia 1.6.3 and 1.7.0 ?!? Has something changed in that regard on Yggdrasil? Error is e.g.

[14:50:28] /workspace/destdir/lib/libuv.a(libuv_la-process.o): In function `uv__kill':
[14:50:28] /workspace/srcdir/libuv/src/win/process.c:1292: undefined reference to `_imp__CoTaskMemFree@4'
[14:50:28] collect2: error: ld returned 1 exit status
[14:50:28] make[2]: *** [Makefile:108: /workspace/srcdir/julia-1.7.0/src/flisp/flisp.exe] Error 1
[14:50:28] make[2]: Leaving directory '/workspace/srcdir/julia-1.7.0/src/flisp'

Aha, libuv; and for 1.6.3 and 1.7.0 we don't specify a version for those. Let's see.

@fingolfin
Copy link
Member Author

Pushed a change which might or might not help 1.6 and 1.7 with mingw, by hardcoding JLL versions there, too.

For darwin, I see these errors:

[14:57:50]  x86_64-apple-darwin14-clang -mmacosx-version-min=10.14 -m64 -O3 -g -DDEP_LIBS="\"libgcc_s.1.dylib:julia/libopenlibm.dylib:@:@libjulia-internal.1.10.dylib:@libjulia-codegen.1.10.dylib:\"" -std=gnu11 -pipe -fPIC -fno-strict-aliasing -D_FILE_OFFSET_BITS=64 -I/workspace/srcdir/julia-1.10.0/src -I/workspace/srcdir/julia-1.10.0/src -I/workspace/srcdir/julia-1.10.0/src/support -I/workspace/srcdir/julia-1.10.0/usr/include -ffreestanding /workspace/srcdir/julia-1.10.0/cli/trampolines/trampolines_x86_64.S -c -o loader_trampolines.o
[14:57:50] error: /opt/x86_64-apple-darwin14/bin/x86_64-apple-darwin14-nm: invalid argument -D
[14:57:50] Usage: /opt/x86_64-apple-darwin14/bin/x86_64-apple-darwin14-nm [-agnopruUmxjlfAPL[s segname sectname] [-] [-t format] [[-arch <arch_flag>] ...] [file ...]

and then a bit later

[14:57:50]  x86_64-apple-darwin14-clang -mmacosx-version-min=10.14 -m64  -std=gnu11 -pipe -fPIC -fno-strict-aliasing -D_FILE_OFFSET_BITS=64 -I/workspace/srcdir/julia-1.10.0/src -I/workspace/srcdir/julia-1.10.0/src -I/workspace/srcdir/julia-1.10.0/src/support -I/workspace/srcdir/julia-1.10.0/usr/include -ffreestanding -shared -O3 -g -DDEP_LIBS="\"libgcc_s.1.dylib:julia/libopenlibm.dylib:@:@libjulia-internal.1.10.dylib:@libjulia-codegen.1.10.dylib:\"" ./loader_lib.o ./loader_trampolines.o -o /workspace/srcdir/julia-1.10.0/usr/lib/libjulia.1.10.0.dylib -Wl,-compatibility_version,1.10 -Wl,-current_version,1.10.0  -ffreestanding -L/workspace/srcdir/julia-1.10.0/usr/lib -L/workspace/srcdir/julia-1.10.0/usr/lib -lSystem
[14:57:50] ld64.lld: error: ./loader_trampolines.o: unhandled file type
[14:57:50] clang-16: error: linker command failed with exit code 1 (use -v to see invocation)
[14:57:50] make[1]: *** [Makefile:117: /workspace/srcdir/julia-1.10.0/usr/lib/libjulia.1.10.0.dylib] Error 1
[14:57:50] make[1]: Leaving directory '/workspace/srcdir/julia-1.10.0/cli'
[14:57:50] make: *** [Makefile:94: julia-cli-release] Error 2
[14:57:50] make: *** Waiting for unfinished jobs....

Also this (harmless) warnings pops up a lot:

[14:57:50] make[1]: xcode-select: No such file or directory

Harmless, but suggests a possible improvement in the Julia build system

@benlorenz
Copy link
Contributor

The loader_trampolines.o error might be a race condition because we are building the debug and non-debug version at the same time. Usually this disappears when re-running the job, see e.g. #7484 (comment).

@fingolfin
Copy link
Member Author

Mingw builds working now.

Darwin/macOS jobs still failing.

@benlorenz Thanks for the hint about loader_trampolines.o , I can try a workaround then.

The x86_64-apple-darwin14-nm: invalid argument -D error also remains.
It comes from src/Makefile:

LLVM_SHLIB_SYMBOL_VERSION := $(shell nm -D --with-symbol-versions $(shell $(LLVM_CONFIG_HOST) --libfiles --link-shared | awk '{print $1; exit}') | \
                               grep _ZN4llvm3Any6TypeId | head -n 1 | sed -e 's/.*@//')

I wonder what's going on there? Both GNU nm as well as llvm-nm support it. Where does x86_64-apple-darwin14-nm come from? @giordano ?

@fingolfin
Copy link
Member Author

Let's see if my patch from JuliaLang/julia#53437 does the job

@giordano
Copy link
Member

Where does x86_64-apple-darwin14-nm come from?

I'm not entirely sure, but it may be cctools, we use an old-ish version:

# MacOS doesn't actually use binutils, it uses cctools
binutils_sources = [
GitSource("https://github.com/tpoechtrager/apple-libtapi.git",
"a66284251b46d591ee4a0cb4cf561b92a0c138d8"),
GitSource("https://github.com/tpoechtrager/cctools-port.git",
"634a084377ee2e2932c66459b0396edf76da2e9f"),
]

@fingolfin
Copy link
Member Author

fingolfin commented Feb 22, 2024

Chasing down nm:

sandbox:${WORKSPACE} # which nm
/opt/bin/x86_64-apple-darwin14-libgfortran3-cxx03/nm
sandbox:${WORKSPACE} # cat /opt/bin/x86_64-apple-darwin14-libgfortran3-cxx03/nm
#!/bin/bash
# This compiler wrapper script brought into existence by `generate_compiler_wrappers!()`

if [ "x${SUPER_VERBOSE}" = "x" ]; then
    vrun() { "$@"; }
else
    vrun() { echo -e "\e[96m$@\e[0m" >&2; "$@"; }
fi

ARGS=( "$@" )
PRE_FLAGS=()
POST_FLAGS=()


if [[ " ${ARGS[@]} " == *"-march="* ]]; then
    echo "BinaryBuilder: Cannot force an architecture via -march" >&2
    exit 1
fi

vrun ${CCACHE} /opt/x86_64-apple-darwin14/bin/x86_64-apple-darwin14-nm "${PRE_FLAGS[@]}" "${ARGS[@]}" "${POST_FLAGS[@]}"
sandbox:${WORKSPACE} # file /opt/x86_64-apple-darwin14/bin/x86_64-apple-darwin14-nm
/opt/x86_64-apple-darwin14/bin/x86_64-apple-darwin14-nm: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, with debug_info, not stripped
sandbox:${WORKSPACE} # strings /opt/x86_64-apple-darwin14/bin/x86_64-apple-darwin14-nm | fgrep -i cc
access
cctools-949.0.1
@(#)PROGRAM:ld  PROJECT:cctools-949.0.1
malformed object (LC_VERSION_MIN_MACOSX is set but the LC_BUILD_VERSION load command is not for MACCATALYST)
malformed object (the two LC_BUILD_VERSION load commands are not for MACOS and MACCATALYST)
GCC: (Alpine 9.3.0) 9.3.0
access

So yeah, cctools.

@fingolfin
Copy link
Member Author

But since we override LLVM_SHLIB_SYMBOL_VERSION anyway, it doesn't matter. Still, it is confusing to see. Ah well.

@fingolfin
Copy link
Member Author

Darwin patch helped. Now 1.10 builds again. but new errors on 1.11 (and 1.12, which at this point is still quite similar):

[17:09:43] make[2]: Leaving directory '/workspace/srcdir/julia/src/flisp'
[17:09:43]  x86_64-apple-darwin14-clang++ -mmacosx-version-min=10.14 -m64 -shared  -pipe -fPIC -fno-rtti -std=c++17 -pedantic -D_FILE_OFFSET_BITS=64   -O0 -DJL_DEBUG_BUILD -fstack-protector -g -D_GNU_SOURCE -I. -I/workspace/srcdir/julia/src -I/workspace/srcdir/julia/src/flisp -I/workspace/srcdir/julia/src/support -I/workspace/destdir/include -I/workspace/srcdir/julia/usr/include -I/workspace/srcdir/julia/deps/valgrind -Wall -Wno-strict-aliasing -fno-omit-frame-pointer -fvisibility=hidden -fno-common -Wno-comment -Wpointer-arith -Wundef -Wno-return-type-c-linkage -DJL_BUILD_ARCH='"x86_64"' -DJL_BUILD_UNAME='"Darwin"' -I -DLLVM_SHLIB "-DJL_SYSTEM_IMAGE_PATH=\"../lib/julia/sys-debug.dylib\"" "-DJL_LIBJULIA_SONAME=\"@rpath/libjulia-debug.1.11.dylib\"" ./jltypes.dbg.obj ./gf.dbg.obj ./typemap.dbg.obj ./smallintset.dbg.obj ./ast.dbg.obj ./builtins.dbg.obj ./module.dbg.obj ./interpreter.dbg.obj ./symbol.dbg.obj ./dlload.dbg.obj ./sys.dbg.obj ./init.dbg.obj ./task.dbg.obj ./array.dbg.obj ./genericmemory.dbg.obj ./staticdata.dbg.obj ./toplevel.dbg.obj ./jl_uv.dbg.obj ./datatype.dbg.obj ./simplevector.dbg.obj ./runtime_intrinsics.dbg.obj ./precompile.dbg.obj ./jloptions.dbg.obj ./mtarraylist.dbg.obj ./threading.dbg.obj ./scheduler.dbg.obj ./stackwalk.dbg.obj ./gc.dbg.obj ./gc-debug.dbg.obj ./gc-pages.dbg.obj ./gc-stacks.dbg.obj ./gc-alloc-profiler.dbg.obj ./gc-page-profiler.dbg.obj ./method.dbg.obj ./jlapi.dbg.obj ./signal-handling.dbg.obj ./safepoint.dbg.obj ./timing.dbg.obj ./subtype.dbg.obj ./rtutils.dbg.obj ./gc-heap-snapshot.dbg.obj ./crc32c.dbg.obj ./APInt-C.dbg.obj ./processor.dbg.obj ./ircode.dbg.obj ./opaque_closure.dbg.obj ./codegen-stubs.dbg.obj ./coverage.dbg.obj ./runtime_ccall.dbg.obj -Wl,-rpath,'@loader_path/' -o /workspace/srcdir/julia/usr/lib/libjulia-internal-debug.1.11.0.dylib -L/workspace/destdir/lib -Wl,-compatibility_version,1.11 -Wl,-current_version,1.11.0 -L/workspace/srcdir/julia/usr/lib -L/workspace/srcdir/julia/usr/lib -Xlinker -all_load ./flisp/libflisp-debug.a -Xlinker -all_load ./support/libsupport-debug.a -ljulia-debug -Xlinker -all_load /workspace/destdir/lib/libuv.a -Xlinker -all_load -lutf8proc   -L/workspace/destdir/lib -lLLVM  -framework CoreFoundation -Wl,-U,__dyld_atfork_parent -Wl,-U,__dyld_atfork_prepare -Wl,-U,__dyld_dlopen_atfork_parent -Wl,-U,__dyld_dlopen_atfork_prepare -Wl,-U,_jl_image_pointers -Wl,-U,_jl_system_image_data -Wl,-U,_jl_system_image_size
[17:09:43] ld64.lld: error: undefined symbol: operator new(unsigned long, std::align_val_t)
[17:09:43] >>> referenced by new:230 (/opt/x86_64-apple-darwin14/x86_64-apple-darwin14/sys-root/usr/include/c++/v1/new:230)
[17:09:43] >>>               ./gc-heap-snapshot.dbg.obj:(symbol std::__1::__libcpp_allocate(unsigned long, unsigned long)+0x37)
[17:09:43] 
[17:09:43] ld64.lld: error: undefined symbol: operator delete(void*, std::align_val_t)
[17:09:43] >>> referenced by new:312 (/opt/x86_64-apple-darwin14/x86_64-apple-darwin14/sys-root/usr/include/c++/v1/new:312)
[17:09:43] >>>               ./processor.dbg.obj:(symbol void std::__1::_DeallocateCaller::__do_call<std::align_val_t>(void*, std::align_val_t)+0x19)
[17:09:43] clang-16: error: linker command failed with exit code 1 (use -v to see invocation)
[17:09:43] make[1]: *** [Makefile:397: /workspace/srcdir/julia/usr/lib/libjulia-internal-debug.1.11.0.dylib] Error 1
[17:09:43] make[1]: Leaving directory '/workspace/srcdir/julia/src'
[17:09:43] make: *** [Makefile:97: julia-src-debug] Error 2
[17:09:44]  ---> make USE_CROSS_FLISP=1 NO_GIT=1 LDFLAGS="${LDFLAGS}" CFLAGS="${CFLAGS}" CXXFLAGS="${CXXFLAGS}" -j${nproc} VERBOSE=1 ${MAKE_TARGET} DSYMUTIL=true
[17:09:44]  ---> make USE_CROSS_FLISP=1 NO_GIT=1 LDFLAGS="${LDFLAGS}" CFLAGS="${CFLAGS}" CXXFLAGS="${CXXFLAGS}" -j${nproc} VERBOSE=1 ${MAKE_TARGET} DSYMUTIL=true
[17:09:44] Previous command exited with 2
[17:09:44] Child Process exited, exit code 2

@giordano
Copy link
Member

Try with clang_use_lld=false, although this error doesn't feel like it should depend on the linker (hopefully?).

@benlorenz
Copy link
Contributor

benlorenz commented Feb 22, 2024

Reading https://reviews.llvm.org/D129198, I think we might need to do the osx framework update dance to get these symbols (to 10.13 or newer, I think the default is 10.12 on x86_64). Not really sure in what form the compiler is supposed to deal with this but if I understand some related comments correct then this operator is defined in the headers but not in the libc++ for 10.12.

Setting -fno-aligned-allocation also seems to work, at least when building x86_64-apple-darwin-julia_version+1.12.0 locally:

    # macos 10.12 has aligned alloc in the header but not in libc++
    # leading to linker errors
    if [[ "${target}" == *x86_64-apple* ]]; then
        CXXFLAGS=-fno-aligned-allocation
    fi

Not totally sure how this will affect the downstream packages, maybe the CXXFLAGS approach is better since then the symbols should not appear anymore, otherwise we might end up needing to update the osx framework for all downstream recipes as well?

fingolfin added a commit to JuliaLang/julia that referenced this pull request Feb 23, 2024
This prevents a race condition when building 'julia-cli-debug
julia-cli-release' simultaneously (as we do for libjulia_jll, and also
generally seems appropriate given what is done for all other source
files.

Motivated by JuliaPackaging/Yggdrasil#8151 so
I'll first see if it works there.

Closes #45002.
@fingolfin
Copy link
Member Author

@benlorenz thanks that did the trick

@fingolfin fingolfin merged commit eec5291 into JuliaPackaging:master Feb 23, 2024
18 checks passed
@fingolfin fingolfin deleted the mh/libjulia branch February 23, 2024 12:23
KristofferC pushed a commit to JuliaLang/julia that referenced this pull request Feb 26, 2024
This prevents a race condition when building 'julia-cli-debug
julia-cli-release' simultaneously (as we do for libjulia_jll, and also
generally seems appropriate given what is done for all other source
files.

Motivated by JuliaPackaging/Yggdrasil#8151 so
I'll first see if it works there.

Closes #45002.

(cherry picked from commit fee198b)
KristofferC pushed a commit to JuliaLang/julia that referenced this pull request Feb 26, 2024
This prevents a race condition when building 'julia-cli-debug
julia-cli-release' simultaneously (as we do for libjulia_jll, and also
generally seems appropriate given what is done for all other source
files.

Motivated by JuliaPackaging/Yggdrasil#8151 so
I'll first see if it works there.

Closes #45002.

(cherry picked from commit fee198b)
KristofferC pushed a commit to JuliaLang/julia that referenced this pull request Feb 26, 2024
This prevents a race condition when building 'julia-cli-debug
julia-cli-release' simultaneously (as we do for libjulia_jll, and also
generally seems appropriate given what is done for all other source
files.

Motivated by JuliaPackaging/Yggdrasil#8151 so
I'll first see if it works there.

Closes #45002.

(cherry picked from commit fee198b)
tecosaur pushed a commit to tecosaur/julia that referenced this pull request Mar 4, 2024
This prevents a race condition when building 'julia-cli-debug
julia-cli-release' simultaneously (as we do for libjulia_jll, and also
generally seems appropriate given what is done for all other source
files.

Motivated by JuliaPackaging/Yggdrasil#8151 so
I'll first see if it works there.

Closes JuliaLang#45002.
mkitti pushed a commit to mkitti/julia that referenced this pull request Mar 7, 2024
This prevents a race condition when building 'julia-cli-debug
julia-cli-release' simultaneously (as we do for libjulia_jll, and also
generally seems appropriate given what is done for all other source
files.

Motivated by JuliaPackaging/Yggdrasil#8151 so
I'll first see if it works there.

Closes JuliaLang#45002.
Drvi pushed a commit to RelationalAI/julia that referenced this pull request Jun 7, 2024
This prevents a race condition when building 'julia-cli-debug
julia-cli-release' simultaneously (as we do for libjulia_jll, and also
generally seems appropriate given what is done for all other source
files.

Motivated by JuliaPackaging/Yggdrasil#8151 so
I'll first see if it works there.

Closes JuliaLang#45002.

(cherry picked from commit fee198b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants