Workaround for FreeBSD linking to outdated system libs #21788

ararslan · 2017-05-11T03:33:56Z

FreeBSD's system libgcc_s declares its GCC version as 4.6, which is too old to build Julia. Since gfortran doesn't come installed by default, when it's installed it's done so through a GCC port, which includes its own libgcc_s. Linking to libgfortran from the port and to the system's libgcc_s causes versioning issues which wreck the build.

This PR works around this issue by determining the version of GCC used for gfortran, then linking in the libgcc_s corresponding to that version everywhere. It's a bit of a kludge, but it gets the job done.
With this change, all libraries which need to link to libgcc_s link to the proper version.

tkelman · 2017-05-11T05:34:23Z

Make.inc

+FCFLAGS += -Wl,-rpath=$(GCCPATH)
+JFFLAGS += -Wl,-rpath=$(GCCPATH)
+
+# This ensures we get the right RPATH even if we're missing FFLAGS somewhere


where does this cause problems if this is left out?

I can't recall, honestly.

would be be ideal if we can add as few flags as were necessary to get things to work

Okay, probably easiest to only add it to FC then, since that's guaranteed to end up everywhere the flags are needed.

tkelman · 2017-05-11T05:39:41Z

deps/llvm.mk

+# Part of the FreeBSD libgcc_s kludge
+ifeq ($(OS),FreeBSD)
+ifneq ($(GCCPATH),)
+LLVM_CMAKE += -DCMAKE_INSTALL_RPATH="\$$ORIGIN:$(GCCPATH)"


this isn't already set to anything?

Not for LLVM. Looks like the only CMake dependencies that set that flag are libgit2 and mbedtls.

aren't you putting it in CMAKE_COMMON? make sure that doesn't overwrite any existing settings

Added logic that avoids overriding those settings.

which cmake dependencies don't currently set this? maybe cleaner to always do in CMAKE_COMMON on freebsd, if that doesn't hurt anything on its own?

Seems like a good idea.

All CMake dependencies should have the RPATH include $ORIGIN, and indeed we're doing it explicitly and separately for libgit2, libssh2, and mbedtls, so perhaps we should just define it centrally in CMAKE_COMMON for all platforms?

tkelman · 2017-05-11T16:38:52Z

Make.inc

+_GCCMAJOR := $(shell $(FC) -dumpversion | cut -d'.' -f1)
+_GCCMINOR := $(shell $(FC) -dumpversion | cut -d'.' -f2)
+
+# The ports system uses major and minor for GCC < 5 (e.g. gcc49 for GCC 4.9), otherwise major only


We have a bit of code that uses -print-search-dirs to set a STD_LIB_PATH variable in the Windows cross-compile case, would that give the same path you're looking for here?

I don't think so, since the path where GCC lives when installed from ports isn't in the default ld search path. Though maybe I'm misunderstanding what you mean.

it's a flag for gcc to tell you its own search paths which should be where it's installed

I played around with that a bit but this solution actually seems cleaner than what I could come up with using that flag.

tkelman · 2017-05-12T19:02:52Z

deps/Makefile

@@ -163,6 +163,21 @@ DEP_LIBS_STAGED := $(filter-out suitesparse suitesparse-wrapper osxunwind,$(DEP_
 ## Common build target prefixes

 default: | $(build_prefix)
+
+# Part of the FreeBSD libgcc_s kludge


this doesn't look deps specific to me, we should maybe rearrange the patchelf definitions so they're available in the top level makefiles

I'm not sure what you mean about being deps-specific. Could you clarify?

this fits more with the kind of things the top-level Makefile does, rather than the rest of the things deps/Makefile does - not a big deal but it sticks out a little

Yeah, it's a little ugly here. Though I'm not sure what the best way to structure the logic elsewhere would be, given that this has to happen before any dependencies are built (or at least before LLVM).

tkelman · 2017-05-13T21:49:40Z

cc @iblis17 may want to take a look at this?

iblislin · 2017-05-14T06:08:47Z

The default version of lang/gcc in ports tree has been upgraded to 5.4.0 a month ago[1].
Do we still have this issue, if we just tell our user upgrade the ports?

[1]. https://github.com/freebsd/freebsd-ports/blob/master/lang/gcc/Makefile

ararslan · 2017-05-14T06:15:35Z

Yes, because it's an issue with FreeBSD's system libraries, not the ports. The only thing that would fix it on FreeBSD's side is for them to update libgcc_s to claim to be a newer GCC version or to rename libgcc_s to libcc_s, both of which have been proposed but no action has been taken AFAICT.

iblislin · 2017-05-14T06:34:08Z

I guess i misunderstand the actual issue...
How to reproduce this build issue?

I'm using this config[1] on my CI.

[1]. https://gist.github.com/iblis17/b4dca8221dcb96efcccde24f9cc2fa8d

ararslan · 2017-05-14T06:39:59Z

To reproduce, just clone Julia into some local directory and build without setting anything in Make.user. It should make it to near the end then fail with a version error when linking in BLAS.

For reference, the changes similar to those made in this PR are done automatically by the FreeBSD Ports system when the Makefile specifies USES=fortran. The changes here are to allow local builds without system libraries, which is useful for development and building for distribution.

iblislin · 2017-05-14T06:44:29Z

@ararslan ok, I have an idea to leverage ports system.
I will try a BSDmakefile out, and see it make life easier or not.

ararslan · 2017-05-14T06:52:32Z

My original solution to this was based on Ports, but I don't think we should make our build system depend on any particular external package provider, which is why I did it this way instead.

iblislin · 2017-05-14T06:59:55Z

Ok, i got your point.
I am trying these patches.

tkelman · 2017-05-14T09:41:42Z

it's useful to have both be an option, if it doesn't make for too much of a mess to support both "clone and make" builds as well as packager builds that use system libraries, custom file layouts, etc

iblislin · 2017-05-14T15:35:31Z

@ararslan In /usr/ports/Mk/Uses/fortran.mk, there is an additional flag for LDFLAGS: -B${LOCALBASE}/bin. But this PR do not include it. I'm curious about what -B actually affect and how ports system deals with libgcc_s without patching libc++.so...

(I'm managing to add -B and remove contrib/fixup-freebsd-libs.sh

ararslan · 2017-05-14T15:48:10Z

packager builds that use system libraries, custom file layouts, etc

There's already a Julia port in the Ports tree for that

I'm managing to add -B and remove contrib/fixup-freebsd-libs.sh

I don't think we should be using -B in this case, since that will make us link to GCC's libstdc++, if I'm not mistaken.

iblislin · 2017-05-15T07:02:55Z

I don't think we should be using -B in this case, since that will make us link to GCC's libstdc++, if I'm not mistaken.

I revert deps/Makefile , add -B and got this

┌─[~/tmp/julia/usr/lib]
└─[iblis@abeing]% ldd ./libLLVM-3.9.so 
./libLLVM-3.9.so:
        librt.so.1 => /usr/lib/librt.so.1 (0x802a63000)
        libthr.so.3 => /lib/libthr.so.3 (0x802c68000)
        libz.so.6 => /lib/libz.so.6 (0x802e8f000)
        libm.so.5 => /lib/libm.so.5 (0x8030a6000)
        libc++.so.1 => /usr/lib/libc++.so.1 (0x8032cf000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x803590000)
        libgcc_s.so.1 => /usr/local/lib/gcc5/libgcc_s.so.1 (0x8037b0000)
        libc.so.7 => /lib/libc.so.7 (0x800824000)

Is this effect we want?

But I encountered some issue blocking me generating sysimg.

iblislin · 2017-05-15T08:28:55Z

@ararslan I build this PR from cloning it into a new dir, but got following error

LoadError("sysimg.jl", 362, LoadError("sparse/sparse.jl", 41, LoadError("sparse/umfpack.jl", 61, ErrorException("error compiling anonymous: could not load library \"libsuitesparse_wrapper\"\n/lib/libgcc_s.so.1: version GCC_4.6.0 required by /usr/local/lib/gcc5/libgfortran.so.3 not found"))))
*** This error is usually fixed by running `make clean`. If the error persists, try `make cleanall`. ***
gmake[1]: *** [Makefile:232: /usr/home/iblis/tmp/jl2/usr/lib/julia/sys.o] Error 1
gmake: *** [Makefile:109: julia-sysimg-release] Error 2

(this error msg is same as -B + remove fixup.sh)

I'm on an old -CURRENT

└─[iblis@abeing]% uname -a
FreeBSD abeing 12.0-CURRENT FreeBSD 12.0-CURRENT #4 r309367: Mon Dec 12 11:10:09 CST 2016     root@abeing:/usr/obj/usr/src/sys/GENERIC  amd64

ararslan · 2017-05-15T18:45:41Z

Is this effect we want?

No, notice that the ldd output contains

        libc++.so.1 => /usr/lib/libc++.so.1 (0x8032cf000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x803590000)

These system libraries link to /lib/libgcc_s.so.1, which will cause the problems you encountered. The fixup-freebsd-libs script patches those files such that they link to /usr/local/lib/gcc5/libgcc_s.so.1 instead.

iblislin · 2017-05-16T05:55:49Z

I build this PR from cloning it into a new dir, but got following error

@ararslan ha, permission problem

diff --git a/contrib/fixup-freebsd-libs.sh b/contrib/fixup-freebsd-libs.sh
old mode 100644
new mode 100755

iblislin · 2017-05-16T05:57:01Z

now my build stopped here

/usr/home/iblis/tmp/jl2/usr/lib/libc++.so.1: program header too large
gmake[4]: *** [lib/IR/CMakeFiles/AttributeCompatFuncTableGen.dir/build.make:90: lib/IR/AttributesCompatFunc.inc.tmp] Error 1
gmake[3]: *** [CMakeFiles/Makefile2:688: lib/IR/CMakeFiles/AttributeCompatFuncTableGen.dir/all] Error 2
gmake[2]: *** [Makefile:152: all] Error 2
gmake[1]: *** [/usr/home/iblis/tmp/jl2/deps/llvm.mk:526: scratch/llvm-3.9.1/build_Release/build-compiled] Error 2
gmake: *** [Makefile:85: julia-deps] Error 2

ararslan · 2017-05-16T06:45:04Z

Are you using USE_SYSTEM_PATCHELF=1?

iblislin · 2017-05-16T06:48:28Z

No, it's a clean clone.

USE_SYSTEM_PATCHELF is 0, I do the following checking as well

└─[iblis@abeing]% g df |cat
diff --git a/contrib/fixup-freebsd-libs.sh b/contrib/fixup-freebsd-libs.sh
old mode 100644
new mode 100755
diff --git a/deps/Makefile b/deps/Makefile
index 4e153791bd..34b0326b16 100644
--- a/deps/Makefile
+++ b/deps/Makefile
@@ -168,6 +168,7 @@ default: | $(build_prefix)
 ifeq ($(OS),FreeBSD)
 ifneq ($(BUILD_CUSTOM_LIBCXX),1)
 default: $(build_libdir)/libc++.so.1
+$(error ${USE_SYSTEM_PATCHELF})
 ifeq ($(USE_SYSTEM_PATCHELF),1)
 $(build_libdir)/libc++.so.1:
        -$(JULIAHOME)/contrib/fixup-freebsd-libs.sh $(PATCHELF) $(build_libdir) $(GCCPATH)
┌─[~/tmp/jl2]
| [Venv(py36)] [-- INSERT --]
└─[iblis@abeing]% gmake
Makefile:171: *** 0.  Stop.
gmake: *** [Makefile:85: julia-deps] Error 2

ararslan · 2017-05-16T18:03:53Z

I was having that issue as well and I don't recall what it was I did to resolve it. Seems that patchelf expands the binary too much when inserting an RPATH into a library that doesn't already have one.

ararslan · 2017-05-17T18:05:38Z

Ah. Now I remember. It was building a custom libc++ and libc++abi (ref #21792) that fixed it for me, as doing so allows it to bypass the patchelf hack here. So I'm not sure where we should go from here, but these GCC changes are critical to get FreeBSD building out of the box.

Edit: Perhaps building a custom libc++ should be the default on FreeBSD?

FreeBSD's system libgcc_s declares its GCC version as 4.6, which is too old to build Julia. Since gfortran doesn't come installed by default, when it's installed it's done so through a GCC port, which includes its own libgcc_s. Linking to libgfortran from the port and to the system's libgcc_s causes versioning issues which wreck the build. This commit works around this issue by determining the version of GCC used for gfortran, then linking in the libgcc_s corresponding to that version everywhere. It's a bit of a kludge, but it gets the job done. With this change, all libraries which need to link to libgcc_s link to the proper version.

ararslan · 2017-05-18T04:58:16Z

Rather than trying to get patchelf to bite off more than it can chew, I've made it so that BUILD_CUSTOM_LIBCXX is set to 1 if not already set on FreeBSD. This now requires #21792 in order to build on FreeBSD.

iblislin · 2017-05-18T11:21:04Z

@ararslan now, it works fine!

ararslan · 2017-05-18T16:24:21Z

Fantastic! Thanks for trying it out, @iblis17.

ararslan · 2017-06-10T01:06:16Z

Now that #21792 has been merged, I rebuilt from this branch on FreeBSD after a gmake -C deps distcleanall and gmake cleanall and emptying the Make.user. Seems to be working great.

Tweak the order of libgcc_s in DT_NEEDED. Make FreeBSD do not require `BUILD_CUSTOM_LIBCXX`. See also: JuliaLang#21788, JuliaLang#22352

Tweak the order of libgcc_s in DT_NEEDED on FreeBSD so that building on FreeBSD does not require `BUILD_CUSTOM_LIBCXX`. See also: #21788, #22352

Tweak the order of libgcc_s in DT_NEEDED on FreeBSD so that building on FreeBSD does not require `BUILD_CUSTOM_LIBCXX`. See also: JuliaLang#21788, JuliaLang#22352

FreeBSD's system libgcc_s declares its GCC version as 4.6, which is too old to build Julia. Since gfortran doesn't come installed by default, when it's installed it's done so through a GCC port, which includes its own libgcc_s. Linking to libgfortran from the port and to the system's libgcc_s causes versioning issues which wreck the build. This commit works around this issue by determining the version of GCC used for gfortran, then linking in the libgcc_s corresponding to that version everywhere. It's a bit of a kludge, but it gets the job done. With this change, all libraries which need to link to libgcc_s link to the proper version. Ref #21788 (cherry picked from commit 7d83d99)

Ref #21788 (cherry picked from commit 78305aa)

Tweak the order of libgcc_s in DT_NEEDED on FreeBSD so that building on FreeBSD does not require `BUILD_CUSTOM_LIBCXX`. See also: #21788, #22352 Ref #22656 (cherry picked from commit d90c215)

FreeBSD's system libgcc_s declares its GCC version as 4.6, which is too old to build Julia. Since gfortran doesn't come installed by default, when it's installed it's done so through a GCC port, which includes its own libgcc_s. Linking to libgfortran from the port and to the system's libgcc_s causes versioning issues which wreck the build. This commit works around this issue by determining the version of GCC used for gfortran, then linking in the libgcc_s corresponding to that version everywhere. It's a bit of a kludge, but it gets the job done. With this change, all libraries which need to link to libgcc_s link to the proper version. Ref #21788 (cherry picked from commit 7d83d99)

Ref #21788 (cherry picked from commit 78305aa)

Tweak the order of libgcc_s in DT_NEEDED on FreeBSD so that building on FreeBSD does not require `BUILD_CUSTOM_LIBCXX`. See also: #21788, #22352 Ref #22656 (cherry picked from commit d90c215)

FreeBSD's system libgcc_s declares its GCC version as 4.6, which is too old to build Julia. Since gfortran doesn't come installed by default, when it's installed it's done so through a GCC port, which includes its own libgcc_s. Linking to libgfortran from the port and to the system's libgcc_s causes versioning issues which wreck the build. This commit works around this issue by determining the version of GCC used for gfortran, then linking in the libgcc_s corresponding to that version everywhere. It's a bit of a kludge, but it gets the job done. With this change, all libraries which need to link to libgcc_s link to the proper version. Ref #21788 (cherry picked from commit 7d83d99)

Ref #21788 (cherry picked from commit 78305aa)

Tweak the order of libgcc_s in DT_NEEDED on FreeBSD so that building on FreeBSD does not require `BUILD_CUSTOM_LIBCXX`. See also: #21788, #22352 Ref #22656 (cherry picked from commit d90c215)

FreeBSD's system libgcc_s declares its GCC version as 4.6, which is too old to build Julia. Since gfortran doesn't come installed by default, when it's installed it's done so through a GCC port, which includes its own libgcc_s. Linking to libgfortran from the port and to the system's libgcc_s causes versioning issues which wreck the build. This commit works around this issue by determining the version of GCC used for gfortran, then linking in the libgcc_s corresponding to that version everywhere. It's a bit of a kludge, but it gets the job done. With this change, all libraries which need to link to libgcc_s link to the proper version. Ref #21788 (cherry picked from commit 7d83d99)

Ref #21788 (cherry picked from commit 78305aa)

Tweak the order of libgcc_s in DT_NEEDED on FreeBSD so that building on FreeBSD does not require `BUILD_CUSTOM_LIBCXX`. See also: #21788, #22352 Ref #22656 (cherry picked from commit d90c215)

ararslan added building Build system, or building Julia or its dependencies system:freebsd Affects only FreeBSD labels May 11, 2017

ararslan requested review from vtjnash and tkelman May 11, 2017 03:33

tkelman reviewed May 11, 2017

View reviewed changes

ararslan force-pushed the aa/freebsd-gfortran branch from 1e8f549 to fa257c6 Compare May 11, 2017 15:47

tkelman reviewed May 11, 2017

View reviewed changes

tkelman reviewed May 12, 2017

View reviewed changes

ararslan force-pushed the aa/freebsd-gfortran branch 2 times, most recently from 260c7bd to 5094a1d Compare May 13, 2017 21:09

ararslan added 2 commits May 17, 2017 21:55

Rearrange CMake RPATH specifications to be centrally defined

78305aa

ararslan force-pushed the aa/freebsd-gfortran branch from 5094a1d to 78305aa Compare May 18, 2017 04:55

ararslan merged commit 99f7336 into master Jun 10, 2017

ararslan deleted the aa/freebsd-gfortran branch June 10, 2017 01:45

ararslan mentioned this pull request Jun 13, 2017

RFC: Avoid setting the RPATH for LLVM #22352

Merged

iblislin added a commit to iblislin/julia that referenced this pull request Jul 2, 2017

FreeBSD: propose a simpler way to deal with libgcc_s

d0a7b1e

Tweak the order of libgcc_s in DT_NEEDED. Make FreeBSD do not require `BUILD_CUSTOM_LIBCXX`. See also: JuliaLang#21788, JuliaLang#22352

iblislin added a commit to iblislin/julia that referenced this pull request Jul 2, 2017

FreeBSD: propose a simpler way to deal with libgcc_s

fba8996

Tweak the order of libgcc_s in DT_NEEDED. Make FreeBSD do not require `BUILD_CUSTOM_LIBCXX`. See also: JuliaLang#21788, JuliaLang#22352

ararslan pushed a commit that referenced this pull request Jul 10, 2017

A simpler way to deal with libgcc_s on FreeBSD (#22656)

d90c215

Tweak the order of libgcc_s in DT_NEEDED on FreeBSD so that building on FreeBSD does not require `BUILD_CUSTOM_LIBCXX`. See also: #21788, #22352

vtjnash mentioned this pull request Aug 16, 2017

[release-0.6] win: avoid stack overflow in initial deserialization #23255

Merged

ararslan mentioned this pull request Aug 22, 2017

Repeated calls to LibGit2.fetch segfaults on FreeBSD 11.1 #23328

Closed

ararslan added a commit that referenced this pull request Apr 25, 2018

Rearrange CMake RPATH specifications to be centrally defined

f3ba902

Ref #21788 (cherry picked from commit 78305aa)

ararslan added a commit that referenced this pull request May 2, 2018

Rearrange CMake RPATH specifications to be centrally defined

d5f4d0d

Ref #21788 (cherry picked from commit 78305aa)

ararslan added a commit that referenced this pull request May 8, 2018

Rearrange CMake RPATH specifications to be centrally defined

62ce311

Ref #21788 (cherry picked from commit 78305aa)

ararslan added a commit that referenced this pull request May 27, 2018

Rearrange CMake RPATH specifications to be centrally defined

a833395

Ref #21788 (cherry picked from commit 78305aa)

Workaround for FreeBSD linking to outdated system libs #21788

Workaround for FreeBSD linking to outdated system libs #21788

Conversation

ararslan commented May 11, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ararslan May 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tkelman commented May 13, 2017

iblislin commented May 14, 2017 • edited Loading

ararslan commented May 14, 2017

iblislin commented May 14, 2017

ararslan commented May 14, 2017

iblislin commented May 14, 2017

ararslan commented May 14, 2017 • edited Loading

iblislin commented May 14, 2017

tkelman commented May 14, 2017

iblislin commented May 14, 2017 • edited Loading

ararslan commented May 14, 2017

iblislin commented May 15, 2017 • edited Loading

iblislin commented May 15, 2017

ararslan commented May 15, 2017

iblislin commented May 16, 2017

iblislin commented May 16, 2017

ararslan commented May 16, 2017

iblislin commented May 16, 2017

ararslan commented May 16, 2017

ararslan commented May 17, 2017 • edited Loading

ararslan commented May 18, 2017

iblislin commented May 18, 2017

ararslan commented May 18, 2017

ararslan commented Jun 10, 2017

ararslan May 12, 2017 •

edited

Loading

iblislin commented May 14, 2017 •

edited

Loading

ararslan commented May 14, 2017 •

edited

Loading

iblislin commented May 14, 2017 •

edited

Loading

iblislin commented May 15, 2017 •

edited

Loading

ararslan commented May 17, 2017 •

edited

Loading