cconv.el.gz fails to compile due to missing libgcc_ext.10.5 #5

jimeh · 2020-08-20T19:59:47Z

Currently cconv.el.gz that's part of Emacs fails to natively compile for me,
leading to a error/warning at start up. It also leaves a zero byte sized *.eln
file behind, which can be dealt by with the work around in #3.

As far as I can tell this doesn't seem to actually break anything or prevent
things from working correctly though.

Details

The compile log error:

Compiling /Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/cconv.el.gz...
uncompressing cconv.el.gz...
uncompressing cconv.el.gz...done
ld: library not found for -lgcc_ext.10.5
libgccjit.so: error: error invoking gcc driver

Initially I thought it was simply missing the libgcc_ext.10.5.dylib from GCC,
as other libs are bundled into Emacs.app, but libgcc_ext was not. However
manually adding it to the libs to embed/bundle yields this error:

fatal error: /Library/Developer/CommandLineTools/usr/bin/install_name_tool: input file: /Users/jimeh/Projects/build-emacs-for-macos/sources/emacs-mirror-emacs-c818c29/nextstep/Emacs.app/Contents/MacOS/lib-x86_64-10.15/libgcc_ext.10.5.dylib is Mach-O dynamic shared library stub file and can't be changed

Sadly some googling for information about this kind of error and possible
workarounds didn't yield much.

So I tried not bundling libgccjit into Emacs.app, and instead referencing it
from the homebrew install path, but that didn't solve the issue either.

For now I'm out of ideas for how to fix or workaround this.

Update: It seems that cconv.el.gz compiles just fine on my machine if I
launch Emacs into terminal mode:

/Applications/Emacs.app/Contents/MacOS/Emacs -nw

I'll try and dig into why it fails when launched with GUI, but doesn't in
terminal.

The text was updated successfully, but these errors were encountered:

jimeh · 2020-08-21T22:35:03Z

At some point today cconv.el.gz was natively compiled for me, cause I now have
a *.eln file for it. I'm still running the exact same build of Emacs as
yesterday, so I'm not sure what's going on 🤔

jakejx · 2020-08-27T10:31:36Z

I am also experiencing the same error but for subr.el. This causes failed compilation for all other packages. I'm still looking into what the cause is.

jimeh · 2020-08-27T10:34:50Z

@jakejx if it helps, cconv.el.gz has been constantly failing for me again since I switched to a newer build after that one time it did compile. It must have been some weird fluke that it didn't fail the one time ¯\_(ツ)_/¯

jakejx · 2020-08-27T12:16:07Z

hmm not sure if it matters, but I am seeing it for a different file and its causing subsequent errors:

Compiling /Users/junxuan/Downloads/build-emacs-for-macos/builds/Emacs.app/Contents/Resources/lisp/emacs-lisp/subr-x.el.gz... uncompressing subr-x.el.gz... uncompressing subr-x.el.gz...done ld: library not found for -lgcc_ext.10.5 libgccjit.so: error: error invoking gcc driver Native elisp load failed: "/Users/junxuan/.emacs.d/eln-cache/x86_64-apple-darwin19.4.0-ef3bf8ba088c7b38/subr-x-7ac74ad07543ba0e24902ea7dd76ea64-e56611d5b77b59b8916db00f6e48777a.eln", "dlopen(/Users/junxuan/.emacs.d/eln-cache/x86_64-apple-darwin19.4.0-ef3bf8ba088c7b38/subr-x-7ac74ad07543ba0e24902ea7dd76ea64-e56611d5b77b59b8916db00f6e48777a.eln, 1): no suitable image found. Did find: /Users/junxuan/.emacs.d/eln-cache/x86_64-apple-darwin19.4.0-ef3bf8ba088c7b38/subr-x-7ac74ad07543ba0e24902ea7dd76ea64-e56611d5b77b59b8916db00f6e48777a.eln: file too short /Users/junxuan/.emacs.d/eln-cache/x86_64-apple-darwin19.4.0-ef3bf8ba088c7b38/subr-x-7ac74ad07543ba0e24902ea7dd76ea64-e56611d5b77b59b8916db00f6e48777a.eln: file too short"

And I get multiple of the Native elisp load failed for all the libraries it tries to natively compile.

jimeh · 2020-08-27T12:25:42Z

@jakejx those errors actually look related to #3, and to some degree #4 as well, cause it's trying to load *.eln files from within Emacs.app itself, and my config workaround on #3 won't be removing empty *.eln files from inside of Emacs.app, just the user-specific eln-cache folder.

That said, I've not actually tried to run the Emacs.app from the builds folder myself, which means due to #4 I've not actually tried starting it with the *.eln files compiled at build time, instead it's always had to re-compile all things, including Lisp that's part of Emacs.

I'd be curious to see what happens if you copy Emacs.app to a different path, so the cached *.eln files bundled into the app are bypassed and everything is native compiled from scratch.

bramadams · 2020-08-27T14:17:37Z

Hi,

I've had the same issue regarding libgcc_ext.10.5, which seems to be solved by putting the following in your early init.el (see https://gist.github.com/mikroskeem/0a5c909c1880408adf732ceba6d3f9ab#gistcomment-3390480):

(setenv "LIBRARY_PATH" "/usr/local/opt/gcc/lib/gcc/10")

However, now emacs complains about a similar issue for "-lgcc":

ld: library not found for -lgcc

I don't find libgcc in the compiled GCC, only libgcc_s. Any ideas about this?

jakejx · 2020-08-27T14:39:02Z

@jimeh I think Emacs is trying to recompile everything from scratch, but its not able to find libgccjit.so. Which is odd because it could find it during the initial compilation.

jimeh · 2020-08-27T19:20:37Z

I seem to have find out how cconv.el was native compiled on my machine that one time. If I launch Emacs in a terminal (/Applications/Emacs.app/Contents/MacOS/Emacs -nw) it compiles it, while if I launch it as GUI app, it fails. I haven't gotten to the bottom of why yet, but I'll try and dig a bit deeper when I have time.

jimeh · 2020-08-27T19:21:42Z

@jakejx Yep, as detailed in #4, it generally need to re-compile all elisp code if the Emacs.app is moved, as the eln-cache files have a hash based on the absolute path of the lisp file being compiled.

jimeh · 2020-08-27T19:25:16Z

@bramadams I'm not sure that actually fixes the issue, looks more like it just breaks things even worse. I mentioned in my earlier comment though, it cconv.el compiles for me if I launch emacs in terminal mode, so I'd suggest trying that :)

bramadams · 2020-08-27T20:18:58Z

@jimeh Yes, I just tried, and the terminal version compiles natively. Reloading the GUI app seems to use the compiled files from the cache, but since compilation does not work at all for me in the GUI app ("library not found for -lgcc_ext.10.5" ), there are still a lot of "no suitable image found" messages. In any case, there's progress.

jimeh · 2020-08-27T20:26:37Z

Strange, in GUI when I get that error, it only prevents cconv.el.gz from compiling, everything else native compiles just fine.

I have even repeatedly reproduce the behavior by manually trashing the cconv eln-cache file, and some others, which causes the GUI to fail compiling cconv, but the rest compile again without issue.

For reference, I'm on latest macOS (10.15.6) and Xcode (11.6).

bramadams · 2020-08-27T20:28:56Z

OK, thanks, I'm on 10.13.6 and Xcode 10.1. I can't upgrade OSX, but I'll try upgrading Xcode.

bramadams · 2020-08-27T21:02:36Z

... and by the way, thanks for making these scripts available, never got this far with native compilation :-)

jakejx · 2020-08-27T21:13:38Z

I can confirm that performing the initial compilation using the terminal version has resolved the issue for me. Any compilation that arises when using the GUI after that seems to work fine. Thanks for the scripts @jimeh!

bramadams · 2020-08-28T05:05:49Z

Small update: the GUI app has native compilation fully functional when I invoke it from the command line. Only when opening the app via Finder, the libgcc_ext problem persists. It must be some kind of environment issue with app bundles.

jakejx · 2020-08-30T14:20:30Z

Hi,

I've had the same issue regarding libgcc_ext.10.5, which seems to be solved by putting the following in your early init.el (see https://gist.github.com/mikroskeem/0a5c909c1880408adf732ceba6d3f9ab#gistcomment-3390480):

(setenv "LIBRARY_PATH" "/usr/local/opt/gcc/lib/gcc/10")

However, now emacs complains about a similar issue for "-lgcc":

ld: library not found for -lgcc

I don't find libgcc in the compiled GCC, only libgcc_s. Any ideas about this?

Just an update on this. By adding one more path to LIBRARY_PATH I am able to resolve the issues. Based on what I understand from https://gcc.gnu.org/onlinedocs/jit/internals/index.html#environment-variables the path to libgcc subdirectory also needs to be set. On my system, using (setenv "LIBRARY_PATH" "/usr/local/Cellar/gcc/10.2.0/lib/gcc/10:/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin19/10.2.0") seems to do the trick.

I am not sure if there is anyway for us to specify these during the build process. Alternatively, is there some emacs variable that we can use to get the path of Emacs itself, so users can set these environment variables relative to the app bundle instead of the system gcc.

jimeh · 2020-08-30T14:40:57Z

@jakejx Thats some awesome digging, thanks :D

I'll give it a try myself, and also experiment with moving those folders into the Emacs.app itself, if that works I can have the built script do it automatically, then it's just a matter of figuring out if the relevant (setenv ...) can be baked into the the app too.

Worst case a work around could be to require users to update their config with it. Some config tweaks are already needed anyway.

jimeh · 2020-08-31T00:41:18Z

I've managed to hand-craft a Emacs.app which is fully self-contained with gcc/libgccjit, and sets the LIBRARY_PATH environment variable correctly without any user config needed. cconv.el.gz compiles just fine when launching it by double clicking in Finder.

I'll update the build script accordingly tomorrow, and hopefully it should resolve both #5 and #7 :)

The solution to get libgccjit properly working, and embedded in Emacs.app included: - The contents of GCC's lib folder (`/usr/local/opt/gcc/lib`) is copied into the `Contents/MacOS/lib-<arch>-<os_version>` folder. - Setting `LIBRARY_PATH` environment variable to correct GCC lib folders within Emacs.app. This is done through a bash launcher script which replaces the regular `Contents/MacOS/Emacs` executable. The main Emacs executable itself is named `Emacs-bin` now instead, so anything that depends on the exact process name will need updating. - The launcher script also adds `Content/MacOS/bin` and `Content/MacOS/libexec` folders to the PATH environment variable, to so ensure binary tools packaged into Emacs itself are available. This is done even when not doing a native-comp build. The launcher script skips setting LIBRARY_PATH if it's not a native-comp build. This should hopefully resolve both #5 and #7.

jimeh · 2020-09-03T01:28:58Z

@jakejx @bramadams I've just created Pull Request #10 that updates the build
script to embed GCC into Emacs.app along with also setting LIBRARY_PATH
correctly on startup without any user config changes through the use of a custom
launcher script.

If you wouldn't mind trying it out, that'd be great :)

Everything seems to work for me, even after I move
/usr/local/Cellar/gcc/10.2.0 out of the way. I did also try it on an older mac
I have which also runs macOS 10.15.6 like my main machine. But it failed to
start due to weird unsupported architecture flag in the executable. It's CPU is
a 2 or 3 generations older than then CPU in my current machine. So seems I'd
need to test by compiling Emacs on the machine, but that kinda defeats the
purposes to ensure the app is fully self-contained.

The solution to get libgccjit properly working, and embedded in Emacs.app included: - The contents of GCC's lib folder (`/usr/local/opt/gcc/lib`) is copied into the `Contents/MacOS/lib-<arch>-<os_version>` folder. - Setting `LIBRARY_PATH` environment variable to correct GCC lib folders within Emacs.app. This is done through a bash launcher script which replaces the regular `Contents/MacOS/Emacs` executable. The main Emacs executable itself is named `Emacs-bin` now instead, so anything that depends on the exact process name will need updating. - The launcher script also adds `Content/MacOS/bin` and `Content/MacOS/libexec` folders to the PATH environment variable, to so ensure binary tools packaged into Emacs itself are available. This is done even when not doing a native-comp build. The launcher script skips setting LIBRARY_PATH if it's not a native-comp build. This should hopefully resolve both #5 and #7.

jimeh · 2020-09-05T04:19:07Z

This should now be fixed thanks to #10.

goofansu · 2020-11-25T03:20:55Z

Hi, thank you for the script.

I can successfully build out the app with ./build-emacs-for-macos -j2 feature/native-comp. But it reports faild to compile in the Warnings buffer:

Warning (comp): libgccjit.so: error: error invoking gcc driver Disable showing Disable logging
Warning (comp): Internal native compiler error: "/Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/cl-lib.el.gz", "failed to compile", "/Users/james/.emacs.d/eln-cache/28.0.50-x86_64-apple-darwin19.6.0-deda61b35cbed626661f68bc7cd413a9/cl-lib-8b9389005ae118dfb032ca52ea69e8d9-28984542edce56cce7e53ec8ccd624af.eln", "error invoking gcc driver" Disable showing Disable logging
Warning (comp): seq.el.gz:395:16: Warning: ‘seq-contains’ is an obsolete generic function (as Disable showing Disable logging
Warning (comp): libgccjit.so: error: error invoking gcc driver Disable showing Disable logging
Warning (comp): Internal native compiler error: "/Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/seq.el.gz", "failed to compile", "/Users/james/.emacs.d/eln-cache/28.0.50-x86_64-apple-darwin19.6.0-deda61b35cbed626661f68bc7cd413a9/seq-c314b42f12831596cfc86f40bcdbfc80-5a24074f219b51ac0086349bdb7f5622.eln", "error invoking gcc driver" Disable showing Disable logging
Warning (comp): libgccjit.so: error: error invoking gcc driver Disable showing Disable logging
Warning (comp): Internal native compiler error: "/Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/gv.el.gz", "failed to compile", "/Users/james/.emacs.d/eln-cache/28.0.50-x86_64-apple-darwin19.6.0-deda61b35cbed626661f68bc7cd413a9/gv-e0cf7478c665ac9dd903b899257854d6-07285e7deb1fdfdad07c352586e0d756.eln", "error invoking gcc driver" Disable showing Disable logging
Warning (comp): libgccjit.so: error: error invoking gcc driver Disable showing Disable logging
Warning (comp): Internal native compiler error: "/Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/cl-extra.el.gz", "failed to compile", "/Users/james/.emacs.d/eln-cache/28.0.50-x86_64-apple-darwin19.6.0-deda61b35cbed626661f68bc7cd413a9/cl-extra-d7051cba75a4863ffba77470a08a226d-d670409d2537bb7bf4fee93d0922c469.eln", "error invoking gcc driver" Disable showing Disable logging
Warning (comp): libgccjit.so: error: error invoking gcc driver Disable showing Disable logging
Warning (comp): Internal native compiler error: "/Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/easymenu.el.gz", "failed to compile", "/Users/james/.emacs.d/eln-cache/28.0.50-x86_64-apple-darwin19.6.0-deda61b35cbed626661f68bc7cd413a9/easymenu-1fb41c0e78f9aa8e9de83e9219437950-c3486f53c54967059ad37854fd3845bd.eln", "error invoking gcc driver" Disable showing Disable logging
Warning (comp): libgccjit.so: error: error invoking gcc driver Disable showing Disable logging
Warning (comp): Internal native compiler error: "/Applications/Emacs.app/Contents/Resources/lisp/help-mode.el.gz", "failed to compile", "/Users/james/.emacs.d/eln-cache/28.0.50-x86_64-apple-darwin19.6.0-deda61b35cbed626661f68bc7cd413a9/help-mode-d4dbae3d39e6a79cefe8c3ec980fa59d-13fbdfb7fa0b4403fd118315b412e149.eln", "error invoking gcc driver" Disable showing Disable logging
...
...

build-emacs-for-macos: 9223ff8
macOS: 10.15.7

Update

~/.emacs.d (develop|✚3) [255] $ doom install
ld: library not found for -lgcc_ext.10.5
libgccjit.so: error: error invoking gcc driver

Update 2:

~/.emacs.d (develop|✚3) [255] $ env LIBRARY_PATH=/usr/local/opt/gcc/lib/gcc/10/ doom install
ld: library not found for -lgcc
libgccjit.so: error: error invoking gcc driver
ld: library not found for -lgcc
libgccjit.so: error: error invoking gcc driver
No catch for tag: exit, 255

Problem resolved: Thanks to #5 (comment), I set LIBRARY_PATH to /usr/local/Cellar/gcc/10.2.0/lib/gcc/10:/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin18/10.2.0, and it works.

meain · 2020-12-06T07:42:57Z

For some reason setting library path as the one in /usr/local/Cellar did not seem to help me. But I could see that there was lib directory in the Emacs.app package and setting that as LIBRARY_PATH seemed to help. If you have Emacs.app in your /Applications folder add this to the early-init.el file.

(setenv "LIBRARY_PATH" "/Applications/Emacs.app/Contents/MacOS/lib/gcc/10:/Applications/Emacs.app/Contents/MacOS/lib/gcc/10/gcc/x86_64-apple-darwin17/10.2.0")

It looks like patches/native-comp-env-setup.patch) is doing this but for some reason it does not seem to work.

jimeh added the native-comp label Aug 20, 2020

jimeh changed the title ~~cconv.el.gz fails to compile due to missing libgcc_ext.10.5~~ cconv.el.gz fails to compile due to missing libgcc_ext.10.5 Aug 20, 2020

jimeh added the bug label Aug 20, 2020

jimeh mentioned this issue Aug 31, 2020

libgccjit is not bundled into Emacs.app properly #7

Closed

jimeh mentioned this issue Sep 3, 2020

feat(native_comp): embedd gcc/libgccjit into Emacs.app #10

Merged

jimeh closed this as completed Sep 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cconv.el.gz fails to compile due to missing libgcc_ext.10.5 #5

cconv.el.gz fails to compile due to missing libgcc_ext.10.5 #5

jimeh commented Aug 20, 2020 •

edited

Loading

jimeh commented Aug 21, 2020

jakejx commented Aug 27, 2020

jimeh commented Aug 27, 2020

jakejx commented Aug 27, 2020 •

edited

Loading

jimeh commented Aug 27, 2020

bramadams commented Aug 27, 2020

jakejx commented Aug 27, 2020

jimeh commented Aug 27, 2020

jimeh commented Aug 27, 2020

jimeh commented Aug 27, 2020

bramadams commented Aug 27, 2020

jimeh commented Aug 27, 2020

bramadams commented Aug 27, 2020

bramadams commented Aug 27, 2020

jakejx commented Aug 27, 2020

bramadams commented Aug 28, 2020

jakejx commented Aug 30, 2020

jimeh commented Aug 30, 2020

jimeh commented Aug 31, 2020

jimeh commented Sep 3, 2020

jimeh commented Sep 5, 2020

goofansu commented Nov 25, 2020 •

edited

Loading

meain commented Dec 6, 2020

cconv.el.gz fails to compile due to missing libgcc_ext.10.5 #5

cconv.el.gz fails to compile due to missing libgcc_ext.10.5 #5

Comments

jimeh commented Aug 20, 2020 • edited Loading

Details

jimeh commented Aug 21, 2020

jakejx commented Aug 27, 2020

jimeh commented Aug 27, 2020

jakejx commented Aug 27, 2020 • edited Loading

jimeh commented Aug 27, 2020

bramadams commented Aug 27, 2020

jakejx commented Aug 27, 2020

jimeh commented Aug 27, 2020

jimeh commented Aug 27, 2020

jimeh commented Aug 27, 2020

bramadams commented Aug 27, 2020

jimeh commented Aug 27, 2020

bramadams commented Aug 27, 2020

bramadams commented Aug 27, 2020

jakejx commented Aug 27, 2020

bramadams commented Aug 28, 2020

jakejx commented Aug 30, 2020

jimeh commented Aug 30, 2020

jimeh commented Aug 31, 2020

jimeh commented Sep 3, 2020

jimeh commented Sep 5, 2020

goofansu commented Nov 25, 2020 • edited Loading

meain commented Dec 6, 2020

jimeh commented Aug 20, 2020 •

edited

Loading

jakejx commented Aug 27, 2020 •

edited

Loading

goofansu commented Nov 25, 2020 •

edited

Loading