Native application gets Killed: 9 on macOS Big Sur with binutils on the path #13856

rsvoboda · 2020-12-12T21:49:29Z

I have a simple project created by maven extension as described in https://quarkus.io/guides/getting-started#bootstrapping-the-project. Running mvn clean verify -Dnative ends with Failed to start native image, process has exited on macOS Big Sur.
EDIT: the main problem is in binutils, see #13856 (comment)

I didn't have this problem on the previous macOS version 10.15 Catalina. I noticed this issue only on macOS Big Sur.

Using GraalVM 20.3.0

native-image --version
GraalVM Version 20.3.0 (Java Version 11.0.9+10-jvmci-20.3-b06)

Running just the native binary gives Killed: 9

target/getting-started-1.0.0-SNAPSHOT-runner
Killed: 9

I tried to remove some metadata from the generated native binary, but that didn't help

sudo xattr -d -r com.apple.quarantine  target/getting-started-1.0.0-SNAPSHOT-runner
target/getting-started-1.0.0-SNAPSHOT-runner
Killed: 9

Native image build went well:

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on GraalVM Version 20.3.0 (Java Version 11.0.9+10-jvmci-20.3-b06)
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] /Users/rsvoboda/.sdkman/candidates/java/20.3.0.r11-grl/bin/native-image -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 -J-Duser.language=en -J-Dfile.encoding=UTF-8 --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy\$BySpaceAndTime -H:+JNI -jar getting-started-1.0.0-SNAPSHOT-runner.jar -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http --no-server -H:-UseServiceLoaderFeature -H:+StackTrace getting-started-1.0.0-SNAPSHOT-runner
[getting-started-1.0.0-SNAPSHOT-runner:1817]    classlist:   2,235.29 ms,  0.96 GB
[getting-started-1.0.0-SNAPSHOT-runner:1817]        (cap):   3,473.40 ms,  1.18 GB
[getting-started-1.0.0-SNAPSHOT-runner:1817]        setup:   5,182.05 ms,  1.18 GB
22:07:27,877 INFO  [org.jbo.threads] JBoss Threads version 3.1.1.Final
[getting-started-1.0.0-SNAPSHOT-runner:1817]     (clinit):     538.76 ms,  2.30 GB
[getting-started-1.0.0-SNAPSHOT-runner:1817]   (typeflow):  11,338.02 ms,  2.30 GB
[getting-started-1.0.0-SNAPSHOT-runner:1817]    (objects):  10,285.37 ms,  2.30 GB
[getting-started-1.0.0-SNAPSHOT-runner:1817]   (features):     479.63 ms,  2.30 GB
[getting-started-1.0.0-SNAPSHOT-runner:1817]     analysis:  23,489.96 ms,  2.30 GB
[getting-started-1.0.0-SNAPSHOT-runner:1817]     universe:   1,402.86 ms,  3.25 GB
[getting-started-1.0.0-SNAPSHOT-runner:1817]      (parse):   3,104.44 ms,  3.25 GB
[getting-started-1.0.0-SNAPSHOT-runner:1817]     (inline):   6,345.91 ms,  3.86 GB
[getting-started-1.0.0-SNAPSHOT-runner:1817]    (compile):  23,821.26 ms,  5.24 GB
[getting-started-1.0.0-SNAPSHOT-runner:1817]      compile:  35,269.13 ms,  5.24 GB
[getting-started-1.0.0-SNAPSHOT-runner:1817]        image:   4,033.51 ms,  5.25 GB
[getting-started-1.0.0-SNAPSHOT-runner:1817]        write:     986.77 ms,  5.25 GB
[getting-started-1.0.0-SNAPSHOT-runner:1817]      [total]:  72,858.51 ms,  5.25 GB
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in 75495ms

The text was updated successfully, but these errors were encountered:

rsvoboda · 2020-12-14T13:18:53Z

@zakkak / @galderz any tips?

zakkak · 2020-12-15T11:19:22Z

@rsvoboda unfortunately no.

Did you check oracle/graal#2984?
It looks like Big Sur is not supported in 20.3.0.

Can you please give 21.0.0-dev a try https://github.com/graalvm/graalvm-ce-dev-builds/releases/ and open an issue in https://github.com/oracle/graal/ if the issue persists?

rsvoboda · 2020-12-15T12:48:57Z

hm, so I am able to compile helloworld from https://www.graalvm.org/docs/getting-started/#native-images using both

GraalVM Version 20.3.0 (Java Version 11.0.9+10-jvmci-20.3-b06)
GraalVM Version 21.0.0-dev (Java Version 11.0.9+10-jvmci-21.0-b03) from https://github.com/graalvm/graalvm-ce-dev-builds/releases/tag/21.0.0-dev-20201215_0301

But using with getting-started based on Quarkus master ends up with Killed: 9 even with the 21.0.0-dev.

Just for the record, I had to sudo xattr -d -r com.apple.quarantine /Users/rsvoboda/Downloads/graalvm-ce-java11-21.0.0-dev because BigSur is super strict about apps from unknown source / internet. gu command was crashing before xattr command.

rsvoboda · 2020-12-15T14:57:42Z

My current workaround is to use container based build of native binaries. I will try to keep an eye on https://github.com/oracle/graal/issues from time to time.

cescoffier · 2021-01-04T08:23:58Z

@rsvoboda I don't have the issue (just tried to build and run the getting started guide). For me, everything works.

rsvoboda · 2021-01-04T11:28:34Z

What version of GraalVM are you using? How did you install it - sdkman / direct download ... ?

cescoffier · 2021-01-04T15:44:25Z

@rsvoboda I use jabba - so jabba install [email protected]

rsvoboda · 2021-01-05T14:41:54Z

This is really strange, I tried jabba way and getting the same Killed: 9

which native-image
/Users/rsvoboda/.jabba/jdk/[email protected]/Contents/Home/bin/native-image
native-image --version
GraalVM Version 20.3.0 (Java Version 11.0.9+10-jvmci-20.3-b06)

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] /Users/rsvoboda/.jabba/jdk/[email protected]/Contents/Home/bin/native-image -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 -J-Duser.language=en -J-Dfile.encoding=UTF-8 --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy\$BySpaceAndTime -H:+JNI -jar getting-started-1.0.0-SNAPSHOT-runner.jar -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http --no-server -H:-UseServiceLoaderFeature -H:+StackTrace getting-started-1.0.0-SNAPSHOT-runner

Edit: dtruss doesn't seem to be the right tool, it fails with curl too
~~Running with dtruss gives:~~
~~sudo dtruss target/getting-started-1.0.0-SNAPSHOT-runner~~

I checked xcode-select --install and command line tools are already installed

Which version of BigSur are you on?
I have 11.1 (20C69)

rsvoboda · 2021-01-05T20:29:58Z

I finally cracked the root cause of my problems. It's iTerm2 terminal in combination with Big Sur :/

I tried system Terminal and everything went well. The difference is that in case of iTerm2 "allow incoming connections" dialog doesn't appear and application get immediately killed.

I tried iTerm2-3_4_4beta2.zip released yesterday and it has the same behavior. So my workaround us to deal with native binaries in system Terminal application.

rsvoboda · 2021-01-05T20:58:49Z

I have filed issue in iterm2 issue tracker

https://gitlab.com/gnachman/iterm2/-/issues/9409

Closing this issue as root cause was identified and workaround (use Terminal application) exists.

galderz · 2021-01-06T07:28:48Z

That dialog is quite a 🤬 . Do you know why it appears? I wonder if it's something that only happens in corporate apple machines. Good catch btw! I've not yet upgraded to Big Sur. Was planning to do it later this month but might hold off a bit longer.

rsvoboda · 2021-01-06T07:51:21Z

That dialog is related to firewall and on corporate machine I can't do much with firewall settings. @cescoffier said he was using iTerm2 too, but he doesn't have the same troubles as I have. I will keep digging, but at least I have workaround ;)

rsvoboda · 2021-01-06T09:09:59Z

So I did more digging because Clement didn't see the same problem I had, iTerm2 is configured to source my ~/.bashrc, that's not the case for Terminal.

I started with bisection of my ~/.bashrc an the problematic line there was export PATH=/usr/local/opt/binutils/bin:$PATH.
So after removing that line and binutils I'm again able to use iTerm2 with native binaries.

brew remove binutils
Uninstalling /usr/local/Cellar/binutils/2.35.1_1... (139 files, 168.6MB)

I installed binutils to be able to get debug symbols for native (objcopy command).

@galderz native-image tries to use something from binutils tools and that's causing my problems.

I tried to remove binutils ++ install it again, did sudo xattr -r -d com.apple.quarantine /usr/local/opt/binutils, also did export LDFLAGS="-L/usr/local/opt/binutils/lib" and export CPPFLAGS="-I/usr/local/opt/binutils/include" in the terminal where I run mvn clean verify -Dnative. So removing binutils is the only option for me atm.

zakkak · 2021-01-06T11:58:09Z

That's interesting. @rsvoboda is this also triggered when compiling a HelloWorld? Do you need to add -g (enable debug symbols generation) to trigger it?

I think we should report it to oracle/graal if it's not Quarkus/Mandrel specific.

rsvoboda · 2021-01-06T12:30:09Z

Hello World from https://www.graalvm.org/docs/getting-started/#native-images worked fine even when I had binutils, see #13856 (comment)

No need to add -g, just build the Quarkus binary and run it.

zakkak · 2021-01-06T12:35:58Z

Hello World from https://www.graalvm.org/docs/getting-started/#native-images worked fine even when I had binutils, see #13856 (comment)

Oops, sorry for not seeing this.

OK so the issue is triggered when Quarkus uses objcopy to strip debug symbols (even when not using -Dquarkus.native.debug.enabled=true) in

quarkus/core/deployment/src/main/java/io/quarkus/deployment/pkg/steps/NativeImageBuildStep.java

Line 299 in a4f84e0

objcopy("--strip-debug", finalPath.toString());

Can you try stripping a C/C++ helloworld with objcopy to see if the same happens?

rsvoboda · 2021-01-06T12:47:26Z

After running objcopy --strip-debug on Hello World binary I get Killed: 9 too

rsvoboda RS $ sdk use java 20.3.0.r11-grl && export GRAALVM_HOME=$JAVA_HOME

Using java version 20.3.0.r11-grl in this shell.

rsvoboda RS $ native-image HelloWorld
[helloworld:95653]    classlist:   1,553.57 ms,  0.96 GB
[helloworld:95653]        (cap):   2,016.78 ms,  0.96 GB
[helloworld:95653]        setup:   3,333.36 ms,  0.96 GB
[helloworld:95653]     (clinit):     170.06 ms,  1.22 GB
[helloworld:95653]   (typeflow):   4,369.42 ms,  1.22 GB
[helloworld:95653]    (objects):   4,178.25 ms,  1.22 GB
[helloworld:95653]   (features):     219.73 ms,  1.22 GB
[helloworld:95653]     analysis:   9,135.71 ms,  1.22 GB
[helloworld:95653]     universe:     350.87 ms,  1.22 GB
[helloworld:95653]      (parse):   1,039.70 ms,  1.22 GB
[helloworld:95653]     (inline):   1,493.86 ms,  1.67 GB
[helloworld:95653]    (compile):   7,670.69 ms,  2.28 GB
[helloworld:95653]      compile:  10,741.89 ms,  2.28 GB
[helloworld:95653]        image:   1,529.52 ms,  2.28 GB
[helloworld:95653]        write:     593.85 ms,  2.28 GB
[helloworld:95653]      [total]:  27,419.89 ms,  2.28 GB
 
rsvoboda RS $ du -h helloworld
7.7M	helloworld

rsvoboda RS $ ./helloworld
Hello, World!

rsvoboda RS $ objcopy --strip-debug helloworld

rsvoboda RS $ du -h helloworld
7.6M	helloworld

rsvoboda RS $ ./helloworld
Killed: 9

zakkak · 2021-01-06T12:49:51Z

OK, so it looks like an objcopy issue.

rsvoboda · 2021-01-06T14:08:29Z

So should be execution of objcopy avoided on macOS BigSur?

System.getProperty("os.name") and System.getProperty("os.version") give os name: Mac OS X and os version: 10.16

zakkak · 2021-01-06T14:13:31Z

That would be an option. In that case we should inform users (through an error message?) that -Dquarkus.native.debug.enabled=true only works in combination with -Dquarkus.native.container-build in macOS BigSur.

galderz · 2021-01-07T10:08:13Z

@zakkak didn't the docu already explained the platforms on which debug symbols are available? I'm unsure about informing them via a message, but I would certainly avoid using objcopy on mac.

zakkak · 2021-01-07T12:35:57Z

We currently explicitly mention installing objcopy on macOS (it was working before BigSur AFAIK)
https://github.com/quarkusio/quarkus/pull/13582/files#diff-faf7fc137fb2be585bdb0ebe4be08bb14542e8c2584c682e23c493500f0d0995R592-R595

So IMO we should change the documentation to reflect that native builds on macOS (without using the builder image) do not support debug symbols generation. In addition to that, I think printing a warning message in case someone tries to generate debug symbols on windows or macOS would improve the user/developer experience and save us some questions/issues :)

buuhsmead · 2021-04-09T19:47:55Z

I am on macOS Big Sur 11.2.3 and the native compiled app got also killed during startup, both under iTerm and Terminal. Removing the objcopy steps as mentioned. Gives for me a working application startup. So I am not using binutils / objcopy any more.

native-image --version
GraalVM Version 21.0.0.2 (Java Version 11.0.10+8-jvmci-21.0-b06)

patped · 2021-09-09T14:00:40Z

Just want you guys to know that this is still an issue and the guide is still pointing macOS users to install objcopy.

The generation of the .debug file depends on objcopy. On common Linux distributions and macOS you will need to install the binutils package:

Is there anything that I'm missing?

galderz · 2021-09-14T07:33:11Z

@patped Looks like some parts of this might have fallen through the cracks.

What's exactly still an issue? Is the binary still getting killed? Or the fact that documentation is misleading? Or both?

Even though the issue was closed and from what I can see from the commits associated with this, I don't see the macOS aspects where addressed. It seems objcopy should not be used at all on macos and the documentation. Is that right?

patped · 2021-09-15T19:09:59Z

What's exactly still an issue? Is the binary still getting killed? Or the fact that documentation is misleading? Or both?

Yes, the binary is still getting killed if I have objcopy installed on macOS Big Sur v.11.6. So I feel the documentation is misleading on this one.

Even though the issue was closed and from what I can see from the commits associated with this, I don't see the macOS aspects where addressed. It seems objcopy should not be used at all on macos and the documentation. Is that right?

I'm not sure if objcopy can't be used at all with macOS but it's not working out of the box with the current state of the bootstrapped project you get from the command:

mvn io.quarkus.platform:quarkus-maven-plugin:2.2.2.Final:create \
    -DprojectGroupId=org.acme \
    -DprojectArtifactId=getting-started \
    -DclassName="org.acme.getting.started.GreetingResource" \
    -Dpath="/hello"

galderz · 2021-09-21T08:38:26Z

@patped I'm looking into it :)

* `objcopy` invocations in macOS make the executable crash on startup. * Debug info is only available for Linux, so avoid `objcopy` altogether on macOS, even if present in PATH. * Updated documentation to avoid confusion.

galderz · 2021-09-21T10:33:28Z

@patped Sent a PR to fix it once and for all (see #20297), but there's a very easy workaround here: Either do brew uninstall binutils, or remove it from the default PATH and only bring it into PATH when you really need it.

* `objcopy` invocations in macOS make the executable crash on startup. * Debug info is only available for Linux, so avoid `objcopy` altogether on macOS, even if present in PATH. * Updated documentation to avoid confusion. * Adjusted warning messages about `objcopy`.

* `objcopy` invocations in macOS make the executable crash on startup. * Debug info is only available for Linux, so avoid `objcopy` altogether on macOS, even if present in PATH. * Updated documentation to avoid confusion. * Adjusted warning messages about `objcopy`. (cherry picked from commit 7c12f02)

patped · 2021-10-01T20:03:39Z

Wow! Thanks for the quick responses and action! 🥇

I almost feel terrible now, responding so late. 😅 👏

* `objcopy` invocations in macOS make the executable crash on startup. * Debug info is only available for Linux, so avoid `objcopy` altogether on macOS, even if present in PATH. * Updated documentation to avoid confusion. * Adjusted warning messages about `objcopy`. (cherry picked from commit 7c12f02)

rsvoboda added the kind/bug Something isn't working label Dec 12, 2020

ghost added the triage/needs-triage label Dec 12, 2020

rsvoboda changed the title ~~Native application gets Killed: 9 on macOS Big Sur~~ Native application gets Killed: 9 on macOS Big Sur + iTerm2 Jan 5, 2021

rsvoboda closed this as completed Jan 5, 2021

rsvoboda changed the title ~~Native application gets Killed: 9 on macOS Big Sur + iTerm2~~ Native application gets Killed: 9 on macOS Big Sur with binutils on the path Jan 6, 2021

gsmet reopened this Jan 6, 2021

bufdev mentioned this issue Jan 20, 2021

buf cli is killed immediately on macOS Big Sur bufbuild/buf#249

Closed

zakkak mentioned this issue Apr 12, 2021

Use builder image's objcopy when building with quarkus.native.container-build=true and quarkus.native.debug.enabled=true #13963

Merged

gsmet closed this as completed in #13963 Apr 22, 2021

quarkus-bot bot removed the triage/needs-triage label Apr 22, 2021

quarkus-bot bot added this to the 2.0 - main milestone Apr 22, 2021

boojongmin mentioned this issue Jul 15, 2021

quarkus boojongmin/memo#16

Open

galderz mentioned this issue Sep 21, 2021

Only use objcopy in Linux environments #20297

Merged

jponge mentioned this issue Oct 6, 2021

Upgrade to Mutiny 1.1.0 and Mutiny Vert.x bindings 2.14.0 jponge/quarkus#57

Closed

cescoffier mentioned this issue Dec 14, 2021

Update to Vert.x 4.1.7 #22212

Merged

This was referenced Jan 5, 2022

[2.2] Update Vert.x to version 4.1.8 #22638

Merged

[2.2] Protobuf Java and gRPC version updates (CVE-2021-22569) #22761

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Native application gets Killed: 9 on macOS Big Sur with binutils on the path #13856

Native application gets Killed: 9 on macOS Big Sur with binutils on the path #13856

rsvoboda commented Dec 12, 2020 •

edited

Loading

rsvoboda commented Dec 14, 2020

zakkak commented Dec 15, 2020

rsvoboda commented Dec 15, 2020

rsvoboda commented Dec 15, 2020

cescoffier commented Jan 4, 2021

rsvoboda commented Jan 4, 2021

cescoffier commented Jan 4, 2021

rsvoboda commented Jan 5, 2021 •

edited

Loading

rsvoboda commented Jan 5, 2021

rsvoboda commented Jan 5, 2021

galderz commented Jan 6, 2021

rsvoboda commented Jan 6, 2021 •

edited

Loading

rsvoboda commented Jan 6, 2021

zakkak commented Jan 6, 2021

rsvoboda commented Jan 6, 2021

zakkak commented Jan 6, 2021

rsvoboda commented Jan 6, 2021

zakkak commented Jan 6, 2021

rsvoboda commented Jan 6, 2021

zakkak commented Jan 6, 2021

galderz commented Jan 7, 2021 •

edited

Loading

zakkak commented Jan 7, 2021

buuhsmead commented Apr 9, 2021

patped commented Sep 9, 2021

galderz commented Sep 14, 2021

patped commented Sep 15, 2021 •

edited

Loading

galderz commented Sep 21, 2021

galderz commented Sep 21, 2021

patped commented Oct 1, 2021

Native application gets Killed: 9 on macOS Big Sur with binutils on the path #13856

Native application gets Killed: 9 on macOS Big Sur with binutils on the path #13856

Comments

rsvoboda commented Dec 12, 2020 • edited Loading

rsvoboda commented Dec 14, 2020

zakkak commented Dec 15, 2020

rsvoboda commented Dec 15, 2020

rsvoboda commented Dec 15, 2020

cescoffier commented Jan 4, 2021

rsvoboda commented Jan 4, 2021

cescoffier commented Jan 4, 2021

rsvoboda commented Jan 5, 2021 • edited Loading

rsvoboda commented Jan 5, 2021

rsvoboda commented Jan 5, 2021

galderz commented Jan 6, 2021

rsvoboda commented Jan 6, 2021 • edited Loading

rsvoboda commented Jan 6, 2021

zakkak commented Jan 6, 2021

rsvoboda commented Jan 6, 2021

zakkak commented Jan 6, 2021

rsvoboda commented Jan 6, 2021

zakkak commented Jan 6, 2021

rsvoboda commented Jan 6, 2021

zakkak commented Jan 6, 2021

galderz commented Jan 7, 2021 • edited Loading

zakkak commented Jan 7, 2021

buuhsmead commented Apr 9, 2021

patped commented Sep 9, 2021

galderz commented Sep 14, 2021

patped commented Sep 15, 2021 • edited Loading

galderz commented Sep 21, 2021

galderz commented Sep 21, 2021

patped commented Oct 1, 2021

rsvoboda commented Dec 12, 2020 •

edited

Loading

rsvoboda commented Jan 5, 2021 •

edited

Loading

rsvoboda commented Jan 6, 2021 •

edited

Loading

galderz commented Jan 7, 2021 •

edited

Loading

patped commented Sep 15, 2021 •

edited

Loading