Quarkus NativeImageBuildStep fails with perm denied with podman 2.x #10637

jerboaa · 2020-07-10T10:12:14Z

Describe the bug
This bug originated here:
https://bugzilla.redhat.com/show_bug.cgi?id=1854738
and here:
https://groups.google.com/g/quarkus-dev/c/8LVoqk4G5uc/m/M5MAVopLAwAJ

Basically with podman 1.x --userns=keep-id contained a bug which got fixed with 2.x which results in quarkus build failures with options -Pnative -Dquarkus.native.container-build=true -Dquarkus.native.container-runtime=podman in rootless mode.

Expected behavior
Native image build passes. No "Permission Denied" error.

Actual behavior
Native image build in the container with mounted volume fails due to "Permission Denied".

To Reproduce
Steps to reproduce the behavior:

Install podman 2.0 (for example in F32)
mvn package -Pnative -Dquarkus.native.container-build=true -Dquarkus.native.container-runtime=podman

Additional context
Looking at this podman bug:
containers/podman#3990

... it seems podman won't help with getting permissions correctly set up for volume mounts.

My experiments showed that doing some podman unshare tricks prior the native image build might work. Something like this:

$ CONTAINER_UID=$(podman run --rm --entrypoint /bin/bash -ti quay.io/quarkus/ubi-quarkus-native-image:19.3.1-java11 -c 'id -u' | sed 's/\r//g')
$ podman unshare chown -R $CONTAINER_UID target/getting-started-1.0-SNAPSHOT-native-image-source-jar
$ podman run -v getting-started/target/getting-started-1.0-SNAPSHOT-native-image-source-jar:/project:z --env LANG=C --rm quay.io/quarkus/ubi-quarkus-native-image:19.3.1-java11 -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 -J-Duser.language=en -J-Dfile.encoding=UTF-8 --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy\$BySpaceAndTime -H:+JNI -jar getting-started-1.0-SNAPSHOT-runner.jar -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:-IncludeAllTimeZones -H:EnableURLProtocols=http --no-server -H:-UseServiceLoaderFeature -H:+StackTrace getting-started-1.0-SNAPSHOT-runner

Getting the UID the container runs as that way is important as it might change.

The text was updated successfully, but these errors were encountered:

rmarting · 2020-07-27T11:19:27Z

Same failure error with quay.io/quarkus/ubi-quarkus-native-image:20.1.0-java11 image and podman version 2.0.3.

galderz · 2020-07-28T13:35:21Z

Assigning this to myself.

galderz · 2020-07-29T10:21:29Z

I'm unassigning myself from this because I won't be able to work on in for the next week or so. I've tried to replicate it on macOS but that won't work because you can't run podman on top of docker. Also tried centos 7 server, but podman version there is only 1.x. Finally tried fedora 32 aarch64 but the quarkus images don't support that arch (see quarkusio/quarkus-images#83). This should be better handled by someone with amd64 env with a more recent linux version.

zakkak · 2020-07-29T12:58:59Z

Unfortunately podman unshare chown -R $CONTAINER_UID path will leave you with a path that the host user won't be able to edit/delete and will need podman unshare rm -rf path to achieve it. So after the build is complete we need to chown the path back to the host user.

zakkak · 2020-07-30T13:45:53Z

Note that if the image is using the root user this issue doesn't appear:

$ mkdir project
$ podman run -it --rm -v $PWD/project:/project:z --entrypoint /bin/bash fedora:32 -c 'id; touch /project/lala'
uid=0(root) gid=0(root) groups=0(root)
$ ls -la project/lala 
-rw-r--r--. 1 zakkak zakkak 0 Jul 30 16:41 project/lala

vs

$ mkdir project
$ podman run -it --rm -v $PWD/project:/project:z --entrypoint=/bin/bash quay.io/quarkus/ubi-quarkus-mandrel:20.1.0.1.Alpha2-java11 -c 'id; touch /project/lala'

uid=1001(quarkus) gid=1001(quarkus) groups=1001(quarkus)
touch: cannot touch '/project/lala': Permission denied
$ ls -la project/lala
ls: cannot access 'project/lala': No such file or directory

So an alternative would be to generate images that use the root user instead of the quarkus user

zakkak · 2020-07-30T13:50:17Z

@cescoffier would using the root user in the images cause any issues?

gsmet · 2020-07-30T13:56:44Z

@geoand do you know who could look at that one?

geoand · 2020-07-30T13:57:46Z

@gastaldi maybe? I have never used podman, otherwise I would take a look myself

gsmet · 2020-07-30T13:58:19Z

@Ladicek maybe?

gastaldi · 2020-07-30T14:01:54Z

I've played with Podman once, I can have a look if no one beats me to it

geoand · 2020-07-30T14:03:55Z

That's better than me, none and I don't think I can even install it :)

gastaldi · 2020-07-30T14:07:46Z

I wonder if tqvarnst@0b52b07 actually fixes this. @tqvarnst can you confirm?

zakkak · 2020-07-30T14:09:28Z

I wonder if tqvarnst@0b52b07 actually fixes this. @tqvarnst can you confirm?

It looks like it applies @jerboaa 's suggestion, but it still needs to reset the ownership at the end (see #10637 (comment))

Ladicek · 2020-07-30T14:16:17Z

Just like @geoand, I have exactly zero experience with Podman.

zakkak · 2020-07-30T14:35:40Z

I think I've got a solution (we essentially need to add -u=$(id -u):$(id -g)):

$ mkdir project
$ podman run -it --rm -v $PWD/project:/project:z --userns=keep-id -u=$(id -u):$(id -g) --entrypoint=/bin/bash quay.io/quarkus/ubi-quarkus-mandrel:20.1.0.1.Alpha2-java11 -c 'id; touch /project/lala'
uid=1000(1000) gid=1000 groups=1000
$ ls -la project/lala 
-rw-r--r--. 1 zakkak zakkak 0 Jul 30 17:34 project/lala
$ rm project/lala

geoand · 2020-07-30T14:46:16Z

@zakkak if that works, please open a PR and ask @rsvoboda to test

gastaldi · 2020-07-30T15:06:25Z

@zakkak is it OK if I assign this issue to you?

zakkak · 2020-07-30T15:08:14Z

@gastaldi Yes. I do have something working, just polishing it and I will soon open a PR.

raffaelespazzoli · 2020-11-26T15:19:46Z

hello, I'm having this issue with 1.8.3.

zakkak · 2020-11-27T16:27:08Z

@raffaelespazzoli can you please share info on how to reproduce it? Using code.quarkus.io/api.download and editing the pom.xml file to use 1.8.3 works as expected for me.

raffaelespazzoli · 2020-11-27T16:47:03Z

the link doesn't work can you give me the link of the app? I was using the following tutorial: http://web-m3-quarkus-bjrvk-guides.apps.cluster-952wq.952wq.sandbox763.opentlc.com/workshop/quarkus-lab/lab/springstrangle but I am convinced it can be reproduced with any project. It seems to me the issue depends on the combination of OS and podman. I my case: FC 33 podman version 2.1.1 and podman is aliased to docker.

…

On Fri, Nov 27, 2020 at 11:27 AM Foivos ***@***.***> wrote: @raffaelespazzoli <https://github.com/raffaelespazzoli> can you please share info on how to reproduce it? Using code.quarkus.io/api.download and editing the pom.xml file to use 1.8.3 works as expected for me. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10637 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPERXDOESYSKC5OODGQRELSR7HOZANCNFSM4OWOM6YQ> .

-- ciao/bye Raffaele

zakkak · 2020-11-27T16:55:09Z

I did

curl -O -J  https://code.quarkus.io/api/download
unzip code-with-quarkus.zip
cd code-with-quarkus
sed -i 's/1.9.2.Final/1.8.3.Final/g' pom.xml
./mvnw package -Pnative -Dquarkus.native.container-runtime=podman -Dquarkus.native.container-build=true

$ ls target        
classes/                              code-with-quarkus-1.0.0-SNAPSHOT-native-image-source-jar/  generated-sources/  maven-status/
code-with-quarkus-1.0.0-SNAPSHOT.jar  code-with-quarkus-1.0.0-SNAPSHOT-runner*                   maven-archiver/     quarkus-app/
$ ./target/code-with-quarkus-1.0.0-SNAPSHOT-runner 
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2020-11-27 18:55:38,086 INFO  [io.quarkus] (main) code-with-quarkus 1.0.0-SNAPSHOT native (powered by Quarkus 1.8.3.Final) started in 0.012s. 
2020-11-27 18:55:38,087 INFO  [io.quarkus] (main) Profile prod activated. 
2020-11-27 18:55:38,087 INFO  [io.quarkus] (main) Installed features: [cdi]
hello commando
2020-11-27 18:55:38,087 INFO  [io.quarkus] (main) code-with-quarkus stopped in 0.000s

I am on Fedora 33 with podman 2.1.1 as well

raffaelespazzoli · 2020-11-27T16:57:35Z

that server has been retired, hashidoc of the tutorial can be found here: https://github.com/RedHat-Middleware-Workshops/quarkus-workshop-m1m2/tree/rhtr2020/docs On Fri, Nov 27, 2020 at 11:46 AM raffaele spazzoli < [email protected]> wrote:

…

the link doesn't work can you give me the link of the app? I was using the following tutorial: http://web-m3-quarkus-bjrvk-guides.apps.cluster-952wq.952wq.sandbox763.opentlc.com/workshop/quarkus-lab/lab/springstrangle but I am convinced it can be reproduced with any project. It seems to me the issue depends on the combination of OS and podman. I my case: FC 33 podman version 2.1.1 and podman is aliased to docker. On Fri, Nov 27, 2020 at 11:27 AM Foivos ***@***.***> wrote: > @raffaelespazzoli <https://github.com/raffaelespazzoli> can you please > share info on how to reproduce it? Using code.quarkus.io/api.download > and editing the pom.xml file to use 1.8.3 works as expected for me. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#10637 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABPERXDOESYSKC5OODGQRELSR7HOZANCNFSM4OWOM6YQ> > . > -- ciao/bye Raffaele

-- ciao/bye Raffaele

raffaelespazzoli · 2020-11-27T17:07:52Z

it worked with this option -Dquarkus.native.container-runtime=podman, perhaps it should be made more apparent in the docs.

zakkak · 2020-11-29T10:03:38Z

Hi @raffaelespazzoli, this is documented in https://github.com/quarkusio/quarkus/blob/master/docs/src/main/asciidoc/building-native-image.adoc#creating-a-linux-executable-without-graalvm-installed
Improvements (through PR) or suggestions are welcome :)

gsmet · 2020-11-29T10:20:20Z

Do we have a way to detect if docker is in fact podman and automatically enable the podman specific bits?

geoand · 2020-11-29T16:21:33Z

That's a good idea. I am sure it should be possible. @gastaldi any ideas? Perhaps just docker --version when backed by podman is enough?

gastaldi · 2020-11-29T21:08:29Z

Yeah docker --version should work AFAIK. I'm on PTO until Dec 2nd but I can provide a PR with this auto-detection algorithm after that

zakkak · 2020-12-01T11:56:19Z

While at it, could we make -Dquarkus.native.container-runtime=podman optional, i.e., only required when both docker and podman are available?

If docker is not present and podman is available Quarkus should automatically chose it and vice versa.
If both docker and podman are available, Quarkus should default to one of them and check if it's an alias to the other, to enable the specific bits.
If -Dquarkus.native.container-runtime=podman is used Quarkus should use podman and fail if podman is an alias to docker, similarly for -Dquarkus.native.container-runtime=docker, Quarkus should use docker and fail if docker is an alias to podman.

WDYT?

gastaldi · 2020-12-01T12:54:54Z

@zakkak sure, the auto-detection should kick in only if the -Dquarkus.native.container-runtime is not specified

gastaldi · 2020-12-03T14:50:29Z

I think we can even simplify the auto-detection algorithm by returning podman if podman --version returns something like podman version 2.1.1 and fallback to docker otherwise, because if you have a docker alias to podman, you'll certainly have the podman executable available

jerboaa added the kind/bug Something isn't working label Jul 10, 2020

rmarting mentioned this issue Jul 27, 2020

NativeImageBuildStep fails if 'docker' is not installed #10984

Closed

galderz self-assigned this Jul 28, 2020

tqvarnst added a commit to tqvarnst/quarkus that referenced this issue Jul 28, 2020

Fixes for issue quarkusio#10637

0b52b07

n1hility added the priority/blocker label Jul 29, 2020

galderz removed their assignment Jul 29, 2020

gsmet added this to the 1.7.0.Final milestone Jul 30, 2020

gastaldi assigned zakkak Jul 30, 2020

zakkak mentioned this issue Jul 30, 2020

NativeImageBuild: Add --user flag to podman #11100

Merged

gsmet closed this as completed in #11100 Jul 30, 2020

gsmet modified the milestones: 1.7.0.Final, 1.7.0.CR1 Jul 31, 2020

gastaldi mentioned this issue Dec 3, 2020

Auto-detect container runtime #13669

Merged

reda-alaoui mentioned this issue Jul 22, 2022

Quarkus NativeImageBuildStep fails with perm denied with docker rootless #26890

Closed

galderz mentioned this issue May 8, 2023

Podman fails to build native on macOS #33188

Closed

parawanderer mentioned this issue Sep 19, 2023

Rootless docker & podman build fails on Linux with /usr/bin/ld: cannot open output file /project/x: Permission denied #36007

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Quarkus NativeImageBuildStep fails with perm denied with podman 2.x #10637

Quarkus NativeImageBuildStep fails with perm denied with podman 2.x #10637

jerboaa commented Jul 10, 2020

rmarting commented Jul 27, 2020

galderz commented Jul 28, 2020

galderz commented Jul 29, 2020

zakkak commented Jul 29, 2020

zakkak commented Jul 30, 2020

zakkak commented Jul 30, 2020 •

edited

Loading

gsmet commented Jul 30, 2020

geoand commented Jul 30, 2020

gsmet commented Jul 30, 2020

gastaldi commented Jul 30, 2020

geoand commented Jul 30, 2020

gastaldi commented Jul 30, 2020

zakkak commented Jul 30, 2020

Ladicek commented Jul 30, 2020

zakkak commented Jul 30, 2020

geoand commented Jul 30, 2020

gastaldi commented Jul 30, 2020

zakkak commented Jul 30, 2020 •

edited

Loading

raffaelespazzoli commented Nov 26, 2020

zakkak commented Nov 27, 2020

raffaelespazzoli commented Nov 27, 2020 via email

zakkak commented Nov 27, 2020 •

edited

Loading

raffaelespazzoli commented Nov 27, 2020 via email

raffaelespazzoli commented Nov 27, 2020

zakkak commented Nov 29, 2020

gsmet commented Nov 29, 2020

geoand commented Nov 29, 2020

gastaldi commented Nov 29, 2020 •

edited

Loading

zakkak commented Dec 1, 2020

gastaldi commented Dec 1, 2020

gastaldi commented Dec 3, 2020 •

edited

Loading

Quarkus NativeImageBuildStep fails with perm denied with podman 2.x #10637

Quarkus NativeImageBuildStep fails with perm denied with podman 2.x #10637

Comments

jerboaa commented Jul 10, 2020

rmarting commented Jul 27, 2020

galderz commented Jul 28, 2020

galderz commented Jul 29, 2020

zakkak commented Jul 29, 2020

zakkak commented Jul 30, 2020

zakkak commented Jul 30, 2020 • edited Loading

gsmet commented Jul 30, 2020

geoand commented Jul 30, 2020

gsmet commented Jul 30, 2020

gastaldi commented Jul 30, 2020

geoand commented Jul 30, 2020

gastaldi commented Jul 30, 2020

zakkak commented Jul 30, 2020

Ladicek commented Jul 30, 2020

zakkak commented Jul 30, 2020

geoand commented Jul 30, 2020

gastaldi commented Jul 30, 2020

zakkak commented Jul 30, 2020 • edited Loading

raffaelespazzoli commented Nov 26, 2020

zakkak commented Nov 27, 2020

raffaelespazzoli commented Nov 27, 2020 via email

zakkak commented Nov 27, 2020 • edited Loading

raffaelespazzoli commented Nov 27, 2020 via email

raffaelespazzoli commented Nov 27, 2020

zakkak commented Nov 29, 2020

gsmet commented Nov 29, 2020

geoand commented Nov 29, 2020

gastaldi commented Nov 29, 2020 • edited Loading

zakkak commented Dec 1, 2020

gastaldi commented Dec 1, 2020

gastaldi commented Dec 3, 2020 • edited Loading

zakkak commented Jul 30, 2020 •

edited

Loading

zakkak commented Jul 30, 2020 •

edited

Loading

zakkak commented Nov 27, 2020 •

edited

Loading

gastaldi commented Nov 29, 2020 •

edited

Loading

gastaldi commented Dec 3, 2020 •

edited

Loading