buildah fails to pull openjdk:15 in container with permission denied #3384
Comments
|
We disable CAP_MKNOD by default if you are using Podman or Kubernetes, so you have to add the cap --cap-add mknod |
|
Sorry for the noise @rhatdan , may be I'm missing something important, but adding The command is almost the same (podman is running as root) Strace output from the process which gets |
|
Are you trying to do this rootless? |
|
Could you check if SELinux or SECCOMP is complaining? sudo ausearch -m seccomp -ts recent |
I think no, because I'm running podman as root user and docker is rootfull by default. Above output and strace was from ubuntu 20.04 system, apparmor is completely disabled in kernel options. I tried to update Centos 8.4 (on centos I primarily tested this issue with docker) and podman with the same result as before and as in docker. SELinux is complaining about reading Dockerfile (because it is mouted from host and is in root folder), but selinux is in Permissive mode and is not complaining anything about mknodeat syscall. Other things: |
|
Strace in centos shows the same EPERMed syscall I noticed that there is |
|
To give some context. We are using buildah in docker as primary image building tool in gitlab-runner on centos 8 host. Because buildah does not compromise host machine by requiring docker.socket to be mounted from host or privileged container to run in. If you need more information I'll be glad to provide it, I've created two machines (ubuntu 20.04 and centos 8) dedicated for this case. |
|
I have especially published port 80 which should only be possible with rootfull containers, If I correctly understand. |
|
Can you do |
|
I have tried this yesterday on both ubuntu and centos with same result. |
|
On freshly installed Fedora Server 34 in Virtual box I observe same behavior. It is very easily reproduced, just 3 commands as root (after installing podman) |
|
Hi @rhatdan, would you mind reopening this issue if you don't think this is desired buildah behavior? |
|
A friendly reminder that this issue had no activity for 30 days. |
|
I don't see this issue with on fresh podman and buildah both with Reproducer which i tried echo 'FROM docker.io/openjdk:15' > Dockerfile
podman run --rm --device /dev/fuse -v /tmp/exp:/test:Z -w /test quay.io/buildah/stable buildah --storage-driver=overlay --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs build .Output STEP 1/1: FROM docker.io/openjdk:15
Trying to pull docker.io/library/openjdk:15...
Getting image source signatures
Copying blob sha256:ab2540feecc546a000ab358bde2118583f78b6074e7141f2b75c2f804970d429
Copying blob sha256:9509c6b41a37fbf5dbb93aedded1aff0dc6ed45ab2d334440e10a5c8d112531c
Copying blob sha256:1a0005db77786a07e4e5d56adc224c9dc85320b46354f9110eb174ce7df9df04
Copying blob sha256:ab2540feecc546a000ab358bde2118583f78b6074e7141f2b75c2f804970d429
Copying blob sha256:9509c6b41a37fbf5dbb93aedded1aff0dc6ed45ab2d334440e10a5c8d112531c
Copying blob sha256:1a0005db77786a07e4e5d56adc224c9dc85320b46354f9110eb174ce7df9df04
Copying config sha256:bae9931e822b1762f91550ecbefea67b9421c249125e305954e3f10ac78a4632
Writing manifest to image destination
Storing signatures
COMMIT
--> bae9931e822
bae9931e822b1762f91550ecbefea67b9421c249125e305954e3f10ac78a4632I'm closing this issue but feel free to reopen if i'm mistaken. |
|
Hello, I'm facing an issue similar to the one described by @Peter-Sh but with official image mysql:5.7 this time. Buildah outputs the following error : I've straced the Buildah process, the error is the same as the excerpt posted above : I've attached the full strace dump for convenience. The same error is also raised for image We are using Buildah in a CI/CD setup : build pods are instantiated in an Openshift 4.10 cluster running with the As Buildah is run in a container, we have implemented the configuration described in this comment : openshift/enhancements#362 (comment). The To investigate, I've created an SCC which is explicitly allowing the In this pod, an As stated in the initial comment of this issue, the only way to pull this image is to make the container privileged, with the associated security implications. A few information about our Buildah build image : @rhatdan , @flouthoc : is it a good practice from these images to embed special devices ? Shouldn't the Antoine |
|
Yes Images should not embed special devices within them. I think to make this work, you could add --cap-add mknod |
Indeed, I just tested adding As I understand it, pulling container images embedding special devices on a In this context, do you think that implementing a command-line switch to optionally skip the extraction of special devices from the image would be possible ? IMHO such a switch would improve Buildah robustness and would be beneficial for end-users having to deal with these images in a more constrained environment. Antoine |
|
Are you running in rootless mode? The cap-add mknod would only work in the case where the container around buildah had dropped it. |
|
The aforementioned The build image Containerfile contains a |
|
@giuseppe ideas? What does |
|
is the mount program specified? Otherwise we might try to set a privileged extended attribute (containers/storage#1312 fixes it), that rootless users cannot set |
The |
|
any chance you could strace the process to see what syscall is failing? |
|
I think I actually did that in #3384 (comment), there is a full strace attached there. Tell me if something is missing from it, I will create a new trace of the process if needed. |
|
no sorry, that is enough. I think that what is happening is that buildah detects to not have CAP_SYS_ADMIN and creates a user namespace to gain that capability. Doing so, it loses access to CAP_MKNOD in the initial user namespace, so it is not able to create these devices anymore |
|
Ok that makes sense for user namespace isolation. However, to avoid granting the SYS_ADMIN capability (which is required to create user namespaces I believe) to the build container, I had set the BUILDAH_ISOLATION environment variable to Is the same reasoning applicable for chroot isolation ? |
Description
Buildah fails with permission denied when pulling docker.io/openjdk:15 if running in container (podman or docker)
This is may be related to #1709, because strace also shows EPERM on mknodeat syscall, but it happens with vfs and fuse-overlayfs.
Steps to reproduce the issue:
The same happens with docker (to run buidah with docker custom seccomp should be provided) and with --storage-driver=vfs
Workaround is to add
--privilegedEnvironment
Tested buildah versions:
Podman version:
Docker versions:
Operating systems:
Describe the results you received:
Program output from last Ubuntu 20.04 run:
Describe the results you expected:
Successful build.
The text was updated successfully, but these errors were encountered: