Cannot build images with podman build 1.8.0 in Docker containers #2175
Comments
|
@mathstuf I think this is the same issue a: From that, if you can, please try:
@giuseppe any other thoughts? |
|
I've tried the workarounds. While they've gotten me further along, its still not working: |
|
There's no 1.8.0 image available (due to this), but if I manually update podman inside the container to 1.8.0, it works! |
|
Okay it seems i have the same issue, yet it still persists even with v1.8.0. [root@66d0aa0fb67e ~]# podman --cgroup-manager=cgroupfs build -t foo .
STEP 1: FROM ubuntu
STEP 2: RUN cat /etc/*release
Error: error building at STEP "RUN cat /etc/*release": error mounting container "1bb40fa7899f4d698d5fe4c9592144e55697aa49f2c87269717ac82fbd2ee4c5": error mounting build container "1bb40fa7899f4d698d5fe4c9592144e55697aa49f2c87269717ac82fbd2ee4c5": failed to canonicalise path for /var/lib/containers/storage/overlay/a50e8d976ab30165352c31692764efb560adf4b81fcdd2b7d9ed701f401ee78b/merged: lstat /var/lib/containers/storage/overlay/a50e8d976ab30165352c31692764efb560adf4b81fcdd2b7d9ed701f401ee78b/merged: invalid argumentI used the exact same steps as @paleozogt on a local test REHL 7.x server (with docker 1.13.1) and on PWD (play-with-docker, i had no other test machine and windows was note really an option), still resulted in the same error. (sorry if this a different issue and I should have opened a new issue, I shall open on if deemed so) EDIT: I might want to add that i get the same error with buildah (if i try to build the same Dockerfile via |
|
|
|
I'm seeing both @h1dden-da3m0n's result as well as @mathstuf's, depending on how I run it. On a regular bare-metal host (Ubuntu Bionic), if I run docker directly it'll work as in my comment above. However, if I run from within a container (such as a kubernetes pod or play-with-docker), it'll fail. Here's it failing on play-with-docker: |
|
Hmm. I'm seeing it work within Docker images on a Fedora 31 host (executed via Docker due to needing the Docker socket for |
|
I've been digging into this some more. On k8s clusters with Ubuntu Bionic hosts, I can run a pod that is able to use podman. However, on k8s clusters with Centos 7 hosts, running a pod using podman will see the What features/OS version does buildah/podman need on the host system in order to work? |
|
RHEL 7.* < 7.8 can not run rootless containers with fuse-overlay. So this can cause some issues. As of 7.8, RHEL7 should work better with rootless environments. |
|
I'll hav a look which REHL 7.x version we have deployed on moday and report back. |
|
I've got Centos 7.4.1708 and 7.6.1810, neither of which work. So this tracks with @rhatdan's comment. |
|
okay, just checked we are running REHL 7.7, so is it really just a Host OS version issue? |
|
Can we close this issue, and wait for RHEL7.8 or even better, move to rhel8? That is where the cool kids are going... :^) |
|
I'm fine with closing it; a solution seems to have been found for everyone's issues here. And FYI, RHEL 8 needs a podman release with containers/podman#4499 in a stable enough state before I can deploy with it for container usage (though the official Docker repos are also available I suppose, I don't like adding repos like that unless necessary). |
|
I thought REHL 8.1 was the one for cool kids 😁 , but fine bye me to close this. |
|
I am thoroughly confused by this issue. I was responding to a few comments back that was attempting to execute rootless podman on a RHEL/Centsos 7.4 box. |
|
Not sure if I was the confusing factor for @rhatdan or not (sorry if I was indeed), but as far as I see it Buildah/Podman has some problem running inside of a container. In this issue there are the following host OSes listed to face the same issue as the OP as well as an issue that seems to pop up as soon as you pass the OP issue:
The OP error seems to get "fixed" once you update podman / buildah inside of the container to the latest version (1.8.0 for podman I think), yet that then leads to the I think I "summarized" the issue so far, hope this helps clear up the confusion. edit:
The key words here being containers with fuse-overlay and that such containers wont work on REHL < 7.8 (or 8). Finally, I want to apologise for the holdup on closing the issue and maybe confusing the one or the other participant. TLDR: |
|
This is happening to me while running inside Gitlab's free runner. Is there any solution? 😕 |
|
@yajo can you confirm the versions of Podman that are in use and were you able to use any of the workarounds listed in the comments above? If not, can you talk a little bit about your environment, especially OS versions and the such. |
Yes, you can read that in the attached log, but let me summarize: |
Oh sorry forgot this. I cannot use the container as privileged, because it's running in public free CI runners, which let you specify the image where to run from and it is executed through docker. You can see the full So, podman is trying to run inside a So my case is podman-in-docker, and it's not something I can escape from 😕 If I can help any further, just tell me 😊 |
|
You're just going to need privileged rights inside of a Docker runner to use podman. If the CI infra doesn't offer it, you'll need a different solution I'm afraid. You could stand your own runners up, but then you're probably waiting on other gitlab bugs/feature requests related to that. I think you're missing something anyways, but maybe this is just the way it manifests:
|
|
Ah, that is the error message at the beginning here. Maybe the |
|
OK I have done a very simple test: Inside Gitlab's free runbot offering (based on Docker, unprivileged), just run See the full log: https://gitlab.com/yajoman/rpi_cooler/-/jobs/569793360 Is there anything else I could do to make it work? Maybe using VFS storage driver? 🤔 |
What does that podman command look like? |
|
Did I do anything wrong? See https://gitlab.com/yajoman/rpi_cooler/-/jobs/575997744: |
|
@giuseppe PTAL |
|
$ podman --cgroup-manager cgroupfs --storage-driver fuse-overlayfs container run --rm -it hello-world fuse-overlayfs is not a graph driver, this should be overlay. And you should enable the fuse-overlay line in the /etc/containers/storage.conf inside of the container image. You should also specify podman command with podman --device /dev/fuse --cgroup-manager ... That will get you closer. |
|
Yikes, no way. I discovered that gitlab shared runners are actually privileged, so you can reproduce locally easily with this: ➤ docker container run --rm --privileged -it registry.gitlab.com/yajoman/ci-tools/fedora podman --cgroup-manager cgroupfs --storage-driver fuse-overlayfs container run --rm -it hello-world
ERRO[0000] Failed to GetDriver graph fuse-overlayfs /var/lib/containers/storage
Error: error creating libpod runtime: failed to GetDriver graph fuse-overlayfs /var/lib/containers/storage: driver not supportedIt seems ➤ docker container run --rm --privileged -it registry.gitlab.com/yajoman/ci-tools/fedora podman --device /dev/fuse --cgroup-manager cgroupfs --storage-driver fuse-overlayfs container run --rm -it hello-world
Error: unknown flag: --deviceAlso enabling the fuse-overlay line doesn't seem to help: ➤ docker container run --rm --privileged -it registry.gitlab.com/yajoman/ci-tools/fedora bash -c "sed -i 's@#mount_program = \"/usr/bin/fuse-overlayfs\"@mount_program = \"/usr/bin/fuse-overlayfs\"@' /etc/containers/storage.conf && podman --cgroup-manager cgroupfs --storage-driver fuse-overlayfs container run --rm -it hello-world"
ERRO[0000] Failed to GetDriver graph fuse-overlayfs /var/lib/containers/storage
Error: error creating libpod runtime: failed to GetDriver graph fuse-overlayfs /var/lib/containers/storage: driver not supportedThere must be a way... 🤔 Any ideas? |
|
I see VFS works out of the box, but it's too much not-optimized, mostly compared to overlay: ➤ docker container run --rm --privileged -it registry.gitlab.com/yajoman/ci-tools/fedora podman --cgroup-manager cgroupfs --storage-driver vfs container run --rm -it hello-world
Trying to pull registry.fedoraproject.org/hello-world...
manifest unknown: manifest unknown
Trying to pull registry.access.redhat.com/hello-world...
name unknown: Repo not found
Trying to pull registry.centos.org/hello-world...
manifest unknown: manifest unknown
Trying to pull docker.io/library/hello-world...
Getting image source signatures
Copying blob 0e03bdcc26d7 done
Copying config bf756fb1ae done
Writing manifest to image destination
Storing signatures
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/No way to do it with fuse-overlayfs? |
|
--storage-driver=fuse-overlayfs is wrong, it should be --storage-driver=overlay. |
|
Still not working: ➤ docker container run --rm --privileged -it registry.gitlab.com/yajoman/ci-tools/fedora podman --cgroup-manager cgroupfs --storage-opt storage.options.overlay.mount_program=/usr/bin/fuse-overlayfs --storage-driver overlayfs container run --device /dev/fuse --rm -it hello-world
ERRO[0000] Failed to GetDriver graph overlayfs /var/lib/containers/storage
Error: error creating libpod runtime: failed to GetDriver graph overlayfs /var/lib/containers/storage: driver not supported
➤ docker container run --device /dev/fuse --rm --privileged -it registry.gitlab.com/yajoman/ci-tools/fedora podman --cgroup-manager cgroupfs --storage-opt storage.options.overlay.mount_program=/usr/bin/fuse-overlayfs --storage-driver overlayfs container run --device /dev/fuse --rm -it hello-world
ERRO[0000] Failed to GetDriver graph overlayfs /var/lib/containers/storage
Error: error creating libpod runtime: failed to GetDriver graph overlayfs /var/lib/containers/storage: driver not supportedAre you sure this is supposed to work? 🤔 |
|
It's working for me. Host is CentOS 7 or 8 if that helps. |
|
overlayfs->overlay |
|
🎉 🚀 Made it work, thanks! ➤ docker container run --rm --privileged -it registry.gitlab.com/yajoman/ci-tools/fedora podman --cgroup-manager cgroupfs --storage-opt overlay.mount_program=/usr/bin/fuse-overlayfs --storage-driver overlay container run --rm -it hello-world
Trying to pull registry.fedoraproject.org/hello-world...
manifest unknown: manifest unknown
Trying to pull registry.access.redhat.com/hello-world...
name unknown: Repo not found
Trying to pull registry.centos.org/hello-world...
manifest unknown: manifest unknown
Trying to pull docker.io/library/hello-world...
Getting image source signatures
Copying blob 0e03bdcc26d7 done
Copying config bf756fb1ae done
Writing manifest to image destination
Storing signatures
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/I didn't even need I also modified It works on Gitlab CI too: https://gitlab.com/yajoman/rpi_cooler/-/jobs/582567308 ❤️ |
|
Nice, now another question. If I remove ➤ docker container run --rm -it registry.gitlab.com/yajoman/ci-tools/fedora podman --cgroup-manager cgroupfs --storage-opt overlay.mount_program=/usr/bin/fuse-overlayfs --storage-driver overlay container run --device /dev/fuse --rm -it hello-world
Error: error creating libpod runtime: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permittedIs there a way to replace |
|
--cap-add SYS_ADMIN |
|
You might have to disable SELinux if you are using it and seccomp. |
I'm fulfilling my promise from ansible#2714 (comment) by documenting specific stuff learned from containers/buildah#2175 (comment).
I'm fulfilling my promise from #2714 (comment) by documenting specific stuff learned from containers/buildah#2175 (comment).
|
We have added some docs on this, closing this issue. |
|
@rhatdan can you link to the docs? |
Description
podman buildfails with:The container is not systemd-as-pid1, so enabling the journal doesn't work. Passing
--events-backend=filealso doesn't avoid the issue.Full output (including dep resolution for podman itself): https://gitlab.kitware.com/ben.boeckel/cmb-superbuild/-/jobs/4816230
Steps to reproduce the issue:
fedora:31in my case)podman buildDescribe the results you received:
Failure to connect to sd-bus.
Describe the results you expected:
A built image
Output of
podman versionif reporting apodman buildissue:Output of
cat /etc/*release:Output of
uname -a:Taken from the host of the builder in question.
Output of
cat /etc/containers/storage.conf:Fedora default configuration.
The text was updated successfully, but these errors were encountered: