mount `tmpfs` to `/dev`: Permission denied #16332

dpkshetty · 2022-10-28T10:01:46Z

dpkshetty
Oct 28, 2022

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

My gitlab-ci.yml (code snip which is failing...)

    - echo "Building container on ppc64le"
    - buildah --storage-driver=vfs build -t $APP .
    - echo "Build complete."

    - echo "Listing buildah images"
    - buildah --storage-driver=vfs images

    - echo "Push image to ${IMAGE_REGISTRY}:${TAG}"
    - buildah --storage-driver=vfs push --creds $quay_user:$quay_passwd $APP ${IMAGE_REGISTRY}:${TAG}-ppc64le

As you can see, I am using podman image to create a Dockerfile for my app. Link to Dockerfile
https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/blob/main/Dockerfile
podman build fails as below...

$ podman --storage-driver=vfs build -t $APP .
time="2022-10-28T07:49:58Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"
STEP 1/7: FROM registry.access.redhat.com/ubi8/python-38
Trying to pull registry.access.redhat.com/ubi8/python-38:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob sha256:d0d2e9042a028b50d899ac4c9a760d0a2305d515d0ad127b377a8e437dde0623
Copying blob sha256:d172799e5c5b9a62cce0219edd286f524a25155a018d7eda982c0d0706ed9fa9
Copying blob sha256:32af3091e80b9aa028a918579576c3a34d03a59746461d16990a9077ddb8541b
Copying blob sha256:9fb9feb1f76f717c6ae977c6d19b7aa7970602cbda1aea0072e9a669f0c94ad5
Copying config sha256:5e3e4648bb5c196e28e438f5ee535b7ffb36b0394dfab8ea1159d3ef5d3[26](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3241428683#L26)95a
Writing manifest to image destination
Storing signatures
STEP 2/7: USER 0
--> 74803e52454
STEP 3/7: ADD app-src /tmp/src
--> [35](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3241428683#L35)[38](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3241428683#L38)03cc6da
STEP 4/7: RUN chown -R 1001:0 /tmp/src
error running container: error from /usr/bin/crun creating container for [/bin/sh -c chown -R 1001:0 /tmp/src]: mount `tmpfs` to `/dev`: Permission denied
: exit status 1
Error: error building at STEP "RUN chown -R 1001:0 /tmp/src": error while running runtime: exit status 1
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: command terminated with exit code 1

Describe the results you received:

podman build fail

Describe the results you expected:

podman build should succeed

Additional information you deem important (e.g. issue happens only occasionally):

Please note that in the above gitlab-ci.yml, if I replace podman with buildah (image and CLI), it works!!

Output of podman version:

(paste your output here)

Output of podman info:

(paste your output here)

Package info (e.g. output of rpm -q podman or apt list podman or brew info podman):

(paste your output here)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes/No

Additional environment details (AWS, VirtualBox, physical, etc.):

podman container is running in openshift as part of gitlab CI pipeline

rhatdan · 2022-10-28T10:52:33Z

rhatdan
Oct 28, 2022
Maintainer

Have you read https://www.redhat.com/sysadmin/podman-inside-kubernetes?

5 replies

dpkshetty Oct 28, 2022
Author

I took a glance at that article
I am not an expert so a bit difficult to digest what exactly to try out... for some issue the article asks to change/edit host config files which can't be done in k8s/openshift context. Can u point me to the exact solution I should be using ?

dpkshetty Oct 28, 2022
Author

Does it help, if I say.. that the exact same buildah steps work successfully on OCP on ppc64le (IBM Power) cluster, but fails with the above error on OCP on x86 cluster! Strange!

rhatdan Oct 28, 2022
Maintainer

Are you running both with the same security profile.?

@umohnani8 PTAL

dpkshetty Oct 28, 2022
Author

I am just executing my Gitlab pipeline and it should be running the runner pods with the same security profile. I am using the arch specific gitlab runners , installed via the respective operators. I have created custom SA and associated with anyuid perms on both and it works like a charm on ppc64le but fails on x86 cluster.

dpkshetty Oct 28, 2022
Author

@rhatdan my bad.. let me stand corrected... I was trying both buildah and podman methods - hence confusion
My apologies... above i gave buildah steps (from my gitlab CI pipeline) but error msg was from podman

Here is what is happening...

buildah method works on ppc64le (per the steps listed at the top of this discussion) but fails on x86
podman fails on both ppc64le and x86 clusters - as shown in error msg above
Bottomline.. I am unable to build successfully for both ppc64le and x86 clusters as neither buildah works entirely nor podman

rhatdan · 2022-10-28T10:53:14Z

rhatdan
Oct 28, 2022
Maintainer

I moved this to a discussion, since this is not a bug in Podman, but a setup issue in OpenShift.

0 replies

rhatdan · 2022-10-28T18:46:27Z

rhatdan
Oct 28, 2022
Maintainer

I would need to see the audit.log on both boxes to see if there are SELinux issues.

2 replies

dpkshetty Nov 25, 2022
Author

@rhatdan is there a way to connect on Slack so that I can show you the issue. I have switched to using podman now. I don't use buildah at all in my CI pipeline. There is a different issue I am facing now... which a week back magically disappeared when i used --isolation chroot but it re-appears now! Having a slack discussion would be better, once root caused I can dump all the relevant info here for the sake of community good. Also I don't own these openshift env. I provision them via a internal portal of ours so might need some help (hence slack discussion) to get the relevant files like audit.log. Let me know.

FYI: I am using the same CI pipeline which runs on x86_64 and ppc64le arch. clusters.

Below is the failure (only seen on x86_64) while on ppc64le it works fine

Building container on x86
$ podman --storage-driver=vfs build --isolation chroot -t $APP -f ./Dockerfile --no-cache
time="2022-11-25T08:57:40Z" level=warning msg="Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user"
Error: cannot write uid_map: write /proc/25/uid_map: operation not permitted

Building container on ppc64le
$ podman --storage-driver=vfs build --isolation chroot -t $APP -f ./Dockerfile --no-cache
STEP 1/7: FROM registry.access.redhat.com/ubi7/python-38
Trying to pull registry.access.redhat.com/ubi7/python-38:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob sha256:3e12824404c8f9ece1102b36177afcc00b6a175bee6feb71746d4925b0c0b5f4
Copying blob sha256:58910fdfeb79e1c95a750d913adf6d0f9d224b2517a1544ec4f9112a2758668e
Copying blob sha256:7b827a621e5acd17204d927aa4306a362d2e7ee4f875eb84973e4e062de2184b
Copying blob sha256:019f02db9974ecbb2caa2a3f0[29](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3378529332#L29)4fc145add8f86f97f86a[30](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3378529332#L30)fe542006f9db79a
Copying config sha256:9edfd1fa2ba506edea3570ce51fed30964ff66457d9e1de07d1443a357886f4a
Writing manifest to image destination
Storing signatures
STEP 2/7: USER 0
--> 62973fda67a
STEP 3/7: ADD app-src /tmp/src
--> 666c5dccdbd
STEP 4/7: RUN chown -R 1001:0 /tmp/src
--> 4d7077bd9ff
STEP 5/7: USER 1001
--> d309dca4[31](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3378529332#L31)c
STEP 6/7: RUN /usr/libexec/s2i/assemble
---> Installing application source ...
---> Installing dependencies ...
Collecting flask
  Downloading Flask-2.2.2-py3-none-any.whl (101 kB)
Collecting Jinja2>=3.0
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting itsdangerous>=2.0
  Downloading itsdangerous-2.1.2-py3-none-any.whl (15 kB)
Collecting click>=8.0
  Downloading click-8.1.3-py3-none-any.whl (96 kB)
Collecting importlib-metadata>=3.6.0
  Downloading importlib_metadata-5.1.0-py3-none-any.whl (21 kB)
Collecting Werkzeug>=2.2.2
  Downloading Werkzeug-2.2.2-py3-none-any.whl (2[32](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3378529332#L32) kB)
Collecting zipp>=0.5
  Downloading zipp-3.10.0-py3-none-any.whl (6.2 kB)
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.1.1.tar.gz (18 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Using legacy 'setup.py install' for MarkupSafe, since package 'wheel' is not installed.
Installing collected packages: zipp, MarkupSafe, Werkzeug, Jinja2, itsdangerous, importlib-metadata, click, flask
    Running setup.py install for MarkupSafe: started
    Running setup.py install for MarkupSafe: finished with status 'done'
Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.1 Werkzeug-2.2.2 click-8.1.3 flask-2.2.2 importlib-metadata-5.1.0 itsdangerous-2.1.2 zipp-3.10.0
WARNING: You are using pip version 21.3.1; however, version 22.3.1 is available.
You should consider upgrading via the '/opt/app-root/bin/python3.8 -m pip install --upgrade pip' command.
--> 57b8570b5d6
STEP 7/7: CMD /usr/libexec/s2i/run
COMMIT dpk-pyflask
--> b8abe0c3d60
Successfully tagged localhost/dpk-pyflask:latest
b8abe0c3d6065a01ccaf[35](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3378529332#L35)7fef29621b135e7589fe73b87[51](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3378529332#L51)293aa7[63](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3378529332#L63)bf7[67](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3378529332#L67)af
$ echo "Build complete."
Build complete

dpkshetty Nov 25, 2022
Author

I have referred to podman troubleshooting guide and tried to disable SELinux, but that didn't help.

$ echo "Building container on x86"
Building container on x86
$ podman --storage-driver=vfs build --isolation chroot --security-opt label=disable -t $APP -f ./Dockerfile --no-cache
time="2022-11-25T09:10:10Z" level=warning msg="Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user"
Error: cannot write uid_map: write /proc/27/uid_map: operation not permitted
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: command terminated with exit code 1

Again, this works fine on ppc64le cluster

dpkshetty · 2022-11-25T10:16:03Z

dpkshetty
Nov 25, 2022
Author

@rhatdan I have referred to the podman in k8s but I don't see this exact problem referenced there. I still went ahead and tried --userns option. x86 it still fails the same way and on ppc64le i think that option (syscall?) isn't supported looks like (might need another bug for that ?)

Building container on x86
$ podman --storage-driver=vfs build --isolation chroot --security-opt label=disable --userns=keep-id -t $APP -f ./Dockerfile --no-cache
time="2022-11-[25](https://gitlab.com/dpkshetty/gitlab-s2i-pyflask-demo/-/jobs/3378790103#L25)T09:19:09Z" level=warning msg="Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user"
Error: cannot write uid_map: write /proc/24/uid_map: operation not permitted
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: command terminated with exit code 1

Building container on ppc64le
$ podman --storage-driver=vfs build --isolation chroot --security-opt label=disable --userns=keep-id -t $APP -f ./Dockerfile --no-cache
Error: checking user namespace: stat keep-id: no such file or directory
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: command terminated with exit code 1

6 replies

rhatdan Nov 28, 2022
Maintainer

Don't use --userns=keep-id, I am not sure this is supported for podman build, only podman run.

giuseppe Dec 1, 2022
Maintainer

Error: cannot write uid_map: write /proc/24/uid_map: operation not permitted

this seems like either CAP_SETUID/CAP_SETGID are not available in the container, or the newuidmap/newgidmap do not have file capabilities.

dpkshetty Dec 1, 2022
Author

Any suggestions on what to do next ?
The exact same gitab CI yaml works fine for a ppc64le cluster I have. But fails with the above error on a x86_64 cluster.

rhatdan Dec 2, 2022
Maintainer

getcap /usr/bin/newuidmap /usr/bin/newgidmap
/usr/bin/newuidmap cap_setuid=ep
/usr/bin/newgidmap cap_setgid=ep

dpkshetty Dec 2, 2022
Author

@rhatdan @giuseppe thanks for your support. I tried to put some debug prints as suggested above and here are the dumps for x86 and ppc64le case. My CI yaml file is here - x86 fails, ppc64le works fine.

x86 - fails
https://gist.github.com/dpkshetty/050fde23b679d123d5cf5ae6afe177a1

ppc64le - succeeds
https://gist.github.com/dpkshetty/f59c85fea6e5d69dc82e1b8b464aa114

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mount `tmpfs` to `/dev`: Permission denied #16332

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 13 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

mount tmpfs to /dev: Permission denied #16332

dpkshetty Oct 28, 2022

Replies: 4 comments · 13 replies

rhatdan Oct 28, 2022 Maintainer

dpkshetty Oct 28, 2022 Author

dpkshetty Oct 28, 2022 Author

rhatdan Oct 28, 2022 Maintainer

dpkshetty Oct 28, 2022 Author

dpkshetty Oct 28, 2022 Author

rhatdan Oct 28, 2022 Maintainer

rhatdan Oct 28, 2022 Maintainer

dpkshetty Nov 25, 2022 Author

dpkshetty Nov 25, 2022 Author

dpkshetty Nov 25, 2022 Author

rhatdan Nov 28, 2022 Maintainer

giuseppe Dec 1, 2022 Maintainer

dpkshetty Dec 1, 2022 Author

rhatdan Dec 2, 2022 Maintainer

dpkshetty Dec 2, 2022 Author

mount `tmpfs` to `/dev`: Permission denied #16332

dpkshetty
Oct 28, 2022

Replies: 4 comments 13 replies

rhatdan
Oct 28, 2022
Maintainer

dpkshetty Oct 28, 2022
Author

dpkshetty Oct 28, 2022
Author

rhatdan Oct 28, 2022
Maintainer

dpkshetty Oct 28, 2022
Author

dpkshetty Oct 28, 2022
Author

rhatdan
Oct 28, 2022
Maintainer

rhatdan
Oct 28, 2022
Maintainer

dpkshetty Nov 25, 2022
Author

dpkshetty Nov 25, 2022
Author

dpkshetty
Nov 25, 2022
Author

rhatdan Nov 28, 2022
Maintainer

giuseppe Dec 1, 2022
Maintainer

dpkshetty Dec 1, 2022
Author

rhatdan Dec 2, 2022
Maintainer

dpkshetty Dec 2, 2022
Author