crun 0.18 leaking mount options when it shouldn't

[cevich]

After building fresh VM images for podman CI, I'm seeing a unique failure @mheon thinks is coming from an updated crun package:
crun-0.18-4.fc33.x86_64. We only see this failure when running podman-in-podman. The test passes fine when run simply on the host.

I am able to reproduce this manually similar to the test, starting on a F33 VM as root:

# podman run -it --rm --privileged --net=host --cgroupns=host quay.io/libpod/fedora_podman:c4720503194648576
# mkdir -p /tmp/foobar
# podman --cgroup-manager cgroupfs --events-backend file --storage-driver vfs run --rm -v /tmp/foobar:/tmp/foobar:suid,dev,exec quay.io/libpod/alpine:latest grep /tmp/foobar /proc/self/mountinfo | grep nodev

According to the test code, this should not be finding nodev - but it does.

The same reproducer when run on Ubuntu 20.04 VM and containers (using runc) does not reproduce the issue:

root@cevich-prior-ubuntu-c4720503194648576:/# podman run -it --rm --privileged --net=host --cgroupns=host quay.io/libpod/prior-ubuntu_podman:c4720503194648576
Trying to pull quay.io/libpod/prior-ubuntu_podman:c4720503194648576...
Getting image source signatures
Copying blob ddbfc45e327d done
Copying blob 7f5bc2bf3694 done
Copying blob 858af6fdd912 done
Copying blob 29e62100b008 done
Copying blob 62580ea44675 done
Copying blob 8035b4508349 done
Copying blob d684334e3e53 done
Copying blob dec9f6c4ee9c done
Copying blob 373b8de4ab19 done
Copying blob 3045e1375c02 done
Copying config b34bd57a79 done
Writing manifest to image destination
Storing signatures
WARNING: The same type, major and minor should not be used for multiple devices.
root@cevich-prior-ubuntu-c4720503194648576:/var/tmp/go/src/github.com/containers/podma
n# cd /
root@cevich-prior-ubuntu-c4720503194648576:/# mkdir /tmp/foobar
root@cevich-prior-ubuntu-c4720503194648576:/# podman --cgroup-manager cgroupfs --event
s-backend file --storage-driver vfs run --rm -v /tmp/foobar:/tmp/foobar:suid,dev,exec
quay.io/libpod/alpine:latest grep /tmp/foobar /proc/self/mountinfo | grep nodev
Trying to pull quay.io/libpod/alpine:latest...
Getting image source signatures
Copying blob 9d16cba9fb96 done
Copying config 9617696764 done
Writing manifest to image destination
Storing signatures
root@cevich-prior-ubuntu-c4720503194648576:/#
root@cevich-prior-ubuntu-c4720503194648576:/# dpkg -l cri-o-runc
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  cri-o-runc     1.0.0~rc93.1 amd64        Open Container Project - runtime
root@cevich-prior-ubuntu-c4720503194648576:/# exit
root@cevich-prior-ubuntu-c4720503194648576:/# dpkg -l cri-o-runc
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  cri-o-runc     1.0.0~rc93.1 amd64        Open Container Project - runtime

[giuseppe]

I am still in the process of trying to reproduce locally (my first attempt on F34 worked fine).

What is the output you get from the last command?

EDIT: nevermind my question...

[giuseppe]

I am still not able to reproduce unless I modify the first container with: # mount -o remount,nodev /

Can you show me the output of cat /proc/self/mountinfo in the first container?

If the source bind mount has nodev that is going to be propagated no matter what.

It could help to run mount -o remount,dev / in the first container before running the tests

[giuseppe]

tentative fix: #640

marked as a draft as I need to test it better

[cevich]

Ya, I'm not able to repro. on F34beta either, only F33.

[root@F33 VM]# podman run -it --rm --privileged --net=host --cgroupns=host quay.io/libpod/fedora_podman:c4720503194648576
[root@container]# cat /proc/self/mountinfo
525 433 0:40 / / rw,nodev,relatime - overlay overlay rw,context="system_u:object_r:container_file_t:s0:c676,c727",lowerdir=/var/lib/containers/storage/overlay/l/J3WNPWC7SZA7N4UXHBLUQJCJUV:/var/lib/containers/storage/overlay/l/UAJWO3N2WGN3FBSZFCZHJHJC42:/var/lib/containers/storage/overlay/l/KEDGJ2OTAZSXZCPPS73NHWYQYL:/var/lib/containers/storage/overlay/l/N76NZ27ZZBBNRJW2EGEGVJOW4D:/var/lib/containers/storage/overlay/l/JHOVFBRNXLQUA7BFVGJFGFC6W6:/var/lib/containers/storage/overlay/l/FB2AB2COHKTBUJRAJWLHKYFHLT:/var/lib/containers/storage/overlay/l/YLL2B2MGWSOOPVR2V4J26JZ3K6:/var/lib/containers/storage/overlay/l/4OWJCV24SKKUFCAFXWQV2CNFRT,upperdir=/var/lib/containers/storage/overlay/b4834b029e0f91f6ca7ec9ee1f53ad9439ee10ecb556697ad99a58eb007c2ffb/diff,workdir=/var/lib/containers/storage/overlay/b4834b029e0f91f6ca7ec9ee1f53ad9439ee10ecb556697ad99a58eb007c2ffb/work
526 525 0:43 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
527 525 0:44 / /dev rw,nosuid - tmpfs tmpfs rw,context="system_u:object_r:container_file_t:s0:c676,c727",size=65536k,mode=755,inode64
528 525 0:22 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw,seclabel
529 527 0:45 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,context="system_u:object_r:container_file_t:s0:c676,c727",gid=5,mode=620,ptmxmode=666
530 527 0:42 / /dev/mqueue rw,nosuid,nodev,noexec,relatime - mqueue mqueue rw,seclabel
531 525 0:26 /containers/storage/overlay-containers/a84436a67c12c9d99a228b47f15260729d2b3b9c23dc9e1b5fcc9aa6b49f7a72/userdata/resolv.conf /etc/resolv.conf rw,nosuid,nodev - tmpfs tmpfs rw,seclabel,size=803932k,nr_inodes=819200,mode=755,inode64
532 525 0:26 /containers/storage/overlay-containers/a84436a67c12c9d99a228b47f15260729d2b3b9c23dc9e1b5fcc9aa6b49f7a72/userdata/hosts /etc/hosts rw,nosuid,nodev - tmpfs tmpfs rw,seclabel,size=803932k,nr_inodes=819200,mode=755,inode64
533 527 0:39 / /dev/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,context="system_u:object_r:container_file_t:s0:c676,c727",size=64000k,inode64
534 525 0:26 /containers/storage/overlay-containers/a84436a67c12c9d99a228b47f15260729d2b3b9c23dc9e1b5fcc9aa6b49f7a72/userdata/hostname /etc/hostname rw,nosuid,nodev - tmpfs tmpfs rw,seclabel,size=803932k,nr_inodes=819200,mode=755,inode64
535 525 0:26 /containers/storage/overlay-containers/a84436a67c12c9d99a228b47f15260729d2b3b9c23dc9e1b5fcc9aa6b49f7a72/userdata/.containerenv /run/.containerenv rw,nosuid,nodev - tmpfs tmpfs rw,seclabel,size=803932k,nr_inodes=819200,mode=755,inode64
536 525 0:26 /containers/storage/overlay-containers/a84436a67c12c9d99a228b47f15260729d2b3b9c23dc9e1b5fcc9aa6b49f7a72/userdata/run/secrets /run/secrets rw,nosuid,nodev - tmpfs tmpfs rw,seclabel,size=803932k,nr_inodes=819200,mode=755,inode64
537 528 0:27 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - cgroup2 cgroup2 rw,seclabel
434 527 0:45 /0 /dev/console rw,nosuid,noexec,relatime - devpts devpts rw,context="system_u:object_r:container_file_t:s0:c676,c727",gid=5,mode=620,ptmxmode=666

If the source bind mount has nodev that is going to be propagated no matter what.

So ya, that does appear to be the case. Though this is a test that's newly failing on F33, it wasn't failing until we updated packages.

[cevich]

tentative fix: #640

This seems like it will resolve the issue. I cloned the repo, checked out your PR, built crun and copied it to /usr/bin/crun. Now:

[root@F33 VM]# cp crun /usr/bin/crun
[root@F33 VM]# podman run -it --rm --privileged --net=host --cgroupns=host quay.io/libpod/fedora_podman:c4720503194648576
[root@container]# mkdir -p /tmp/foobar
[root@container]# podman --cgroup-manager cgroupfs --events-backend file --storage-driver vfs run --rm -v /tmp/foobar:/tmp/foobar:suid,dev,exec quay.io/libpod/alpine:latest grep /tmp/foobar /proc/self/mountinfo | grep nodev
Trying to pull quay.io/libpod/alpine:latest...
Getting image source signatures
Copying blob 9d16cba9fb96 done
Copying config 9617696764 done
Writing manifest to image destination
Storing signatures

<yay! no output!>

[root@container]# podman --cgroup-manager cgroupfs --events-backend file --storage-driver vfs run --rm -v /tmp/foobar:/tmp/foobar:suid,dev,exec quay.io/libpod/alpine:latest grep /tmp/foobar /proc/self/mountinfo
544 523 0:40 /tmp/foobar /tmp/foobar rw,relatime - overlay overlay rw,context="system_u:object_r:container_file_t:s0:c29,c785",lowerdir=/var/lib/containers/storage/overlay/l/J3WNPWC7SZA7N4UXHBLUQJCJUV:/var/lib/containers/storage/overlay/l/UAJWO3N2WGN3FBSZFCZHJHJC42:/var/lib/containers/storage/overlay/l/KEDGJ2OTAZSXZCPPS73NHWYQYL:/var/lib/containers/storage/overlay/l/N76NZ27ZZBBNRJW2EGEGVJOW4D:/var/lib/containers/storage/overlay/l/JHOVFBRNXLQUA7BFVGJFGFC6W6:/var/lib/containers/storage/overlay/l/FB2AB2COHKTBUJRAJWLHKYFHLT:/var/lib/containers/storage/overlay/l/YLL2B2MGWSOOPVR2V4J26JZ3K6:/var/lib/containers/storage/overlay/l/4OWJCV24SKKUFCAFXWQV2CNFRT,upperdir=/var/lib/containers/storage/overlay/0d1034a9ed0afc2e02f337f7fbbc844b785b99957082bfdc8651cc5ac88989a9/diff,workdir=/var/lib/containers/storage/overlay/0d1034a9ed0afc2e02f337f7fbbc844b785b99957082bfdc8651cc5ac88989a9/work

[root@container]# cat /proc/self/mountinfo
525 433 0:40 / / rw,relatime - overlay overlay rw,context="system_u:object_r:container_file_t:s0:c29,c785",lowerdir=/var/lib/containers/storage/overlay/l/J3WNPWC7SZA7N4UXHBLUQJCJUV:/var/lib/containers/storage/overlay/l/UAJWO3N2WGN3FBSZFCZHJHJC42:/var/lib/containers/storage/overlay/l/KEDGJ2OTAZSXZCPPS73NHWYQYL:/var/lib/containers/storage/overlay/l/N76NZ27ZZBBNRJW2EGEGVJOW4D:/var/lib/containers/storage/overlay/l/JHOVFBRNXLQUA7BFVGJFGFC6W6:/var/lib/containers/storage/overlay/l/FB2AB2COHKTBUJRAJWLHKYFHLT:/var/lib/containers/storage/overlay/l/YLL2B2MGWSOOPVR2V4J26JZ3K6:/var/lib/containers/storage/overlay/l/4OWJCV24SKKUFCAFXWQV2CNFRT,upperdir=/var/lib/containers/storage/overlay/0d1034a9ed0afc2e02f337f7fbbc844b785b99957082bfdc8651cc5ac88989a9/diff,workdir=/var/lib/containers/storage/overlay/0d1034a9ed0afc2e02f337f7fbbc844b785b99957082bfdc8651cc5ac88989a9/work
...cut...