Podman + runc: Cannot mount read-only filesystem on host as a volume

[afbjorklund]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Trying to use rootless containers on ubuntu 21.10 with podman 3.2.1 using the default "runc" fails.

Installing "crun" makes it work again, with runc the volume mounts are failing (OCI permission denied)

Steps to reproduce the issue:

1. limactl start examples/podman.yaml

2. export CONTAINER_HOST=unix://$HOME/podman.sock

3. podman --remote run -it -v $PWD/hello:/hello docker.io/busybox cat /hello/world

Describe the results you received:

Error: error preparing container e552b677afd051af9dccb902ea038e527445a5daa1d08d59623364907700d497 for attach: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/home/anders/lima/hello" to rootfs at "/hello" caused: operation not permitted: OCI permission denied

Describe the results you expected:

Hello, World!

Additional information you deem important (e.g. issue happens only occasionally):

It works after installing crun, it fails with only runc (which is the apt default)

limactl shell podman sudo apt install -y crun

Output of podman version:

WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available 
WARN[0000] For using systemd, you may need to login using an user session 
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root) 
WARN[0000] Falling back to --cgroup-manager=cgroupfs    
Version:      3.2.1
API Version:  3.2.1
Go Version:   go1.16.7
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

(paste your output here)

Package info (e.g. output of rpm -q podman or apt list podman):

podman/impish,now 3.2.1+ds1-2ubuntu3 amd64 [installed]

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Ubuntu 21.10

runc version 1.0.1-0ubuntu2
spec: 1.0.2-dev
go: go1.16.5
libseccomp: 2.5.1

https://packages.ubuntu.com/impish/podman

https://packages.ubuntu.com/impish/runc

[mheon]

I recall seeing similar things on RHEL, which turned out to be a too-old runc - newer version fixed it.

[afbjorklund]

Could be a bug in the podman packaging then, currently it says "runc (>= 1.0.0~rc92~) | crun"

And if course runc is a system package, so that means that it is stuck on OS (currently runc 1.0.1)

But I will try uninstall crun, and install 1.0.2 from binary

EDIT: Nope, same issue also with runc 1.0.2 (latest) ?

[rhatdan]

If this is a runc issue, it should be opened over their. If this is a runc version issue, then Podman upstream can not fix. Is there anything upstream Podman can do to fix this, then we need to close the issue. (We are in bugfix mode and need to get our issues under 200.) :^)

[afbjorklund]

It's possible to use docker with runc, and podman with crun. Just not podman with runc, which might be expected ?

Something about the fuse-mounted directory, that makes it fail the mount when podman tries to make the OCI do it.

TARGET       SOURCE        FSTYPE     OPTIONS
/home/anders :/home/anders fuse.sshfs ro,nosuid,nodev,relatime,user_id=1000,group_id=1000,allow_other

This was rootless containers under cgroups v2, it did work if you used sudo podman.

[mheon]

Update: I still think this is a runc issue, but on examining the original issue I thought it was, it is not (that bug was specific to --pid=host).

Would be interesting to see an OCI spec from Docker/Containerd vs one from Podman. Fix is probably still in runc but knowing that exactly is causing it to blow up would help a lot.

[afbjorklund]

You should be able to duplicate this on your own machine, by using the "docker" and "podman" example files.

https://github.com/lima-vm/lima/tree/master/examples

[afbjorklund]

@mheon : It turned out to be very simple, podman was trying to mount the read-only file system as rw

podman

    {
      "destination": "/hello",
      "type": "bind",
      "source": "/home/anders/lima/hello",
      "options": [
        "rw",
        "rprivate",
        "nosuid",
        "nodev",
        "rbind"
      ]
    },

docker

    {
      "destination": "/hello",
      "type": "bind",
      "source": "/home/anders/lima/hello",
      "options": [
        "rbind",
        "rprivate",
        "nosuid",
        "ro",
        "nodev"
      ]
    },

Changing the volume mapping to :ro makes it work, but it should probably be able to detect this ?

[mheon]

That definitely seems like a bug - concur we should detect this. I'll rename the issue.

[afbjorklund]

crunhappily accepts the OCI spec with read-write - and then goes on to mount it read-only (like the original fs).

runc tries to run the OCI spec with the read-write, so that is why docker changes the spec so that it doesn't fail

---

it works for root:

21658 mount("/home/anders/lima/hello", "/proc/self/fd/7", 0xc0000c07d6, MS_NOSUID|MS_NODEV|MS_REMOUNT|MS_BIND|MS_REC, NULL) = 0

but not for user:

21868 mount("/home/anders/lima/hello", "/proc/self/fd/7", 0xc0000c67c6, MS_NOSUID|MS_NODEV|MS_REMOUNT|MS_BIND|MS_REC, NULL) = -1 EPERM (Operation not permitted)

[giuseppe]

The way crun implemented it is to first try the mount, and if it fails then check the mount options for the source and attempt again with ro if it is present there.

How to check it beforehand? Does docker adds ro anytime the source is mounted as read only?

[kolyshkin]

@afbjorklund does it work if you run it as podman --remote run -it -v $PWD/hello:/hello:ro docker.io/busybox cat /hello/world (e.g. add ro flag)?

[afbjorklund]

Adding an explicit :ro to the mount made it work also with runc, but would require modifying every command line and specification and would be very intrusive. It would be better if this modification was done by podman, in that case.

Running with root also "worked".

[afbjorklund]

Does docker adds ro anytime the source is mounted as read only?

That is my understanding:

                        if !m.Writable {
                                opts = append(opts, "ro")
                        }

https://github.com/moby/moby/blob/20.10/daemon/oci_linux.go#L640

[afbjorklund]

Anyway, switched over to crun in lima since it seems that runc is not tested as much.
You might want to do the same in the ubuntu packaging too, or perhaps flip the order ?

      export DEBIAN_FRONTEND=noninteractive
      apt-get update
      apt-get install -y podman crun

From the comments above, it seems like podman could add the "ro" instead of "crun".
That would make it work with any runtime, even if it doesn't implement the retry itself ?

Here was the crun only fix:

• Downgrade the default mount flag for bind mounts from 'rw' to 'ro' if the source is mounted read-only #4061

[mheon]

We're not using crun as the default in Ubuntu already? That's news to me.

[afbjorklund]

We're not using crun as the default in Ubuntu already? That's news to me.

The depends line in deb is: runc (>= 1.0.0~rc92~) | crun

Default: runc

The requires line in rpm is: oci-runtime (i.e. crun or runc)

package: containers-common-4:1-30.fc35.noarch
  dependency: oci-runtime
   provider: runc-2:1.0.2-2.fc35.x86_64
   provider: crun-1.3-1.fc35.x86_64

Default: crun

If you want "the other", you have to add it to the installation:

apt install podman crun

dnf install podman runc

[kolyshkin]

Tried to repro this with runc and a read-only mount.

Was able to reproduce with fuse-sshfs.

Was not able to reproduce with

1. a directory read-only bind-mounted to itself;
2. a read-only tmpfs mount;
3. a read-only ext2 mount on a loop device.

So, for some reason, this is really specific to fuse. Maybe a bug in fuse?

[afbjorklund]

It was seen when running podman under lima. It mounts the home directory read-only (reverse sshfs) by default.

It has been "fixed", by changing from runc to crun. Leaving the other one to be used only with nerdctl and docker.

[afbjorklund]

So, for some reason, this is really specific to fuse. Maybe a bug in fuse?

On the command line we use fusermount rather than mount, maybe something similar is needed for this remount ?

[kolyshkin]

On the command line we use fusermount rather than mount, maybe something similar is needed for this remount ?

What runtime does is a bind mount, so it does not really matter if this is a fuse mount or not.

Anyway, I am fixing this in runc (and the most complicated part is writing a test case).

[kolyshkin]

@afbjorklund can you test the fix? opencontainers/runc#3283

(You can get the static binary by navigating to Checks -> validate, when at the very bottom of the page under "Artifacts" you'll find "release-NNNNN" file which contains pre-built binaries).

[cmoulliard]

I also suffer from a similar issue when I try to mount volumes between my mac machine and the podman vm

 podman --remote --log-level=debug --remote run -it -v $PWD/workspace:/workspace:ro -v $PWD/cahche:/cache:rw -it kaniko-app
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman --remote --log-level=debug --remote run -it -v /Users/cmoullia/code/redhat-buildpacks/poc/kaniko/workspace:/workspace:ro -v /Users/cmoullia/code/redhat-buildpacks/poc/kaniko/cahche:/cache:rw -it kaniko-app) 
DEBU[0000] SSH Ident Key "/Users/cmoullia/.ssh/macvm" SHA256:lWpZxjgjPZVaziq2Tkiz+YUkJzs5QeWBVWaTrFOFb18 ssh-ed25519 
DEBU[0000] Found SSH_AUTH_SOCK "/private/tmp/com.apple.launchd.WoPNlEeF4F/Listeners", ssh-agent signer(s) enabled 
DEBU[0000] DoRequest Method: GET URI: http://d/v3.4.2/libpod/_ping 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] No credentials for registry.tanzu.vmware.com found 
DEBU[0000] No credentials for quay.io found             
DEBU[0000] No credentials for registry.local:5000 found 
DEBU[0000] No credentials for https://index.docker.io/v1/ found 
DEBU[0000] No credentials for registry.pivotal.io found 
DEBU[0000] No credentials for localhost:5000 found      
DEBU[0000] No credentials for 95.217.159.244:32500 found 
DEBU[0000] No credentials for dev.registry.pivotal.io found 
DEBU[0000] DoRequest Method: POST URI: http://d/v3.4.2/libpod/images/pull 
DEBU[0000] User mount /Users/cmoullia/code/redhat-buildpacks/poc/kaniko/workspace:/workspace options [ro] 
DEBU[0000] User mount /Users/cmoullia/code/redhat-buildpacks/poc/kaniko/cahche:/cache options [rw] 
DEBU[0000] DoRequest Method: POST URI: http://d/v3.4.2/libpod/containers/create 
Error: statfs /Users/cmoullia/code/redhat-buildpacks/poc/kaniko/workspace: no such file or directory
...

ls -la /Users/cmoullia/code/redhat-buildpacks/poc/kaniko/workspace
total 16
drwxr-xr-x   4 cmoullia  staff  128 Nov 18 15:39 .
drwxr-xr-x  10 cmoullia  staff  320 Nov 18 16:51 ..
-rw-r--r--   1 cmoullia  staff   63 Nov 18 15:39 Dockerfile
-rw-r--r--   1 cmoullia  staff   18 Nov 18 15:02 hello.txt

@kolyshkin

[kolyshkin]

@cmoulliard this issue is about podman + runc inability to bind-mount a read-only fuse mount to a container (because podman do es not add ro flag to the runtime spec, and runc does not retry with ro).

What you see is a different and most probably unrelated issue; please feel free to file a separate bug.

[afbjorklund]

@cmoulliard Note that the containers mount files on the VM, so you need to have some remote filesystem (like 9p or sshfs) mounted if you want to access files on your host. You can check the available files with podman machine ssh ls /

• Issue Support native source folder for volume mount in remote model #8016

Due to a limitation in CoreOS (i.e. read-only), it is not possible to mount at the path /Users.

[afbjorklund]

@kolyshkin : I can confirm that this new runc binary fixes the issue, where the old runc still shows it.

opencontainers/runc@d7846bc (actual commit, as per PR)

opencontainers/runc@ecf09297 (merge commit, per binary)

working:

runc version 1.0.0+dev
commit: v1.0.0-379-gecf09297
spec: 1.0.2-dev
go: go1.17.3
libseccomp: 2.5.2

ubuntu: (not working)

runc version 1.0.1-0ubuntu2
spec: 1.0.2-dev
go: go1.16.5
libseccomp: 2.5.1

Validated by creating a lima VM with podman.yaml, and then installing and selecting "runc" as the engine in the config

anders@lima-podman:/home/anders/lima$ more /etc/containers/containers.conf 
[engine]
runtime = "runc"

[rhatdan]

So is this fixed in upstream runc? Or is this something we need to fix in Podman, or can I close this issue?

[afbjorklund]

Once the new runc has been released, it will work with the current Podman and the rw volumes on a read-only sshfs disk.

Until then, one has to use crun since the current runc only works with Docker and its ro volumes (when disk is read-only).

---

Currently this mostly affected lima users, but an update has been pushed there...

Unless some other project (including Podman Machine) also starts using sshfs volumes ?

[rhatdan]

Ok I am going to close this issue, since it is not a issue that Podman can fix. Although I guess we could update the spec file to require the latest runc once it is released.

[afbjorklund]

Current ETA is runc 1.1.0

[kolyshkin]

Current ETA is runc 1.1.0

I did a backport for runc 1.0.3, too, which might come before 1.1.0.