Mount error when using ':z' on podman 3.2

[pascapascapascapasca]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When I start up a pod with a container that has a shared mount (:z), the mount fails. I also see this error in the journal: kernel: overlayfs: unrecognized mount option "volatile" or missing value.

I only get this issue since I've updated to podman 3.2. Other mount types (:Z and :ro) seem to work fine.

I tried switching runtimes between runc and crun and both failed.

I'm on AlmaLinux 8.4.

Steps to reproduce the issue:

1. Start pod with podman play kube with this definition. I use this one as example, but, it happens to all my pods with :z mounts. The config.yml file exists on disk at the right location.

---
apiVersion: v1
kind: Pod
metadata:
  name: etcd-pod
  labels:
    app.kubernetes.io/name: etcd
    app.kubernetes.io/instance: <...>
    group: <...>
spec:
  hostNetwork: true
  restartPolicy: Never
  volumes:
    - name: etcd
      hostpath:
        path: /var/lib/etcd
  containers:
    - name: etcd
      image: gcr.io/etcd-development/etcd:v3.4.15
      volumeMounts:
        - name: etcd
          mountPath: /etcd:z
      env:
        - name: ECTDCTL_API
          value: 3
      command:
        - /usr/local/bin/etcd
        - --config-file
        - /etcd/config.yml

2. Start container with podman start -a etcd-pod-etcd

Describe the results you received:

The container does not mount the volume and the command fails.

E | etcdmain: error verifying flags, open /etcd/config.yml: no such file or directory. See 'etcd --help'.

I also see kernel: overlayfs: unrecognized mount option "volatile" or missing value in the journal.

Describe the results you expected:

Volume should be mounted with :z.

Output of podman version:

Version:      3.2.2
API Version:  3.2.2
Go Version:   go1.15.13
Built:        Fri Jul  9 20:40:27 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:                                                                                                                                                                                                       [45/5776]
  arch: amd64
  buildahVersion: 1.21.0
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.29-1.el8.3.5.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: '
  cpus: 2
  distribution:
    distribution: '"almalinux"'
    version: "8.4"
  eventLogger: journald
  hostname: appservices_alma.local
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-305.7.1.el8_4.x86_64
  linkmode: dynamic
  memFree: 113958912
  memTotal: 3918151680
  ociRuntime:
    name: crun
    package: crun-0.20.1-1.el8.3.2.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.20.1
      commit: 0d42f1109fd73548f44b01b3e84d04a279e99d2e
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 1h 5m 24.89s (Approximately 0.04 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 11
    paused: 0
    running: 8
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 13
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.2.2
  Built: 1625863227
  BuiltTime: Fri Jul  9 20:40:27 2021
  GitCommit: ""
  GoVersion: go1.15.13
  OsArch: linux/amd64
  Version: 3.2.2

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.2.2-1.el8.2.1.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

OpenStack

[mheon]

@rhatdan PTAL

[rhatdan]

This seems like the :z is a red herring. The volatile support is most likely the real issue.

if you do

$ mkdir /tmp/foo
$ podman run -v /tmp/foo:/foo:z fedora ls -lZd /foo

Does it work.

[pascapascapascapasca]

It seems to work indeed.

# podman run -v /tmp/foo:/foo:z fedora ls -lZd /foo
drwxr-x---. 2 root root system_u:object_r:container_file_t:s0 6 Jul 19 18:03 /foo

What does that indicate? I am not familiar with volatile mounts. Is there anything in my config that specifies volatile mounts?

[rhatdan]

No they happen under the covers. They are only supposed to be used if the kernel supports them though, and the OS containers/storage should have checked during the initialization phase. @giuseppe WDYT?

[rhatdan]

When I play your kube.yaml it works fine.

# podman play kube /tmp/kube.yaml 
Trying to pull gcr.io/etcd-development/etcd:v3.4.15...
Getting image source signatures
Copying blob e1d5804360f1 done  
Copying blob 481a903d31c6 done  
Copying blob 621b2e0141ca done  
Copying blob 39fafc05754f done  
Copying blob 21c549d41ff0 done  
Copying blob 543c1ea18bb7 done  
Copying config 87c4f81d88 done  
Writing manifest to image destination
Storing signatures
Pod:
d9630de4ec9c2ef18b9715a291ee7175b3862133445ffd7036e730c4dd19c8a6
Container:
644d63245cf49c5ab405c2addba84d90c0e3fb90c988f572216b0d798cb76ad7

# podman -v
podman version 3.2.2

[rhatdan]

Ok there is still a bug, we created the etcd:z/ inside of the container and never fixed the label.

[pascapascapascapasca]

@rhatdan the play kube works for me too. The error happens when trying to use the mounted volume.

[root@appservices_alma ~]# podman play kube /etc/rbbn/appservices/def/etcd.yaml
Pod:
6af0c10bb9a865cd39df8dcccbf8dbe0e5180dd7bd0d04b8d504ea3d6069eee3
Container:
2a94b8489cf3d69952cc0733af10db6f28f3658913e37bd4ea191463e085c00a

[root@appservices_alma ~]# podman start -a etcd-pod-etcd
2021-07-20 13:19:16.203252 I | etcdmain: Loading server configuration from "/etcd/config.yml". Other configuration command line flags and environment variables will be ignored if provided.
2021-07-20 13:19:16.203283 E | etcdmain: error verifying flags, open /etcd/config.yml: no such file or directory. See 'etcd --help'.


[root@appservices_alma ~]# ls -lZ /var/lib/appservices/etcd/
total 4
-rw-r-xr-x. 1 root root system_u:object_r:container_file_t:s0 800 Jul 16 19:28 config.yml
drw-r-xr-x. 3 root root system_u:object_r:container_file_t:s0  20 Jul 20 13:14 data

[root@appservices_alma ~]# cat /etc/rbbn/appservices/def/etcd.yaml
---
apiVersion: v1
kind: Pod
metadata:
  name: etcd-pod
  labels:
    app.kubernetes.io/name: etcd
    app.kubernetes.io/instance: etcd-appservices_alma
    group: appservices
spec:
  hostNetwork: true
  restartPolicy: Never
  volumes:
    - name: etcd
      hostpath:
        path: /var/lib/appservices/etcd
  containers:
    - name: etcd
      image: gcr.io/etcd-development/etcd:v3.4.15
      volumeMounts:
        - name: etcd
          mountPath: /etcd:z
      env:
        - name: ECTDCTL_API
          value: 3
      command:
        - /usr/local/bin/etcd
        - --config-file
        - /etcd/config.yml

[root@appservices_alma ~]# podman -v
podman version 3.2.2

If I remove the :z and re-create the pod, the start command works, but it won't work on a new system since the SELinux labels won't be changed.

Could you try that too?

[rhatdan]

Could you try with the patch in #10992, I also found a bug.

[pascapascapascapasca]

Works with this patch! Thanks!

[rhatdan]

Could you add a LGTM with that comment to the PR, so I can merge.

[pascapascapascapasca]

Sorry, was slow on this one!