Empty mounted volumes (rootless)

[ghost]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

podman play kube: In some cases volumes are mounted as empty folders into the containers.

Steps to reproduce the issue:

SCENARIO 1:

1. I have a folder in my home directory with a file in it: /home/podman/testdir/testfile
2. I write a Dockerfile with VOLUME step:

FROM docker.io/centos:centos8

VOLUME /home/podman/testdir/

CMD tail -f /dev/null

3. Create a test.yaml file with a container that mounts this volume:

---

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: pod
  name: pod
spec:
  containers:
  - name: testcontainer
    image: localhost/testimage
    env:
     - name: SOME_ENV
       value: "VALUE"
    ports:
    - containerPort: 111111
      hostPort: 11111
      protocol: TCP
    securityContext:
      allowPrivilegeEscalation: false
      capabilities: {}
      privileged: false
      readOnlyRootFilesystem: true
      runAsGroup: 10000
      runAsUser: 10000
      seLinuxOptions:
        type: spc_t
    volumeMounts:
    - mountPath: /home/podman/testdir/
      name:  testdir

  volumes:
  - name: testdir
    hostPath:
      path: /home/podman/testdir/
      type: Directory

4. Play the file: podman play kube test.yaml
5. List the contents on the folder:

podman exec pod-testcontainer ls /home/podman/testdir/
testfile

The folder is not empty.

---

SCENARIO 2:

Like SCENARIO 1 with a different path: /data/testdir/. /data is a mounted volume on the host:

mount | grep data
/dev/sdb1 on /data ...

Dockerfile:

FROM docker.io/centos:centos8

VOLUME /data/testdir/

CMD tail -f /dev/null

test.yaml:

---

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: pod
  name: pod
spec:
  containers:
  - name: testcontainer
    image: localhost/testimage
    env:
     - name: SOME_ENV
       value: "VALUE"
    ports:
    - containerPort: 111111
      hostPort: 11111
      protocol: TCP
    securityContext:
      allowPrivilegeEscalation: false
      capabilities: {}
      privileged: false
      readOnlyRootFilesystem: true
      runAsGroup: 10000
      runAsUser: 10000
      seLinuxOptions:
        type: spc_t
    volumeMounts:
    - mountPath: /data/testdir/
      name:  testdir

  volumes:
  - name: testdir
    hostPath:
      path: /data/testdir/
      type: Directory

After playing this the folder is empty:

podman exec pod-testcontainer ls /data/testdir/

---

SCENARIO 2B:

Remove the trailing slashes from VOLUME in the Dockerfile and all paths in test.yaml and it works again, the file is in the folder.

---

SCENARIO 2C:

Remove the VOLUME... line from the Dockerfile but keep the trailing slashes in test.yaml and it works, too, the file is in the folder.

Describe the results you received:

In some - I admit, hard to define - scenarios the folder is not mounted and we only get an empty volume that was created because of the VOLUME command in the Dockerfile (at least that's how I understand it).

Describe the results you expected:

The volume should not be empty in any scenario.

Additional information you deem important (e.g. issue happens only occasionally):

SCENARIO 2 used to work with podman 2.1.1.

Output of podman version:

Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.14.12
Built:        Mon Mar  1 15:31:01 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.4
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.26-4.el8.5.1.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.26, commit: c30e47104fbc00a7381abe56ad8d54ab151f3fde'
  cpus: 4
  distribution:
    distribution: '"centos"'
    version: "8"
  eventLogger: journald
  hostname: mymachine
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 795
      size: 1
    - container_id: 1
      host_id: 100000
      size: 10001
    uidmap:
    - container_id: 0
      host_id: 988
      size: 1
    - container_id: 1
      host_id: 100000
      size: 10001
  kernel: 4.18.0-240.10.1.el8_3.x86_64
  linkmode: dynamic
  memFree: 6644281344
  memTotal: 8346820608
  ociRuntime:
    name: crun
    package: crun-0.18-3.el8.5.1.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.18
      commit: 808420efe3dc2b44d6db9f1a3fac8361dde42a95
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /tmp/podman-run-988/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-4.el8.7.1.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 2143285248
  swapTotal: 2147479552
  uptime: 22h 59m 3.58s (Approximately 0.92 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 2
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.4.0-3.el8.4.1.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.4
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /home/podman/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 10
  runRoot: /tmp/podman-run-988/containers
  volumePath: /home/podman/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1614609061
  BuiltTime: Mon Mar  1 15:31:01 2021
  GitCommit: ""
  GoVersion: go1.14.12
  OsArch: linux/amd64
  Version: 3.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

rpm -qa podman
podman-3.0.1-2.el8.3.1.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@g574 interested in opening a PR to fix this?

[ghost]

@rhatdan I'm sorry, I'm just a podman user, no programmer. For now I can live with just removing those slashes.

[EduardoVega]

I can work on this one

[rhatdan]

Thanks @EduardoVega