Rootless podman in rootless podman container fails with inconsistent messages #20812

adelton · 2023-11-28T13:04:22Z

Issue Description

I try to debug some rootless setups in OpenShift per https://www.redhat.com/sysadmin/podman-inside-kubernetes. I know it says

Disable SELinux: SELinux does not allow containerized processes to mount all of the file systems required to run inside a container. So we need to disable SELinux on the host that is running the Kubernetes cluster.

and I try to find out what exactly would fail, to possibly amend the SELinux / OpenShift policies. So it is expected that my attempt to run a rootless podman container in a rootless container fails.

However, the error message I get when running a rootless container in a rootless container seem not stable which is worrying.

Steps to reproduce the issue

$ podman run --rm -ti --user podman quay.io/podman/stable
[podman@8c2d1ecef2c7 /]$ podman run --rm -ti --user podman quay.io/podman/stable
[podman@8c2d1ecef2c7 /]$ podman run --rm -ti --user podman quay.io/podman/stable

Describe the results you received

$ podman run --rm -ti --user podman quay.io/podman/stable
Trying to pull quay.io/podman/stable:latest...
Getting image source signatures
Copying blob 4d9671c387bd done   | 
Copying blob 6eb636413202 done   | 
Copying blob 2d9850dbb0db done   | 
Copying blob f2ead6108236 done   | 
Copying blob 85325264fc3e done   | 
Copying blob ec2b80c473bf done   | 
Copying blob 0cdb70f634e5 done   | 
Copying blob 5bf68aba73a4 done   | 
Copying blob 6df6e4a6e148 done   | 
Copying config d716b1dbdf done   | 
Writing manifest to image destination
WARN[0006] Path "/run/secrets/etc-pki-entitlement" from "/etc/containers/mounts.conf" doesn't exist, skipping 
WARN[0006] Path "/run/secrets/rhsm" from "/etc/containers/mounts.conf" doesn't exist, skipping 
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
[podman@744b807a6ee9 /]$ podman run --rm -ti --user podman quay.io/podman/stable
WARN[0000] Path "/run/secrets/etc-pki-entitlement" from "/etc/containers/mounts.conf" doesn't exist, skipping 
WARN[0000] Path "/run/secrets/rhsm" from "/etc/containers/mounts.conf" doesn't exist, skipping 
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
[podman@744b807a6ee9 /]$ podman run --rm -ti --user podman quay.io/podman/stable
WARN[0000] Path "/run/secrets/etc-pki-entitlement" from "/etc/containers/mounts.conf" doesn't exist, skipping 
WARN[0000] Path "/run/secrets/rhsm" from "/etc/containers/mounts.conf" doesn't exist, skipping 
Error: crun: set propagation for `proc`: Permission denied: OCI permission denied
[podman@744b807a6ee9 /]$ podman run --rm -ti --user podman quay.io/podman/stable
WARN[0000] Path "/run/secrets/etc-pki-entitlement" from "/etc/containers/mounts.conf" doesn't exist, skipping 
WARN[0000] Path "/run/secrets/rhsm" from "/etc/containers/mounts.conf" doesn't exist, skipping 
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
[podman@744b807a6ee9 /]$ podman run --rm -ti --user podman quay.io/podman/stable
WARN[0000] Path "/run/secrets/etc-pki-entitlement" from "/etc/containers/mounts.conf" doesn't exist, skipping 
WARN[0000] Path "/run/secrets/rhsm" from "/etc/containers/mounts.conf" doesn't exist, skipping 
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
[podman@744b807a6ee9 /]$ podman run --rm -ti --user podman quay.io/podman/stable
WARN[0000] Path "/run/secrets/etc-pki-entitlement" from "/etc/containers/mounts.conf" doesn't exist, skipping 
WARN[0000] Path "/run/secrets/rhsm" from "/etc/containers/mounts.conf" doesn't exist, skipping 
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
[podman@744b807a6ee9 /]$ podman run --rm -ti --user podman quay.io/podman/stable
WARN[0000] Path "/run/secrets/etc-pki-entitlement" from "/etc/containers/mounts.conf" doesn't exist, skipping 
WARN[0000] Path "/run/secrets/rhsm" from "/etc/containers/mounts.conf" doesn't exist, skipping 
Error: crun: set propagation for `proc`: Permission denied: OCI permission denied

Describe the results you expected

I expect the error message to be the same every time.

podman info output

On the host:

$ podman info
host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.8-2.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 98.25
    systemPercent: 0.36
    userPercent: 1.39
  cpus: 2
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    version: "39"
  eventLogger: journald
  freeLocks: 2048
  hostname: cc-vm2p.tpb.lab.eng.brq.redhat.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.5.12-300.fc39.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1435463680
  memTotal: 3029131264
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-2.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.11.2-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.11.2
      commit: ab0edeef1c331840b025e8f1d38090cfb8a0509d
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231107.g56d9f6d-1.fc39.x86_64
    version: |
      pasta 0^20231107.g56d9f6d-1.fc39.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 3028283392
  swapTotal: 3028283392
  uptime: 0h 43m 28.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/test/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/test/.local/share/containers/storage
  graphRootAllocated: 16039018496
  graphRootUsed: 2292322304
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/test/.local/share/containers/storage/volumes
version:
  APIVersion: 4.7.2
  Built: 1698762721
  BuiltTime: Tue Oct 31 15:32:01 2023
  GitCommit: ""
  GoVersion: go1.21.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.2

In the container:

[podman@744b807a6ee9 /]$ podman info
host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.8-2.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 98.01
    systemPercent: 0.42
    userPercent: 1.58
  cpus: 2
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: container
    version: "39"
  eventLogger: file
  freeLocks: 2048
  hostname: 744b807a6ee9
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
  kernel: 6.5.12-300.fc39.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 871120896
  memTotal: 3029131264
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-2.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.11.2-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.11.2
      commit: ab0edeef1c331840b025e8f1d38090cfb8a0509d
      rundir: /tmp/podman-run-1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231107.g56d9f6d-1.fc39.x86_64
    version: |
      pasta 0^20231107.g56d9f6d-1.fc39.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /tmp/podman-run-1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 3028283392
  swapTotal: 3028283392
  uptime: 0h 45m 21.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/podman/.local/share/containers/storage
  graphRootAllocated: 16039018496
  graphRootUsed: 2841436160
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /tmp/containers-user-1000/containers
  transientStore: false
  volumePath: /home/podman/.local/share/containers/storage/volumes
version:
  APIVersion: 4.7.2
  Built: 1698762721
  BuiltTime: Tue Oct 31 14:32:01 2023
  GitCommit: ""
  GoVersion: go1.21.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.2



### Podman in a container

Yes

### Privileged Or Rootless

Rootless

### Upstream Latest Release

Yes

### Additional environment details

Additional environment details

### Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

The text was updated successfully, but these errors were encountered:

rhatdan · 2023-11-29T12:04:57Z

Try:
podman run --rm -ti --security-opt unmask=/proc --user podman quay.io/podman/stable

We need to fix these Warnings about the /run/secrets directory.

rhatdan · 2023-11-29T12:07:20Z

Also can you do
podman run --rm -ti -v /dev/null:/etc/containers/mounts.conf --security-opt unmask=/proc --user podman quay.io/podman/stable
To see if this cleans it up.

adelton · 2023-11-30T16:50:16Z

Do you mean to try these for the first podman, or that podman-in-podman invocation?

rhatdan · 2023-12-01T00:11:32Z

First podman, Kernel does not allow a processes to modify a modified /proc.

adelton · 2023-12-03T08:14:08Z

The behaviour, including the nondeterminism, seems to still be there:

$ podman run --rm -ti -v /dev/null:/etc/containers/mounts.conf --security-opt unmask=/proc --user podman quay.io/podman/stable
[podman@5c678908c060 /]$ mount | grep proc
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /proc/acpi type tmpfs (ro,relatime,context="system_u:object_r:container_file_t:s0:c392,c957",size=0k,uid=1000,gid=1000,inode64)
devtmpfs on /proc/kcore type devtmpfs (ro,nosuid,seclabel,size=4096k,nr_inodes=4062785,mode=755,inode64)
devtmpfs on /proc/keys type devtmpfs (ro,nosuid,seclabel,size=4096k,nr_inodes=4062785,mode=755,inode64)
devtmpfs on /proc/latency_stats type devtmpfs (ro,nosuid,seclabel,size=4096k,nr_inodes=4062785,mode=755,inode64)
devtmpfs on /proc/timer_list type devtmpfs (ro,nosuid,seclabel,size=4096k,nr_inodes=4062785,mode=755,inode64)
tmpfs on /proc/scsi type tmpfs (ro,relatime,context="system_u:object_r:container_file_t:s0:c392,c957",size=0k,uid=1000,gid=1000,inode64)
proc on /proc/asound type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
[podman@5c678908c060 /]$ mount | grep ' on /proc '
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
[podman@5c678908c060 /]$ podman run --rm -ti --user podman quay.io/podman/stable
Trying to pull quay.io/podman/stable:latest...
Getting image source signatures
Copying blob 77c9c34091b4 done   | 
Copying blob 1e910112bc7f done   | 
Copying blob 10327c9af971 done   | 
Copying blob 9fa763129095 done   | 
Copying blob 4e8a4684a6a4 done   | 
Copying blob 718a00fe3212 done   | 
Copying blob fd72a3378718 done   | 
Copying blob 3f30707a1d42 done   | 
Copying blob a681dc7022f8 done   | 
Copying config 314b296d26 done   | 
Writing manifest to image destination
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
[podman@5c678908c060 /]$ podman run --rm -ti --user podman quay.io/podman/stable
Error: crun: set propagation for `proc`: Permission denied: OCI permission denied

dominic-p · 2024-01-13T01:41:35Z

In a possibly related issue, I'm seeing similar warning messages running quay.io/buildah/stable on my CRI-O kubernetes cluster:

WARN[0000] Path "/run/secrets/etc-pki-entitlement" from "/etc/containers/mounts.conf" doesn't exist, skipping 
WARN[0000] Path "/run/secrets/rhsm" from "/etc/containers/mounts.conf" doesn't exist, skipping

rhatdan · 2024-01-15T20:40:21Z

Yes we need to figure out how to fix this on systems that do not have subscption-manager installed on them, Probably drop the Warning to info.

quay.io/buildah/stable and quay.io/podman/stable images now forward the mounts.conf subscriptions into their containers but if the host is not using subscription manager these pass throughs warn about missing files, which is not useful to the user. fixes: containers/podman#20812 Signed-off-by: Daniel J Walsh <[email protected]>

adelton · 2024-01-16T14:20:03Z

This issue however was about that

Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

vs.

Error: crun: set propagation for `proc`: Permission denied: OCI permission denied

not about those mountpoint WARNs.

edsantiago · 2024-01-16T14:30:28Z

"no logs from conmon" is #10927, one of our longest-standing and most annoying flakes. I've never seen it on run, only exec. And I tried your reproducer on my f39 laptop, no luck.

quay.io/buildah/stable and quay.io/podman/stable images now forward the mounts.conf subscriptions into their containers but if the host is not using subscription manager these pass throughs warn about missing files, which is not useful to the user. fixes: containers/podman#20812 Signed-off-by: Daniel J Walsh <[email protected]>

adelton added the kind/bug Categorizes issue or PR as related to a bug. label Nov 28, 2023

rhatdan mentioned this issue Jan 15, 2024

Drop warning to info on missing content from mounts.conf containers/common#1801

Merged

openshift-merge-bot bot closed this as completed in containers/common#1801 Jan 16, 2024

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Apr 17, 2024

github-actions bot locked as resolved and limited conversation to collaborators Apr 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rootless podman in rootless podman container fails with inconsistent messages #20812

Rootless podman in rootless podman container fails with inconsistent messages #20812

adelton commented Nov 28, 2023 •

edited

Loading

rhatdan commented Nov 29, 2023

rhatdan commented Nov 29, 2023

adelton commented Nov 30, 2023

rhatdan commented Dec 1, 2023

adelton commented Dec 3, 2023

dominic-p commented Jan 13, 2024

rhatdan commented Jan 15, 2024

adelton commented Jan 16, 2024

edsantiago commented Jan 16, 2024

Rootless podman in rootless podman container fails with inconsistent messages #20812

Rootless podman in rootless podman container fails with inconsistent messages #20812

Comments

adelton commented Nov 28, 2023 • edited Loading

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

rhatdan commented Nov 29, 2023

rhatdan commented Nov 29, 2023

adelton commented Nov 30, 2023

rhatdan commented Dec 1, 2023

adelton commented Dec 3, 2023

dominic-p commented Jan 13, 2024

rhatdan commented Jan 15, 2024

adelton commented Jan 16, 2024

edsantiago commented Jan 16, 2024

adelton commented Nov 28, 2023 •

edited

Loading