Rootless Containers: no such file or directory: OCI not found

[Hyper200]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

1. Install RHEL 8.4

2. Install Podman

3. reboot

4. podman create docker.io/library/registry:2

5. podman start container_name
   Describe the results you received:
   Error: unable to start container "2820a02215ef24c41d2eee82167c1a911cbfd44e6b8d9649295dcdacf4639d0b": container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: open /sys/fs/cgroup/user.slice/user-1007.slice/[email protected]/user.slice/libpod-2820a02215ef24c41d2eee82167c1a911cbfd44e6b8d9649295dcdacf4639d0b.scope/pids.max: no such file or directory: OCI runtime attempted to invoke a command that was not found

Describe the results you expected:
Expected the system to start the container

Additional information you deem important (e.g. issue happens only occasionally):
Allways - On around 4 identical machines

Output of podman version:

Version:      3.2.3
API Version:  3.2.3
Go Version:   go1.15.7
Built:        Tue Jul 27 07:29:39 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.21.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.29-1.module+el8.4.0+11822+6cc1e7d7.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: ae467a0c8001179d4d0adf4ada381108a893d7ec'
  cpus: 4
  distribution:
    distribution: '"rhel"'
    version: "8.4"
  eventLogger: file
  hostname: svd-log-01
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1008
      size: 1
    - container_id: 1
      host_id: 200000
      size: 1001
    - container_id: 1002
      host_id: 558752
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1007
      size: 1
    - container_id: 1
      host_id: 200000
      size: 1001
    - container_id: 1002
      host_id: 558752
      size: 65536
  kernel: 4.18.0-305.12.1.el8_4.x86_64
  linkmode: dynamic
  memFree: 15014359040
  memTotal: 16546336768
  ociRuntime:
    name: runc
    package: runc-1.0.0-74.rc95.module+el8.4.0+11822+6cc1e7d7.x86_64
    path: /usr/bin/runc
    version: |-
      runc version spec: 1.0.2-dev
      go: go1.15.13
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    path: /run/user/1007/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.module+el8.4.0+11822+6cc1e7d7.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 18m 2.15s
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/container/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 0
    stopped: 4
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.6-1.module+el8.4.0+11822+6cc1e7d7.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.6
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /home/container/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 2
  runRoot: /tmp/podman-run-1007/containers
  volumePath: /home/container/.local/share/containers/storage/volumes
version:
  APIVersion: 3.2.3
  Built: 1627370979
  BuiltTime: Tue Jul 27 07:29:39 2021
  GitCommit: ""
  GoVersion: go1.15.7
  OsArch: linux/amd64
  Version: 3.2.3

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Virtual machine, loginctl enable-linger for user enabled

[mheon]

Missing PID cgroup controller, possibly?

@giuseppe Haven't we dealt with this one recently?

[mheon]

Can you install the crun OCI runtime and try and run a container using it (--runtime crun to use it)?

[Hyper200]

Hi All,

I used crun, that made no diffrence (the error message changed slighly), after a bit of redhat support and some digging i found out that our systemd user sessions where not starting with the error:

Aug 11 11:34:05 svd-log-01 systemd[1893]: PAM failed: Conversation error
Aug 11 11:34:05 svd-log-01 systemd[1893]: [email protected]: Failed to set up PAM session: Operation not permitted

It turned out to be this specific entry in the pam that was causing it:

session required pam_lastlog.so showfailed

in system-auth

This being disabled, the user sessions now work and the podman commands run as expected.

We've had this config in our system-auth for years, so i'm a bit suprised about this. I also had no idea that podman required systemd user sessions (but it makes sense)

The actual issue is a bit bizarre and i've raised a redhat support request on it and will get them to create a bug.

Thanks

Joe.

[mheon]

Hm. That's actually a very strange error message for a missing systemd user session - I would have expected us to blow up trying to talk to the systemd dbus session in order to create a cgroup (which would also be a bad error message, but would provide a lot more breadcrumbs than this one).

[flouthoc]

@Hyper200 @mheon I think following controllers are mandatory for rootless cgroup v2: cpu cpuset io memory pids , @Hyper200 could you please share output of cat /proc/self/cgroup , cat /proc/cgroups and cat /proc/self/mountinfo . If above controllers are not their consider enabling them.

[Hyper200]

As Requested

cat /proc/self/cgroup
0::/user.slice/user-1002.slice/session-48.scope
cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 0 359 1
cpu 0 359 1
cpuacct 0 359 1
blkio 0 359 1
memory 0 359 1
devices 0 359 1
freezer 0 359 1
net_cls 0 359 1
perf_event 0 359 1
net_prio 0 359 1
hugetlb 0 359 1
pids 0 359 1
rdma 0 359 1
[root@svd-log-01 mad56570]# cat /proc/self/mountinfo
22 63 0:21 / /sys rw,nosuid,nodev,noexec,relatime shared:2 - sysfs sysfs rw,seclabel
23 63 0:5 / /proc rw,nosuid,nodev,noexec,relatime shared:15 - proc proc rw
24 63 0:6 / /dev rw,nosuid shared:11 - devtmpfs devtmpfs rw,seclabel,size=8062044k,nr_inodes=2015511,mode=755
25 22 0:7 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:3 - securityfs securityfs rw
26 24 0:22 / /dev/shm rw,nosuid,nodev shared:12 - tmpfs tmpfs rw,seclabel
27 24 0:23 / /dev/pts rw,nosuid,noexec,relatime shared:13 - devpts devpts rw,seclabel,gid=5,mode=620,ptmxmode=000
28 63 0:24 / /run rw,nosuid,nodev shared:14 - tmpfs tmpfs rw,seclabel,mode=755
29 22 0:25 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:4 - cgroup2 cgroup2 rw,seclabel,nsdelegate
30 22 0:26 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:5 - pstore pstore rw,seclabel
31 22 0:27 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:6 - efivarfs efivarfs rw
32 22 0:28 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:7 - bpf bpf rw,mode=700
33 22 0:12 / /sys/kernel/tracing rw,relatime shared:8 - tracefs none rw,seclabel
60 22 0:29 / /sys/kernel/config rw,relatime shared:9 - configfs configfs rw
63 1 253:0 / / rw,relatime shared:1 - xfs /dev/mapper/vg_SVD--SVD--LOG01--00-lv_root rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
34 22 0:20 / /sys/fs/selinux rw,relatime shared:10 - selinuxfs selinuxfs rw
35 24 0:30 / /dev/hugepages rw,relatime shared:16 - hugetlbfs hugetlbfs rw,seclabel,pagesize=2M
36 22 0:8 / /sys/kernel/debug rw,relatime shared:17 - debugfs debugfs rw,seclabel
37 23 0:31 / /proc/sys/fs/binfmt_misc rw,relatime shared:18 - autofs systemd-1 rw,fd=41,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=23619
38 24 0:19 / /dev/mqueue rw,relatime shared:19 - mqueue mqueue rw,seclabel
39 22 0:32 / /sys/fs/fuse/connections rw,relatime shared:20 - fusectl fusectl rw
83 63 253:7 / /tmp rw,nosuid,nodev,relatime shared:41 - xfs /dev/mapper/vg_SVD--SVD--LOG01--00-LogVol00 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
86 63 253:4 / /var rw,nodev,relatime shared:43 - xfs /dev/mapper/vg_SVD--SVD--LOG01--00-LogVol03 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
89 63 253:5 / /home rw,nodev,relatime shared:45 - xfs /dev/mapper/vg_SVD--SVD--LOG01--00-LogVol02 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
92 63 8:2 / /boot rw,relatime shared:47 - xfs /dev/sda2 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
95 63 253:6 / /opt rw,nodev,relatime shared:49 - xfs /dev/mapper/vg_SVD--SVD--LOG01--00-LogVol01 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
98 86 253:3 / /var/log rw,nosuid,nodev,noexec,relatime shared:51 - xfs /dev/mapper/vg_SVD--SVD--LOG01--00-LogVol04 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
101 86 253:2 / /var/lib/elasticsearch rw,nodev,relatime shared:53 - xfs /dev/mapper/vg_SVD--SVD--LOG01--00-LogVol05 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
104 92 8:1 / /boot/efi rw,relatime shared:55 - vfat /dev/sda1 rw,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro
223 86 0:33 / /var/lib/nfs/rpc_pipefs rw,relatime shared:115 - rpc_pipefs sunrpc rw
297 28 0:34 / /run/user/1002 rw,nosuid,nodev,relatime shared:156 - tmpfs tmpfs rw,seclabel,size=1615852k,mode=700,uid=1002,gid=1002
309 28 0:36 / /run/user/1008 rw,nosuid,nodev,relatime shared:163 - tmpfs tmpfs rw,seclabel,size=1615852k,mode=700,uid=1008,gid=1009
442 28 0:38 / /run/user/1007 rw,nosuid,nodev,relatime shared:170 - tmpfs tmpfs rw,seclabel,size=1615852k,mode=700,uid=1007,gid=1008
511 63 0:39 / /media/netcompany_share rw,relatime shared:258 - cifs //10.54.4.34/MottMacdonald rw,vers=3.1.1,cache=strict,username=SVC_MottMacdonald,uid=0,noforceuid,gid=0,noforcegid,addr=10.54.4.34,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1
425 28 0:35 / /run/user/1010 rw,nosuid,nodev,relatime shared:221 - tmpfs tmpfs rw,seclabel,size=1615852k,mode=700,uid=1010,gid=1011

[flouthoc]

ah i missed reading #11197 (comment) , but indeed error message does not seems related at all 😕 . @Hyper200 is this resolved for you by making sure user sessions are up.

@mheon @giuseppe I am not sure but should we check at podman layer if there are no sessions available and end early with more intuitive error message ?

[Hyper200]

@flouthoc no worries - Sorry i though that infomation was helpful for you in another way!

I would welcome a better error message (having had to figure out on my own that it was related)

[giuseppe]

I think the issue on RHEL could be: https://bugzilla.redhat.com/show_bug.cgi?id=1897579

[jeremy-chua]

Hi All,

I used crun, that made no diffrence (the error message changed slighly), after a bit of redhat support and some digging i found out that our systemd user sessions where not starting with the error:

Aug 11 11:34:05 svd-log-01 systemd[1893]: PAM failed: Conversation error
Aug 11 11:34:05 svd-log-01 systemd[1893]: [email protected]: Failed to set up PAM session: Operation not permitted

It turned out to be this specific entry in the pam that was causing it:

session required pam_lastlog.so showfailed

in system-auth

This being disabled, the user sessions now work and the podman commands run as expected.

We've had this config in our system-auth for years, so i'm a bit suprised about this. I also had no idea that podman required systemd user sessions (but it makes sense)

The actual issue is a bit bizarre and i've raised a redhat support request on it and will get them to create a bug.

Thanks

Joe.

I had the same issue. How do you actually disable the system-auth pam issue?

[flouthoc]

@jeremy-chua You need to reset systemd session if systemd is being used please check resolution steps here https://github.com/containers/podman/blob/main/troubleshooting.md#30-podman-run-fails-with-erro0000-xdg_runtime_dir-directory-runuser0-is-not-owned-by-the-current-user-or-error-error-creating-tmpdir-mkdir-runuser1000-permission-denied

[jeremy-chua]

I think its a separate error.
I tried the following and still same error.

[admin@one-system one-system]$ loginctl enable-linger admin
[admin@one-system one-system]$ podman run alpine
Error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: open /sys/fs/cgroup/user.slice/user-1000.slice/[email protected]/user.slice/libpod-4355518cf344b76cfc1768e95c60113a9594f7d880fb3e03caedc99064f60d7a.scope/pids.max: no such file or directory: OCI runtime attempted to invoke a command that was not found

[flouthoc]

@jeremy-chua I think this is dup of #11632

[jeremy-chua]

i reloaded systemd deamon and it worked.

sudo systemctl daemon-reload

[flouthoc]

@jeremy-chua Would you like to add this to resolution steps in troubleshooting docs ?

[jeremy-chua]

my bad, it doesn't seem to work after i restarted the OS.
still trying hard to figure out the solution.

[eztiwan]

Any luck further on this @jeremy-chua?

[jeremy-chua]

Any luck further on this @jeremy-chua?

My bad for not responding as i shifted to FCOS. which comes with podman/docker and kubermetes.