rootless exec: OCI permission denied

[droogmic]

/kind bug

Description

Trying to get podman rootless working in archlinux. https://wiki.archlinux.org/index.php/Linux_Containers#Enable_support_to_run_unprivileged_containers_(optional)

Containers can be started correctly, but exec does not. I think this worked when I first set podman up, but after a few days of tinkering it does not work anymore.

Steps to reproduce the issue:

1. podman run --name=exec_test centos:8 sleep 60 &

2. podman container exec exec_test echo TEST

Describe the results you received:

Error: writing file `/sys/fs/cgroup//user.slice/user-1000.slice/[email protected]/user.slice/libpod-5e22f237b52c26e8845f98ae4e6bf8cdb8fe220ff9d2a904be308d311b4a043a.scope/container/cgroup.procs`: Permission denied: OCI permission denied

Describe the results you expected:
TEST

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.15.6
Git Commit:   a0d478edea7f775b7ce32f8eb1a01e75374486cb
Built:        Tue Dec  8 22:48:23 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.24, commit: 9cbc71d699291dfb14e7c1e348a0d48feff7a27d'
  cpus: 16
  distribution:
    distribution: manjaro
    version: unknown
  eventLogger: journald
  hostname: manjaro20
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.9.16-1-MANJARO
  linkmode: dynamic
  memFree: 3480850432
  memTotal: 16129003520
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 0.16
      commit: eb0145e5ad4d8207e84a327248af76663d4e50dd
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 16073093120
  swapTotal: 16147017728
  uptime: 511h 11m 14.65s (Approximately 21.29 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  configFile: /home/droogmic/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fusermount3 version: 3.10.1
        fuse-overlayfs: version 1.3
        FUSE library version 3.10.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/droogmic/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 54
  runRoot: /run/user/1000/containers
  volumePath: /home/droogmic/.local/share/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 1607464103
  BuiltTime: Tue Dec  8 22:48:23 2020
  GitCommit: a0d478edea7f775b7ce32f8eb1a01e75374486cb
  GoVersion: go1.15.6
  OsArch: linux/amd64
  Version: 2.2.1

Package info (e.g. output of rpm -q podman or apt list podman):

Name            : podman
Version         : 2.2.1-1
Description     : Tool and library for running OCI-based containers in pods
Architecture    : x86_64
URL             : https://github.com/containers/libpod
Licenses        : Apache
Groups          : None
Provides        : None
Depends On      : cni-plugins  conmon  containers-common  device-mapper  iptables  libseccomp  runc  slirp4netns  libsystemd  fuse-overlayfs  libgpgme.so=11-64
Optional Deps   : podman-docker: for Docker-compatible CLI
                  btrfs-progs: support btrfs backend devices [installed]
                  catatonit: --init flag support
                  crun: support for unified cgroupsv2 [installed]
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 79.09 MiB
Packager        : Morten Linderud <[email protected]>
Build Date      : Tue 08 Dec 2020 22:48:23 CET
Install Date    : Wed 30 Dec 2020 23:43:06 CET
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

$ cat /etc/subuid 
root:100000:65536
droogmic:165536:65536
$ cat /etc/subgid 
root:100000:65536
droogmic:165536:65536

[mheon]

@giuseppe This look like a systemd cgroup issue, maybe?

[giuseppe]

can you tell me the output for:

ls -l /sys/fs/cgroup//user.slice/user-1000.slice/[email protected]/user.slice/libpod-5e22f237b52c26e8845f98ae4e6bf8cdb8fe220ff9d2a904be308d311b4a043a.scope/container/cgroup.procs

and

cat /sys/fs/cgroup//user.slice/user-1000.slice/[email protected]/user.slice/libpod-5e22f237b52c26e8845f98ae4e6bf8cdb8fe220ff9d2a904be308d311b4a043a.scope/container/cgroup.controllers?

[giuseppe]

could you also try: systemd-run --scope --user podman container exec exec_test echo TEST ?

[droogmic]

$ ls -l /sys/fs/cgroup//user.slice/user-1000.slice/[email protected]/user.slice/libpod-5e22f237b52c26e8845f98ae4e6bf8cdb8fe220ff9d2a904be308d311b4a043a.scope/container/cgroup.procs
-rw-r--r-- 1 droogmic droogmic 0 Feb  3 21:29 /sys/fs/cgroup//user.slice/user-1000.slice/[email protected]/user.slice/libpod-5e22f237b52c26e8845f98ae4e6bf8cdb8fe220ff9d2a904be308d311b4a043a.scope/container/cgroup.procs

$ cat /sys/fs/cgroup//user.slice/user-1000.slice/[email protected]/user.slice/libpod-5e22f237b52c26e8845f98ae4e6bf8cdb8fe220ff9d2a904be308d311b4a043a.scope/container/cgroup.controllers
memory pids

$ systemd-run --scope --user podman container exec exec_test echo TEST
Running scope as unit: run-rb71407b7f1394c35886fdbdb5a2fad65.scope
TEST

[giuseppe]

ok thanks for confirming it, so it is somehow failing to detect the current cgroup ownership.

What do you get with cat /proc/self/cgroup?

[droogmic]

I am not sure why, but the last reboot appears to have fixed the issue. I was rebooting regularly at the start to try and resolve this issue, so I am a bit confused. I will close the issue until I can continue reproducing it. Apologies.