Reproduced #9219 but rebooting does not fix the issue

[m0rr1gan]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
Reproduced #9219 but rebooting does not fix the issue.

Steps to reproduce the issue:

1. podman run --name=test -d docker.io/archlinux sleep 600

2. podman exec test pwd
   Errors with "Error: writing file `/sys/fs/cgroup/user.slice/user-1000.slice/[email protected]/user.slice/libpod-602619c8c44358b2be3022e72ab02970adcaa6bb628a8529c4baba464f7b2cd2.scope/container/cgroup.procs`: Permission denied: OCI permission denied"

3. systemd-run --scope --user podman exec test pwd
   Exists correctly, printing "/".

Describe the results you received:
Unable to exec a command in a container. The container starts and runs to completion successfully.

Describe the results you expected:
Not have to use systemd-run --scope --user ... to define cgroup ownership.

Additional information you deem important (e.g. issue happens only occasionally):
Issue happens consistently, after restarts.
Output of podman version:

Version:      3.1.2
API Version:  3.1.2
Go Version:   go1.16.4
Git Commit:   51b8ddbc22cf5b10dd76dd9243924aa66ad7db39
Built:        Sat May 22 17:22:34 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.20.1
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.0.28-1
    path: /usr/bin/conmon
    version: 'conmon version 2.0.28, commit: 6b18f7e0f2e4cd7f7b016b88141e82210d370008'
  cpus: 24
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: whitearch
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.12.6-arch1-1
  linkmode: dynamic
  memFree: 58380513280
  memTotal: 67413966848
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 0.19.1-1
    path: /usr/bin/crun
    version: |-
      crun version 0.19.1
      commit: 1535fedf0b83fb898d449f9680000f729ba719f5
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.10-1
    version: |-
      slirp4netns version 1.1.10
      commit: baa2bc5ff12fe6db646c1f4f3f966526c0eba5a0
      libslirp: 4.5.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 50m 22.12s
registries: {}
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/user/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 3.1.2
  Built: 1621718554
  BuiltTime: Sat May 22 17:22:34 2021
  GitCommit: 51b8ddbc22cf5b10dd76dd9243924aa66ad7db39
  GoVersion: go1.16.4
  OsArch: linux/amd64
  Version: 3.1.2

Package info (e.g. output of rpm -q podman or apt list podman):

> pacman -Qi podman
Name            : podman
Version         : 3.1.2-2
Description     : Tool and library for running OCI-based containers in pods
Architecture    : x86_64
URL             : https://github.com/containers/libpod
Licenses        : Apache
Groups          : None
Provides        : None
Depends On      : cni-plugins  conmon  containers-common  device-mapper  iptables
                  libseccomp  runc  slirp4netns  libsystemd  fuse-overlayfs
                  libgpgme.so=11-64
Optional Deps   : podman-docker: for Docker-compatible CLI
                  btrfs-progs: support btrfs backend devices [installed]
                  catatonit: --init flag support [installed]
                  crun: support for unified cgroupsv2 [installed]
Required By     : podman-compose
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 72.52 MiB
Packager        : Morten Linderud <[email protected]>
Build Date      : Sat 22 May 2021 05:22:34 PM EDT
Install Date    : Sat 22 May 2021 10:43:20 PM EDT
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes, using the most recent release and no info about this issue is on the troubleshooting page.

Additional environment details (AWS, VirtualBox, physical, etc.):

Bare metal install of Archlinux, up to date. Using Apparmor as the default security model and the lockdown=confidentiality kernel parameter.

/etc/sub{uid,gid} contains
user:100000:65536

Output of command requested but not answered in linked issue:

> cat /proc/self/cgroup
0::/user.slice/user-1000.slice/session-3.scope

[vrothberg]

@giuseppe PTAL

[giuseppe]

could you please show the output for the following commands launched from the same environment where the podman exec fails:

$ cat /proc/self/cgroup
$ ls -al $(cat /proc/self/cgroup  | sed -e's|0::|/sys/fs/cgroup|')
$ ls -al $(dirname $(cat /proc/self/cgroup  | sed -e's|0::|/sys/fs/cgroup|'))
$ ls -al /sys/fs/cgroup/
$ ls -al /sys/fs/cgroup/user.slice/user-1000.slice/[email protected]
$ cat /proc/1/comm

Please also attach the output for podman --log-level debug exec test pwd

[m0rr1gan]

The first command was done and included at the bottom of the issue description.

I ended up reinstalling yesterday and after testing this morning, the problem has gone away and I am no longer able to reproduce.

[LastLightSith]

sometimes rebooting is not an option. I'm also facing this exact issue

[jackal-87]

I had the same problem on a fresh Fedora 35 Server system. It seems Fedora doesn't set some variables when you use su to change the user.
I had to add

export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"
export XDG_RUNTIME_DIR="/run/user/1000"

into the .bashrc and now it works. I also had to loginctl enable-linger. Maybe that helps on Arch too.

[aanno]

Same problem here (on fedora 36). .bashrc trick works for me. However, I do not need to set XDG_RUNTIME_DIR. Thank you for help!

[aanno]

Related fedora bug: https://bugzilla.redhat.com/show_bug.cgi?id=1622259

[rlenferink]

Same problem here (Fedora 37, Podman 4.4.2) and I spend quite some time on this before finding this ticket. I had a VNC server started which I connected to remotely and apparently it sets DBUS_SESSION_BUS_ADDRESS to another value. After setting it to the value @jackal-87 suggested using podman exec worked again for me.

[rlenferink@fedora container-test]$ echo $DBUS_SESSION_BUS_ADDRESS
unix:path=/tmp/dbus-1raSFVD6nV,guid=93d09a82ff7f94462c90aa29641b19a5
[rlenferink@fedora container-test]$ export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1026/bus"

Using systemd-run --scope --user podman exec ... did work though.