podman stats not working, returns cgroup deleted with systemd enabled container on ubuntu 20.04

[towe75]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I can not get stats from a ubuntu 20.04 container on a ubuntu 20.04 host with cgroup v1 and systemd in the container using podman 3.1.2.

Same setup works on a fedora host with cgroup v2. Maybe it is related to #9252 ?

Steps to reproduce the issue:

1. Produce a bare bones systemd enabled image:

c=$(buildah from ubuntu:focal)
buildah run $c apt update
buildah run $c apt install --yes init
buildah commit $c mytest:latest

2. Run the container with systemd as entry point

podman run -d --rm --entrypoint=/sbin/init mytest:latest

3. Try to get stats

podman stats

Error: unable to load cgroup at /machine.slice/libpod-124c8be7e6f27522546aed287e2ca7e2d8d24a206570abc12b557d23c7f99506.scope/init.scope: cgroup deleted

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.1.2
API Version:  3.1.2
Go Version:   go1.15.2
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

No

[mheon]

Running as root, or rootless?

[towe75]

Sorry, i forgot to mention it: running as root. Containers without entrypoint=/sbin/init are fine as well.

[mheon]

Alright. Probably systemd mode, then - we do some things with cgroups on v1 to make systemd work in a container. I wonder if this has been broken for years, but no one has run podman stats against a systemd container?

[towe75]

Quite possible. I can confirm that the same setup but with cgroup v2 (via kernel parameter) works like expected.

[giuseppe]

seems to work fine on Fedora 24 with cgroup v1.

Could you please attach the output for strace -v -s 5000 -f -Z podman stats $CTR_ID --no-stream ?

[towe75]

@giuseppe sure, here is the tail. Ping me if you need the full trace:

[pid 20092] epoll_ctl(4, EPOLL_CTL_ADD, 11, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=3976157976, u64=140007025105688}}) = -1 EPERM (Operation not permitted)
[pid 20092] epoll_ctl(4, EPOLL_CTL_DEL, 11, 0xc0005a96c4) = -1 EPERM (Operation not permitted)
[pid 20091] futex(0x55c5c3dd4bf8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=4, tv_nsec=578372743}) = -1 EAGAIN (Resource temporarily unavailable)
[pid 20092] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=20090, si_uid=0} ---
[pid 20096] futex(0x55c5c3dd4af8, FUTEX_WAIT_PRIVATE, 2, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
[pid 20096] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=20090, si_uid=0} ---
[pid 20096] futex(0x55c5c3dd4af8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 20092] futex(0x55c5c3dd4c10, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 ETIMEDOUT (Connection timed out)
[pid 20096] futex(0x55c5c3dd4c10, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 EAGAIN (Resource temporarily unavailable)
[pid 20096] futex(0x55c5c3dd4c10, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 EAGAIN (Resource temporarily unavailable)
[pid 20096] futex(0x55c5c3dd4c10, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 ETIMEDOUT (Connection timed out)
[pid 20092] epoll_ctl(4, EPOLL_CTL_ADD, 11, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=3976157976, u64=140007025105688}}) = -1 EPERM (Operation not permitted)
[pid 20092] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=20090, si_uid=0} ---
[pid 20092] epoll_ctl(4, EPOLL_CTL_DEL, 11, 0xc0005c161c) = -1 EPERM (Operation not permitted)
[pid 20092] epoll_ctl(4, EPOLL_CTL_ADD, 12, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=3976157976, u64=140007025105688}}) = -1 EPERM (Operation not permitted)
[pid 20092] epoll_ctl(4, EPOLL_CTL_DEL, 12, 0xc0005c160c) = -1 EPERM (Operation not permitted)
[pid 20092] newfstatat(AT_FDCWD, "/run/libpod/exits/2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24", 0xc00057de48, 0) = -1 ENOENT (No such file or directory)
[pid 20092] epoll_ctl(4, EPOLL_CTL_ADD, 11, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=3976157976, u64=140007025105688}}) = -1 EPERM (Operation not permitted)
[pid 20092] epoll_ctl(4, EPOLL_CTL_DEL, 11, 0xc0005c17ec) = -1 EPERM (Operation not permitted)
[pid 20092] epoll_ctl(4, EPOLL_CTL_ADD, 11, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=3976157976, u64=140007025105688}}) = -1 EPERM (Operation not permitted)
[pid 20092] epoll_ctl(4, EPOLL_CTL_DEL, 11, 0xc0005c1714) = -1 EPERM (Operation not permitted)
[pid 20092] newfstatat(AT_FDCWD, "/sys/fs/cgroup/blkio/machine.slice/libpod-2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24.scope/init.scope", 0xc00069b6f8, 0) = -1 ENOENT (No such file or directory)
[pid 20092] newfstatat(AT_FDCWD, "/sys/fs/cgroup/cpu/machine.slice/libpod-2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24.scope/init.scope", 0xc00069b7c8, 0) = -1 ENOENT (No such file or directory)
[pid 20092] newfstatat(AT_FDCWD, "/sys/fs/cgroup/cpuset/machine.slice/libpod-2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24.scope/init.scope", 0xc00069b898, 0) = -1 ENOENT (No such file or directory)
[pid 20092] newfstatat(AT_FDCWD, "/sys/fs/cgroup/memory/machine.slice/libpod-2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24.scope/init.scope", 0xc00069b968, 0) = -1 ENOENT (No such file or directory)
[pid 20092] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=20090, si_uid=0} ---
[pid 20092] rt_sigreturn({mask=[]})     = -1 ENOENT (No such file or directory)
[pid 20092] newfstatat(AT_FDCWD, "/sys/fs/cgroup/pids/machine.slice/libpod-2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24.scope/init.scope", 0xc00069ba38, 0) = -1 ENOENT (No such file or directory)
Error: unable to load cgroup at /machine.slice/libpod-2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24.scope/init.scope: cgroup deleted
[pid 20098] +++ exited with 125 +++
[pid 20097] +++ exited with 125 +++
[pid 20096] +++ exited with 125 +++
[pid 20095] +++ exited with 125 +++
[pid 20094] +++ exited with 125 +++
[pid 20093] +++ exited with 125 +++
[pid 20092] +++ exited with 125 +++
[pid 20091] +++ exited with 125 +++
+++ exited with 125 +++

[giuseppe]

thanks!

Could you also give me the output for: cat /proc/$CONTAINER_PROCESS_PID/cgroup and cat /proc/self/mountinfo?

[towe75]

12:pids:/machine.slice/libpod-2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24.scope
11:memory:/machine.slice/libpod-2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24.scope
10:blkio:/machine.slice/libpod-2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24.scope
9:devices:/machine.slice/libpod-2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24.scope
8:freezer:/machine.slice/libpod-2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24.scope
7:rdma:/machine.slice/libpod-2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24.scope
6:hugetlb:/machine.slice/libpod-2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24.scope
5:cpu,cpuacct:/machine.slice/libpod-2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24.scope
4:net_cls,net_prio:/machine.slice/libpod-2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24.scope
3:cpuset:/machine.slice/libpod-2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24.scope
2:perf_event:/machine.slice/libpod-2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24.scope
1:name=systemd:/machine.slice/libpod-2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24.scope/init.scope
0::/machine.slice/libpod-2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24.scope

I assume you want the /proc/self/mountinfo for the host? It might be a bit confusing:

24 33 0:22 / /sys rw,nosuid,nodev,noexec,relatime shared:11 - sysfs sysfs rw
25 33 0:5 / /proc rw,nosuid,nodev,noexec,relatime shared:18 - proc proc rw
26 33 0:6 / /dev rw,nosuid,noexec,relatime shared:6 - devtmpfs udev rw,size=976276k,nr_inodes=244069,mode=755
27 26 0:23 / /dev/pts rw,nosuid,noexec,relatime shared:7 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
28 33 0:24 / /run rw,nosuid,nodev,noexec,relatime shared:9 - tmpfs tmpfs rw,size=203528k,mode=755
30 33 8:1 / /SYSTEM rw,noatime shared:3 - ext4 /dev/sda1 rw
31 33 7:0 / /rofs ro,noatime shared:2 - squashfs /dev/loop0 ro
33 1 0:26 / / rw,noatime shared:1 - aufs aufs rw,si=f6c6059236f73819
34 33 8:1 /persistent /persistent rw,noatime shared:4 - ext4 /dev/sda1 rw
35 33 8:1 /persistent/var/lib/lxc /var/lib/lxc rw,noatime shared:5 - ext4 /dev/sda1 rw
36 24 0:7 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:12 - securityfs securityfs rw
37 26 0:27 / /dev/shm rw,nosuid,nodev shared:8 - tmpfs tmpfs rw
38 28 0:28 / /run/lock rw,nosuid,nodev,noexec,relatime shared:10 - tmpfs tmpfs rw,size=5120k
39 24 0:29 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:13 - tmpfs tmpfs ro,mode=755
40 39 0:30 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime shared:14 - cgroup2 cgroup2 rw
41 39 0:31 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:15 - cgroup cgroup rw,xattr,name=systemd
42 24 0:32 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:16 - pstore pstore rw
43 24 0:33 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:17 - bpf none rw,mode=700
44 39 0:34 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:19 - cgroup cgroup rw,perf_event
45 39 0:35 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:20 - cgroup cgroup rw,cpuset
46 39 0:36 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:21 - cgroup cgroup rw,net_cls,net_prio
47 39 0:37 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:22 - cgroup cgroup rw,cpu,cpuacct
48 39 0:38 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:23 - cgroup cgroup rw,hugetlb
49 39 0:39 / /sys/fs/cgroup/rdma rw,nosuid,nodev,noexec,relatime shared:24 - cgroup cgroup rw,rdma
50 39 0:40 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:25 - cgroup cgroup rw,freezer
51 39 0:41 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:26 - cgroup cgroup rw,devices
52 39 0:42 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:27 - cgroup cgroup rw,blkio
53 39 0:43 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:28 - cgroup cgroup rw,memory
54 39 0:44 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime shared:29 - cgroup cgroup rw,pids
55 25 0:45 / /proc/sys/fs/binfmt_misc rw,relatime shared:30 - autofs systemd-1 rw,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15093
56 26 0:20 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:31 - mqueue mqueue rw
57 24 0:8 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:32 - debugfs debugfs rw
58 26 0:46 / /dev/hugepages rw,relatime shared:33 - hugetlbfs hugetlbfs rw,pagesize=2M
59 24 0:12 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:34 - tracefs tracefs rw
60 28 0:47 / /run/rpc_pipefs rw,relatime shared:35 - rpc_pipefs sunrpc rw
61 24 0:48 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:36 - fusectl fusectl rw
62 24 0:21 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:37 - configfs configfs rw
144 33 0:49 / /tmp rw,nosuid,nodev,relatime shared:78 - tmpfs tmpfs rw
516 33 0:55 / /var/lib/lxcfs rw,nosuid,nodev,relatime shared:282 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
141 33 8:1 /persistent/localstorage /shared rw,noatime shared:4 - ext4 /dev/sda1 rw
877 34 0:113 / /persistent/var/lib/nomad/alloc/dd3df429-91d8-9cbb-fbb0-d84281b35c3f/pascomd/secrets rw,noexec,relatime shared:302 - tmpfs tmpfs rw,size=1024k
888 34 8:1 /persistent/containers/overlay /persistent/containers/overlay rw,noatime - ext4 /dev/sda1 rw
899 28 0:24 /netns /run/netns rw,nosuid,nodev,noexec,relatime shared:9 - tmpfs tmpfs rw,size=203528k,mode=755
910 899 0:4 net:[4026533078] /run/netns/cni-78bf5553-af26-e2d7-eff1-3d8aa06153b3 rw shared:318 - nsfs nsfs rw
911 28 0:4 net:[4026533078] /run/netns/cni-78bf5553-af26-e2d7-eff1-3d8aa06153b3 rw shared:318 - nsfs nsfs rw
932 34 0:114 / /persistent/containers/overlay-containers/8718315258dc99a531c1c2dbfbf178412787935b08b54000870685db731516ba/userdata/shm rw,nosuid,nodev,noexec,relatime shared:329 - tmpfs shm rw,size=64000k
948 888 0:115 / /persistent/containers/overlay/855a82c08c57b799073c83eee3c87b4e417efd1432497995e30ab517249969bb/merged rw,relatime - overlay overlay rw,lowerdir=/persistent/containers/overlay/l/X565VO6ALFFNM5Y4SUTTPU4QA6:/persistent/containers/overlay/l/UKS47TIPAURIXVOLSVYZ2GC7VF:/persistent/containers/overlay/l/7FXI64PODURHAHDQJ7XMUFWIHK:/persistent/containers/overlay/l/SAHVN6S2AYJH72M5ANYQI6BLP2,upperdir=/persistent/containers/overlay/855a82c08c57b799073c83eee3c87b4e417efd1432497995e30ab517249969bb/diff,workdir=/persistent/containers/overlay/855a82c08c57b799073c83eee3c87b4e417efd1432497995e30ab517249969bb/work,xino=off
333 28 0:56 / /run/user/1000 rw,nosuid,nodev,relatime shared:183 - tmpfs tmpfs rw,size=203524k,mode=700,uid=1000,gid=1000
952 34 0:126 / /persistent/containers/overlay-containers/2521e906f0a0c758c3a7dc4fd68c197e32e96abb4b36807d0b0ede6d014a7c24/userdata/shm rw,nosuid,nodev,noexec,relatime shared:336 - tmpfs shm rw,size=64000k
963 899 0:4 net:[4026532791] /run/netns/cni-3f7a716c-c9a3-dc3b-4f21-4b8af69f08a6 rw shared:342 - nsfs nsfs rw
964 28 0:4 net:[4026532791] /run/netns/cni-3f7a716c-c9a3-dc3b-4f21-4b8af69f08a6 rw shared:342 - nsfs nsfs rw
990 888 0:127 / /persistent/containers/overlay/d57f4b5556334456f782f139242cb63873ebc56b167e41999eb1c7652831afa3/merged rw,relatime - overlay overlay rw,lowerdir=/persistent/containers/overlay/l/IV6EI4JFT4BOPRBRFE6CJ4NSO7:/persistent/containers/overlay/l/UKS47TIPAURIXVOLSVYZ2GC7VF:/persistent/containers/overlay/l/7FXI64PODURHAHDQJ7XMUFWIHK:/persistent/containers/overlay/l/SAHVN6S2AYJH72M5ANYQI6BLP2,upperdir=/persistent/containers/overlay/d57f4b5556334456f782f139242cb63873ebc56b167e41999eb1c7652831afa3/diff,workdir=/persistent/containers/overlay/d57f4b5556334456f782f139242cb63873ebc56b167e41999eb1c7652831afa3/work,xino=off

[giuseppe]

I see what happens. systemd moves itself to a different cgroup under the named hierarchy and that might confuse the cgroup detection code.

[giuseppe]

opened a PR: #10609