podman pod stats fails in rootless mode

[smuething]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

On a current system with Fedora 31, when running in rootless mode, podman pod stats <pod> fails because it cannot read the current memory usage of the individual containers.

Steps to reproduce the issue:

Run the following steps as a regular user:

1. podman pod create --name testpod

2. podman run -dit --pod testpod docker.io/library/fedora:31 /bin/bash

3. podman pod stats testpod

Describe the results you received:

The error message is (wrapped for readability):

Error: unable to obtain cgroup stats:
open /sys/fs/cgroup/user.slice/
user-libpod_pod_d7fed6e241437f78ef3153436b7863a8000678f6037fe6027f5b0b0b0f4921f3.slice/
user-1001.slice/[email protected]/user.slice/
libpod-068c28684eb595a706147fc9e8952db76e2f79d522937122307706105463e9b2.scope/
memory.current:
open /sys/fs/cgroup/user.slice/
user-libpod_pod_d7fed6e241437f78ef3153436b7863a8000678f6037fe6027f5b0b0b0f4921f3.slice/
user-1001.slice/[email protected]/user.slice/
libpod-068c28684eb595a706147fc9e8952db76e2f79d522937122307706105463e9b2.scope/
memory.current: no such file or directory

The given path to the cgroup file does indeed not exist, the correct path is

/sys/fs/cgroup/user.slice/user-1001.slice/[email protected]/user.slice/
user-libpod_pod_d7fed6e241437f78ef3153436b7863a8000678f6037fe6027f5b0b0b0f4921f3.slice/
libpod-068c28684eb595a706147fc9e8952db76e2f79d522937122307706105463e9b2.scope/
memory.current

The order of the path components seems mixed up: user-1001.slice/[email protected]/user.slice/ appears after the pod slice, not before it.

Describe the results you expected:

The normal output of podman pod stats.

Additional information you deem important (e.g. issue happens only occasionally):

The error does not occur when the pod is completely empty (as there are no running containers, not even an infra container).

Output of podman version:

Version:            1.6.2
RemoteAPI Version:  1
Go Version:         go1.13.1
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.13.1
  podman version: 1.6.2
host:
  BuildahVersion: 1.11.3
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.2-1.fc31.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.2, commit: 186a550ba0866ce799d74006dab97969a2107979'
  Distribution:
    distribution: fedora
    version: "31"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  MemFree: 133648470016
  MemTotal: 135088742400
  OCIRuntime:
    name: crun
    package: crun-0.10.6-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.10.6
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 4294963200
  SwapTotal: 4294963200
  arch: amd64
  cpus: 40
  eventlogger: journald
  hostname: slave01.parcomp
  kernel: 5.3.13-300.fc31.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.0-20.1.dev.gitbbd6f25.fc31.x86_64
    Version: |-
      slirp4netns version 0.4.0-beta.3+dev
      commit: bbd6f25c70d5db2a1cd3bfb0416a8db99a75ed7e
  uptime: 1h 8m 33.98s (Approximately 0.04 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - quay.io
store:
  ConfigFile: /home/gitlab-runner/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: vfs
  GraphOptions: {}
  GraphRoot: /home/gitlab-runner/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 8
  RunRoot: /run/user/1001
  VolumePath: /home/gitlab-runner/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.6.2-2.fc31.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

[rhatdan]

This works for me. Are you sure you are using crun for cgroups V2?

$ rpm -q podman
podman-1.6.2-2.fc31.x86_64
$ podman run -d fedora sleep 300
$ podman stats -l --no-stream
ID             NAME            CPU %   MEM USAGE / LIMIT   MEM %   NET IO    BLOCK IO   PIDS
4a33cc4e0291   pensive_knuth   --      2.302MB / 16.42GB   0.01%   -- / --   -- / --    1

[smuething]

podman stats works fine for me as well, the issue is with podman pod stats.