rootless podman doesn't work via sudo/su

[avikivity]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running with --pids-limit causes an error

Error: cannot set limits without cgroups: OCI runtime error

Note: this is not a dup of #6834 which is about cgroupsv1.

Steps to reproduce the issue:

1. podman run --pids-limit 3123 -it --rm docker.io/fedora:32 echo hey

Describe the results you received:

Error: cannot set limits without cgroups: OCI runtime error

Describe the results you expected:

hey

Additional information you deem important (e.g. issue happens only occasionally):

Happens on a particular machine. I did podman system migrate, then podman system reset, still happens.

Output of podman version:

podman version 2.0.4

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.19-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.19, commit: 5dce9767526ed27f177a8fa3f281889ad509fea7'
  cpus: 4
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: sif.cloudius-systems.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 300000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 300000
      size: 65536
  kernel: 5.7.11-200.fc32.x86_64
  linkmode: dynamic
  memFree: 23911043072
  memTotal: 33618317312
  ociRuntime:
    name: crun
    package: crun-0.14.1-1.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.14.1
      commit: 598ea5e192ca12d4f6378217d3ab1415efeddefa
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1001/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 153h 56m 54.03s (Approximately 6.38 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /jenkins/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /jenkins/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 4
  runRoot: /run/user/1001/containers
  volumePath: /jenkins/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 02:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.6
  OsArch: linux/amd64
  Version: 2.0.4

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.0.4-1.fc32.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical, fully updated Fedora 32, happens on this one host.

[mheon]

Can you confirm that podman info is from the affected machine - it's reading cgroups v2 there, which should work. We use the same code to print that bit of podman info and determine if resource limits can be applied, so it podman info says v2, I would expect this works.

[avikivity]

Here's a transcript. It was not edited or abridged in any way.

[jenkins@sif ~]$ podman run --pids-limit 3123 -it --rm docker.io/fedora:32 echo hey
Error: cannot set limits without cgroups: OCI runtime error
[jenkins@sif ~]$ podman info | grep cgroup
  cgroupVersion: v2

[rhatdan]

In crun I see

      if (resources->block_io
          || resources->network
          || resources->hugepage_limits_len
          || resources->memory
          || resources->pids
          || resources->cpu)
        return crun_make_error (err, 0, "cannot set limits without cgroups");

I think your system does not support the resource-pids cgroup.

cat /proc/self/cgroup
0::/user.slice/user-3267.slice/[email protected]/apps.slice/apps-org.gnome.Terminal.slice/vte-spawn-b8623b81-9d60-4add-83c6-1babe95d6bc0.scope

Then take this path and append it to /sys/fs/cgroup

ls /sys/fs/cgroup/user.slice/user-3267.slice/[email protected]/apps.slice/apps-org.gnome.Terminal.slice/vte-spawn-b8623b81-9d60-4add-83c6-1babe95d6bc0.scope
cgroup.controllers      cpu.pressure         memory.oom.group
cgroup.events           cpu.stat             memory.pressure
cgroup.freeze           io.pressure          memory.stat
cgroup.max.depth        memory.current       memory.swap.current
cgroup.max.descendants  memory.events        memory.swap.events
cgroup.procs            memory.events.local  memory.swap.max
cgroup.stat             memory.high          pids.current
cgroup.subtree_control  memory.low           pids.events
cgroup.threads          memory.max           pids.max
cgroup.type             memory.min

@giuseppe WDYT?

[giuseppe]

it looks like it failed to create the cgroup. My guess is that there is no systemd session for the user, so we fallback to cgroupfs. Even if we are running on cgroup v2, we are not able to delegate a subtree for the user.

Is it running with the user jenkins? Does it have a systemd --user daemon running?

[avikivity]

Indeed there is no systemd user daemon status. I don't understand why, since jenkins logins via ssh like an ordinary user.

I suggest improving the error message here.

[avikivity]

And how is a service supposed to work with podman? You can't expect it to be logged in.

[giuseppe]

And how is a service supposed to work with podman? You can't expect it to be logged in.

that is an on-going effort for now. But at the moment we require lingering mode to be configured for the user. I think there is still the issue of ensuring the user session starts before the service, or alternatively install the .service as part of the user configuration (under ~/.config/systemd/user).

[avikivity]

It doesn't work for me even with lingering enabled. Is there some documentation on this?

For jenkins agents, Jenkins works by connecting to them with ssh, and without a pty. Is there any way to make this work?

[giuseppe]

I've not tried with jenkins, so I am not sure what we could be miss to get it working.

Does systemd-run work in the same environment? If systemd-run --scope --user -p TasksMax=3123 echo foo works and Podman+crun don't then we are doing something wrong.

[avikivity]

$ systemd-run --scope --user -p TasksMax=3123 echo foo
Failed to create bus connection: No such file or directory

[avikivity]

(this is with sudo -s ... su - jenkins, which I think is similar to jenkins login-without-tty)

[avikivity]

Is there no workaround?

[rhatdan]

@giuseppe @vrothberg Ideas?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[avikivity]

A workaround is machinectl shell --uid.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@avikivity Is this still an issue?

[avikivity]

I've not needed to do this kind of maintenance recently, so can't confirm or deny. Is it supposed to be fixed in some release? If so, mention it here and close, and I'll reopen if I see it happen in a version where it's supposed to be fixed.

[avikivity]

I just hit this in a new Ubuntu 20.10 install.

[avikivity]

podman version 2.0.6

[avikivity]

Here is the full error in all its glory:

Error: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:415: setting cgroup config for procHooks process caused \\\"cannot set pids limit: container could not join or create cgroup\\\"\"": OCI runtime error

[avikivity]

It turned out that this was another manifestation of --pids-limit breakage with cgroupsv1, and my script misdetected it.

[rhatdan]

I only care about podman 3.*, basically what is in the main branch.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

I believe this is fixed in the current release.