podman run failure on cgroupsv1 rootless related to --pids-limit #6834

goochjj · 2020-07-01T14:13:38Z

/kind bug

Description

pids-limit is getting set to 2048 as a default, which is fine unless you're rootless on cgroups v1, at which point you can't create anything without explicitly setting --pids-limit 0.

Steps to reproduce the issue:

(focal)mrwizard@FocalCG1Dev:~/src/podman
$ podman run --rm -it alpine sh
Error: cannot set limits without cgroups: OCI runtime error

Describe the results you expected:

(focal)mrwizard@FocalCG1Dev:~/src/podman
$ podman run --rm -it --pids-limit 0 alpine sh
/ #

Output of podman version:

Version:      2.1.0-dev
API Version:  1
Go Version:   go1.13.8
Git Commit:   719557bb5055ecc2a20c6c1cd967230ebe36758d-dirty
Built:        Wed Jul  1 10:03:12 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.18, commit: '
  cpus: 8
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: file
  hostname: FocalCG1Dev
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.4.0-39-generic
  linkmode: dynamic
  memFree: 11687772160
  memTotal: 16790687744
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: 11b7a1f2baa6bbb762c470ff0457a0714274c141
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.0
      commit: unknown
      libslirp: 4.2.0
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 47h 43m 3.53s (Approximately 1.96 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/mrwizard/.config/containers/storage.conf
  containerStore:
    number: 8
    paused: 0
    running: 2
    stopped: 6
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.9.0
        fuse-overlayfs: version 0.7.6
        FUSE library version 3.9.0
        using FUSE kernel interface version 7.31
  graphRoot: /home/mrwizard/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 16
  runRoot: /run/user/1000/containers
  volumePath: /home/mrwizard/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 1593612192
  BuiltTime: Wed Jul  1 10:03:12 2020
  GitCommit: 719557bb5055ecc2a20c6c1cd967230ebe36758d-dirty
  GoVersion: go1.13.8
  OsArch: linux/amd64
  Version: 2.1.0-dev

Package info (e.g. output of rpm -q podman or apt list podman):

origin/master

The text was updated successfully, but these errors were encountered:

goochjj · 2020-07-01T14:14:48Z

This chunk of code seems to have handled this before:
https://github.com/containers/libpod/blob/d8718fd62e87d0a364a4092c6333d90e26ce9c8e/pkg/spec/spec.go#L298-L318

Per @mheon

This is definitely a bug. Is this 2.0? pkg/spec is deprecated, we've moved to pkg/specgen/generate - so the offending code likely lives there.

mheon · 2020-07-01T14:16:33Z

I'll take a look after lunch, see if I can't chase this down.

goochjj · 2020-07-01T14:19:26Z

This seems to check at the kernel level
https://github.com/containers/libpod/blob/e84695213e35c22ba085e3831cbd025cd55a4c84/pkg/specgen/generate/validate.go#L71-L80

Looks like it just sees if the pids controller is mounted (in sysInfo) not if we actually have the ability to write to the pids controller for the cgroup we're going to be a part of. I don't see any rootless checks in this either

goochjj · 2020-07-01T14:21:16Z

DefaultPidsLimit was introduced in containers/common

markstos · 2020-07-01T17:36:38Z

Is this related to #6734?

mheon · 2020-07-01T17:43:45Z

I don't think so. I think this is specific to us trying to mimic the system PID limit.

mheon · 2020-07-01T17:57:50Z

Alright, this one is honestly kind of bizarre. It worked fine when I built directly from master. In order to confirm code was running, I added a single logrus.Errorf() and now it started to reproduce.

goochjj · 2020-07-01T18:08:15Z

I only have 1 cgroups v1 host setup to run rootless containers so it's hard for me to narrow down what's causing it...

rhatdan · 2020-07-01T18:21:26Z

@goochjj
podman run --help | grep pids
--pids-limit int Tune container pids limit (set 0 for unlimited, -1 for server defaults) (default 2048)

Show, if this shows as above (default 2048) then this is the issue.

On a cgroups V1 system this should show no default.

mheon · 2020-07-01T18:23:27Z

@rhatdan It's set to 0, but having it set to anything is the problem. crun seems to discard any OCI spec that tries to set resource limits, even empty limits.

rhatdan · 2020-07-01T18:23:42Z

This is the code that sets the default

// PidsLimit returns the default maximum number of pids to use in containers

func (c *Config) PidsLimit() int64 {
	if unshare.IsRootless() {
		if c.Engine.CgroupManager == SystemdCgroupsManager {
			cgroup2, _ := cgroupv2.Enabled()
			if cgroup2 {
				return c.Containers.PidsLimit
			}
			return 0
		}
	}
	return sysinfo.GetDefaultPidsLimit()
}

rhatdan · 2020-07-01T18:25:09Z

If you inspect the container does it show ulimits?

goochjj · 2020-07-01T18:26:45Z

(focal)mrwizard@FocalCG1Dev:~/src/podman
$ podman run --help | grep pids
      --pids-limit int                           Tune container pids limit (set 0 for unlimited, -1 for server defaults)

$ podman run --rm -it alpine sh
Error: cannot set limits without cgroups: OCI runtime error

So look at this, my crun-debug grabs the config.json and dumps it in my XDG_RUNTIME_DIR:

(focal)mrwizard@FocalCG1Dev:~/src/podman
$ podman --runtime /usr/local/bin/crun-debug run --rm -it alpine sh
Error: cannot set limits without cgroups: OCI runtime error

(focal)mrwizard@FocalCG1Dev:~/src/podman
$ jq .linux.resources /run/user/1000/crun/config.json
{
  "pids": {
    "limit": 0
  }
}

vs:

(focal)mrwizard@FocalCG1Dev:~/src/podman
$ podman --runtime /usr/local/bin/crun-debug run --rm -it --pids-limit 0 alpine sh
/ #
(focal)mrwizard@FocalCG1Dev:~/src/podman
$ jq .linux.resources /run/user/1000/crun/config.json
null

I think crun only looks for a resources/pids tree, not that it's actually >0

goochjj · 2020-07-01T18:29:30Z

Yeah, crun just checks that the resources->pids structure is specified (non null), not that resources->pids->limit >0 or resources->pids->limit_present

goochjj · 2020-07-01T18:31:04Z

https://github.com/containers/crun/blob/d38b8c28fc50a14978a27fa6afc69a55bfdd2c11/src/libcrun/cgroup.c#L2765-L2771

mheon · 2020-07-01T18:37:00Z

#6837 should fix it.

However, in some cases (unset limits), we can completely remove the limit and avoid errors. This works around a bug where the Podman frontend is setting a Pids limit of 0 on some rootless systems. For now, this is only implemented for the PID limit. It can easily be extended to other resource limits, but it is a fair bit of code to do so, so I leave that exercise to someone else. Fixes containers#6834 Signed-off-by: Matthew Heon <[email protected]>

rhatdan · 2020-07-02T13:11:10Z

I prefer my fix.
#6842

github-actions · 2020-08-02T00:13:19Z

A friendly reminder that this issue had no activity for 30 days.

zhangguanzhang · 2020-08-02T05:18:33Z

@goochjj If the latest version does not fix this issue, could reopen it.

avikivity · 2021-01-17T10:09:51Z

This started appearing for me on a setup that used to work. Problem started after a dnf update. podman system migrate does not help.

$ podman run -it --rm --pids-limit 2000 docker.io/fedora:33
Error: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: cannot set pids limit: container could not join or create cgroup: OCI runtime error

  arch: arm64
  buildahVersion: 1.15.1
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.20-2.module_el8.3.0+475+c50ce30b.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.20, commit: c579ba7260c95670f55ad33c5a3ce9660f193a43'
  cpus: 96
  distribution:
    distribution: '"centos"'
    version: "8"
  eventLogger: file
  hostname: armada.scylladb.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 2000000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 2000000
      size: 65536
  kernel: 4.18.0-193.6.3.el8_2.aarch64
  linkmode: dynamic
  memFree: 128958726144
  memTotal: 136593145856
  ociRuntime:
    name: runc
    package: runc-1.0.0-68.rc92.module_el8.3.0+475+c50ce30b.aarch64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.2-dev'
  os: linux
  remoteSocket:
    path: /run/user/1002/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-2.module_el8.3.0+475+c50ce30b.aarch64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
  swapFree: 0
  swapTotal: 0
  uptime: 91h 38m 40.86s (Approximately 3.79 days)
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/avi/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-3.module_el8.3.0+507+aa0970ae.aarch64
      Version: |-
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /home/avi/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 15
  runRoot: /run/user/1002
  volumePath: /home/avi/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 1600970443
  BuiltTime: Thu Sep 24 18:00:43 2020
  GitCommit: ""
  GoVersion: go1.14.7
  OsArch: linux/arm64
  Version: 2.0.5

avikivity · 2021-01-17T10:16:44Z

From dnf.log:

2021-01-13T14:06:50Z DEBUG ---> Package podman.aarch64 1.6.4-10.module_el8.2.0+305+5e198a41 will be upgraded
2021-01-13T14:06:50Z DEBUG ---> Package podman.aarch64 2.0.5-5.module_el8.3.0+512+b3b58dca will be an upgrade

rhatdan · 2021-01-18T12:52:30Z

Could you verify that libpod.conf and containers.conf are setup correctly. This is an old bug that has been fixed for many months.

avikivity · 2021-01-18T13:05:59Z

containers.conf is vanilla:

$ locate containers.conf
/usr/share/containers/containers.conf
/usr/share/man/man5/containers.conf.5.gz
$ rpm -qf /usr/share/containers/containers.conf
containers-common-1.1.1-3.module_el8.3.0+475+c50ce30b.aarch64
$ rpm -q --verify containers-common-1.1.1-3.module_el8.3.0+475+c50ce30b.aarch64

There is no libpod.conf on this system.

rhatdan · 2021-01-18T20:16:28Z

Ok, I don't think we currently support pids limit on a cgroupv1 system. @giuseppe @mheon Do you remember?

giuseppe · 2021-01-19T07:48:34Z

no, we don't support any kind of limit on cgroup v1 with rootless

avikivity · 2021-01-19T08:23:02Z

Well, it worked with 1.6.4 and stopped working with 2.0.5.

Can you suggest a workaround? That system is useless now.

giuseppe · 2021-01-19T09:35:53Z

do you have any file under ~/.config/containers/ (containers.conf and libpod.conf) that could be overriding the pids limit?

A possible workaround is to override pids_limit in ~/.config/containers/containers.conf or in /etc/containers/containers.conf

avikivity · 2021-01-19T19:56:08Z

do you have any file under ~/.config/containers/ (containers.conf and libpod.conf) that could be overriding the pids limit?

No.

A possible workaround is to override pids_limit in ~/.config/containers/containers.conf or in /etc/containers/containers.conf

I get the same error. Or do you mean drop --pids-limit and rely on the value from config?

That's unfortunate for me because it means I have to detect versions and have my scripts behave differently depending on what they find (and require the user to adjust config, but for this machine the user is me, so I don't mind).

rhatdan · 2021-01-20T00:38:49Z

In podman 1.6 perhaps we were ignoring the flag. But the pids limit was not being set for rootless containers.

avikivity · 2021-01-20T08:27:51Z

So I'll have to add a version check. From which version can I rely on --pids-limit to work?

mheon · 2021-01-20T14:24:13Z

I suspect 2.0.0 and up - there was an extensive rewrite of container creation to enable the new HTTP API, and this looks like a consequence of it.

avikivity · 2021-01-20T14:25:44Z

This failure is with 2.0.5.

mheon · 2021-01-20T14:48:46Z

I'm 99% sure the new behavior here is correct - cgroups control is required for the PIDs limit to be set, and cgroups v1 + rootless does not have sufficient access to cgroups to set said limit. I strongly suspect v1.9 and lower just ignored the limit, and now we've begun passing it down to the OCI runtime which errors on the limit being set.

If you require a rootless PIDs limit be set, a cgroups v2 system would be a requirement.

@giuseppe Can you confirm?

rhatdan · 2021-01-22T09:28:24Z

Just to make sure this is not a container.conf issue, could you try to run the podman command with

CONTAINERS_CONF=/dev/null podman ...

avikivity · 2021-01-22T11:27:42Z

$ CONTAINERS_CONF=/dev/null podman run -it --rm --pids-limit 2000 docker.io/fedora:33
Error: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: cannot set pids limit: container could not join or create cgroup: OCI runtime error

rhatdan · 2021-01-22T11:46:59Z

Ok, nevermind, I missed the fact that you are specifying the --pids-limit. Which as has been pointed out, will not work on cgroups v2 in rootless mode.

openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Jul 1, 2020

goochjj mentioned this issue Jul 1, 2020

pursuing conventional systemd+podman interaction #6400

Closed

mheon self-assigned this Jul 1, 2020

goochjj mentioned this issue Jul 1, 2020

Implement --sdnotify cmdline option to control sd-notify behavior #6693

Merged

mheon mentioned this issue Jul 1, 2020

In rootless + cgroupsv1, resource limits are an error #6837

Closed

github-actions bot added the stale-issue label Aug 2, 2020

zhangguanzhang closed this as completed Aug 2, 2020

avikivity mentioned this issue Aug 23, 2020

rootless podman doesn't work via sudo/su #7417

Closed

avikivity mentioned this issue Jan 20, 2021

dbuild fails with an error on older kernels (without cgroupsv2) scylladb/scylladb#7938

Closed

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023

podman run failure on cgroupsv1 rootless related to --pids-limit #6834

podman run failure on cgroupsv1 rootless related to --pids-limit #6834

Comments

goochjj commented Jul 1, 2020

goochjj commented Jul 1, 2020

mheon commented Jul 1, 2020

goochjj commented Jul 1, 2020

goochjj commented Jul 1, 2020

markstos commented Jul 1, 2020

mheon commented Jul 1, 2020

mheon commented Jul 1, 2020

goochjj commented Jul 1, 2020

rhatdan commented Jul 1, 2020

mheon commented Jul 1, 2020

rhatdan commented Jul 1, 2020 • edited Loading

rhatdan commented Jul 1, 2020

goochjj commented Jul 1, 2020

goochjj commented Jul 1, 2020

goochjj commented Jul 1, 2020

mheon commented Jul 1, 2020

rhatdan commented Jul 2, 2020

github-actions bot commented Aug 2, 2020

zhangguanzhang commented Aug 2, 2020

avikivity commented Jan 17, 2021

avikivity commented Jan 17, 2021

rhatdan commented Jan 18, 2021

avikivity commented Jan 18, 2021

rhatdan commented Jan 18, 2021

giuseppe commented Jan 19, 2021

avikivity commented Jan 19, 2021

giuseppe commented Jan 19, 2021

avikivity commented Jan 19, 2021

rhatdan commented Jan 20, 2021

avikivity commented Jan 20, 2021

mheon commented Jan 20, 2021

avikivity commented Jan 20, 2021

mheon commented Jan 20, 2021

rhatdan commented Jan 22, 2021

avikivity commented Jan 22, 2021

rhatdan commented Jan 22, 2021

rhatdan commented Jul 1, 2020 •

edited

Loading