Not able to create more than 100 pods on each kind node with podman

[numansiddique]

What happened:

After deploying kind using podman provider, creating more than 100 pods on each kind worker node fails.
Below error messages are seen in kubectl

ul 14 21:56:42 ovn-worker kubelet[44001]: E0714 21:56:42.556516 44001 kuberuntime_manager.go:738] "killPodWithSyncResult failed" err="failed to "KillPodSandbox" for "03b80ada-b5e7-4cae-8ac7-f892bddcb5d0" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"70ad443d7b3e6217ebd3d5a93f1a8b8871dda20ffc3261afa83bc89e6db3ea79\": plugin type=\"ovn-k8s-cni-overlay\" name=\"ovn-kubernetes\" failed (delete): netplugin failed: \"runtime: failed to create new OS thread (have 9 already; errno=11)\\nruntime: may need to increase max user processes (ulimit -u)\\nfatal error: newosproc\\nruntime: failed to create new OS thread (have 10 already; errno=11)\\nruntime: may need to increase max user processes (ulimit -u)\\nfatal error: newosproc\\n\\nruntime stack:\\nruntime.throw({0x1867329?, 0xc0000a3e38?})\\n\\t/usr/local/go/src/runtime/panic.go:992
....

The issue gets resolved if the option - "--pids-limit=-1" is passed when starting the podman kind container - here - https://github.com/kubernetes-sigs/kind/blob/main/pkg/cluster/internal/providers/podman/provision.go#L192

I think its good to address this limitation as we use kind for scale testing to create 250 pods per node.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• kind version: (use kind version):
• Kubernetes version: (use kubectl version):
• Docker version: (use docker info):
• OS (e.g. from /etc/os-release):

[numansiddique]

@aojea FYI

[aojea]

ref containers/podman#11782

Thanks Numan

[aojea]

@numansiddique can you check where podman is getting the cgroup limit?

I prefer to not tweak defaults if possible, this is the values I've got

$ cat /sys/fs/cgroup/pids/system.slice/pids.max
 max
$ cat /sys/fs/cgroup/pids/system.slice/docker.service/pids.max
16488
$ podman run --rm -ti fedora:34  cat /sys/fs/cgroup/pids/pids.max
max
$ docker run --rm -ti fedora:34  cat /sys/fs/cgroup/pids/pids.max                                                                                                                                                                                                                                                                                             
16488

[numansiddique]

I tested on fedora 36 with cgroupsv2 enabled

# cat /sys/fs/cgroup/system.slice/pids.max
max

I started a podman container

# podman run -it registry.fedoraproject.org/fedora:36 bash
root@8044416466b9 /]# cat /sys/fs/cgroup/pids.max   (Inside the container)
2048

On the host

# cat /sys/fs/cgroup/machine.slice/libpod-8044416466b9b13d8655b06116246bf7bc424f0ff880d28afbadb7e539eb477a.scope/container/pids.max
2048

If I pass --pids-limit=-1, then pids.max

# podman run -it --pids-limit=-1 registry.fedoraproject.org/fedora:36 bash
[root@036f8be5fa50 /]# cat /sys/fs/cgroup/pids.max
max

[aojea]

@mheon can you help us here?
where this pid limit of 2048 in podman is coming from? and why I don't see it in my system?
podman version and/or cgroupsv2?

[numansiddique]

I tested with disablin cgroups v2.

With cgroups v1

# podman run -it registry.fedoraproject.org/fedora:36 bash
[root@d010e0d1b5e9 /]# cat /sys/fs/cgroup/pids/pids.max 
2048

And with pids-limit=-1

# podman run -it --pids-limit=-1 registry.fedoraproject.org/fedora:36 bash
[root@c10da30c3c45 /]# cat /sys/fs/cgroup/pids/pids.max
38055

podman version is

podman version
Client:       Podman Engine
Version:      4.1.1
API Version:  4.1.1
Go Version:   go1.18.3
Built:        Wed Jun 22 11:17:44 2022
OS/Arch:      linux/amd64
# rpm -qa | grep podman
podman-4.1.1-2.fc36.x86_64
podman-gvproxy-4.1.1-2.fc36.x86_64
podman-plugins-4.1.1-2.fc36.x86_64
podman-docker-4.1.1-2.fc36.noarch

[mheon]

The 2048 limit is a default for security reasons, IIRC. Th default can be overridden by pids_limit in containers.conf (or by setting it from the command line).

[mheon]

It's a compiled-in default, so as for why you're not seeing it, I'm not sure. Maybe a non-default containers.conf?

[aojea]

It's a compiled-in default, so as for why you're not seeing it, I'm not sure. Maybe a non-default containers.conf?

most probably

Th default can be overridden by pids_limit in containers.conf (or by setting it from the command line).

I don't like the idea of override security defaults, I lean to let users modify their containers.conf to support this

[numansiddique]

I'll test it out by configuring in containers.conf and we can probably close this issue.

Thanks

[numansiddique]

Closing this issue as a user can override using containers.conf.

Thanks for the discussion on this topic.