Flaky ingress behavior using ingress-nginx and rootless podman #3451
Comments
|
Have you followed https://kind.sigs.k8s.io/docs/user/rootless/ ? |
|
This may help? https://access.redhat.com/solutions/22105
|
Yes, I've done everything there.
Thanks, that does seem likely to be the problem. The nproc limit currently seems to be ~256k for the processes in my nginx pod. I don't think that's too low but I'll experiment with raising it.
|
|
Some more progress: I looked at the If I And on the Nginx pod specifically: I see that the |
|
One more thing: apparently Podman's default pids limit is 2048. And in a few places, I've seen |
|
Hey, I got it to work by applying the following patch (to the diff --git a/pkg/cluster/internal/providers/podman/provision.go b/pkg/cluster/internal/providers/podman/provision.go
index c240a292..4b276ba5 100644
--- a/pkg/cluster/internal/providers/podman/provision.go
+++ b/pkg/cluster/internal/providers/podman/provision.go
@@ -130,6 +130,7 @@ func commonArgs(cfg *config.Cluster, networkName string, nodeNames []string) ([]
// standard arguments all nodes containers need, computed once
args := []string{
"--detach", // run the container detached
+ "--pids-limit=65536", // higher pids limit
"--tty", // allocate a tty for entrypoint logs
"--net", networkName, // attach to its own network
// label the node with the cluster IDWith this, the nginx logs look much happier and the curl request to I think perhaps a FWIW, I tested what Docker does (the normal rootful one), and it seems to inherit the |
|
Hi, i was having the same issues with kind/ nginx ingress when provisioning using podman, can confirm it's indeed podman setting default PidsLimit:2048 when creating container; For some strange reason this issue did not appear when trying to create cluster using podman for windows / podman machine, there podman created the container with PidsLimit:0; Setting pids_limit=0 in /etc/containers/containers.conf had the same effect for me as applying the above mentioned patch. my /etc/containers/containers.conf now looks like this: @thomasjm: thanks a lot for investigating |
|
@thomasjm interesting -- I don't think we should apply that patch because the number of pids we'd hardcode would be arbitrary and defeat the local user config that you can set as mentioned in the comment above. You can control this as a user by changing podman config which is probably the right approach for now. We should add a warning about podman pidslimit config to the rootless docs though, and consider asking upstream if podman rootless might default to a higher limit. If not, maybe we could set |
|
And thank you for debugging this! Very appreciated. |
|
Thanks @thomasjm, your patch fixed my issue on nixos and tag |
|
If someone wants to add a note to the rootless docs our contributor guide covers everything including docs https://kind.sigs.k8s.io/docs/contributing/getting-started/ |
What happened:
I'm trying to run the basic Ingress example with Ingress Nginx from here. I'm using rootless podman.
Once I create the example services and ingress, I try to do
curl localhost:12345/foo/hostname. Note that there's a random port12345, due to the fact that I pass0for thehostPortin the Kind cluster config (see below).About 10% of the time it works and I get the desired response
foo-app. The rest of the time, thecurlcommand hangs indefinitely.When I look in the nginx controller logs, I see a lot of messages like the following:
This looks to me like
nginxis spawning a bunch of worker threads, and most of them are failing to create properly. Maybe there's some problem with rootless podman?What you expected to happen:
HTTP requests to the ingress should work reliably.
How to reproduce it (as minimally and precisely as possible):
Just following the Ingress instructions. My exact Kind config is as follows:
Kind config file
Environment:
kind version):0.20.0docker infoorpodman info):podman info output
/etc/os-release):NixOS 23.05kubectl version):1.27.1The text was updated successfully, but these errors were encountered: