Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rootless pods with published ports observed blocking connections #7231

Closed
maybe-sybr opened this issue Aug 5, 2020 · 6 comments
Closed

Rootless pods with published ports observed blocking connections #7231

maybe-sybr opened this issue Aug 5, 2020 · 6 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@maybe-sybr
Copy link
Contributor

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug

Description

I'm using podman play kube to run up some infrastructure I've been working on and have noticed recently that after a number of network transactions with the pod, connections seem to cease to be forwarded to the containers. The blocking present as if SYNs were being lost. I don't yet have a repro I can sensibly share but wanted to get an issue up in the event that this kind of misbehaviour had been seen before.

Steps to reproduce the issue:

  1. TBC repro k8s YAML

  2. Make lots of HTTP requests, watch it block forever at some point

Describe the results you received:

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 2.0.2

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.18-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.18, commit: 6e8799f576f11f902cd8a8d8b45b2b2caf636a85'
  cpus: 8
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: host
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 31337
      size: 1
    - container_id: 1
      host_id: 300000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1337
      size: 1
    - container_id: 1
      host_id: 300000
      size: 65536
  kernel: 5.7.9-200.fc32.x86_64
  linkmode: dynamic
  memFree: 2401247232
  memTotal: 8235970560
  ociRuntime:
    name: crun
    package: crun-0.14.1-1.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.14.1
      commit: 598ea5e192ca12d4f6378217d3ab1415efeddefa
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1337/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 7984250880
  swapTotal: 8392798208
  uptime: 223h 57m 32.82s (Approximately 9.29 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 11
    paused: 0
    running: 11
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/user/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 43
  runRoot: /run/user/1337/containers
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 10:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.3
  OsArch: linux/amd64
  Version: 2.0.2

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.0.2-1.fc32.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):
@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Aug 5, 2020
@maybe-sybr
Copy link
Contributor Author

I also noticed while trying to get a minimal repro this afternoon that spinning curl against the nginxhello image running in a pod would get recv failures and block intermittently. I'm not sure if this is the same behaviour because after about 10-20 seconds, it'll come back.

If someone could drop some handy tips for how I might be able to get some logging out of the rootlessport process, that might make it easier to track down what's going on.

@mheon
Copy link
Member

mheon commented Aug 5, 2020

@AkihiroSuda Does this look like a dupe of #7016

@AkihiroSuda
Copy link
Collaborator

after about 10-20 seconds, it'll come back.

This might be different from #7016. Needs full reproducer for further analysis.

@mheon
Copy link
Member

mheon commented Aug 5, 2020

@maybe-sybr Any chance you can upgrade to Podman 2.0.4 and re-test? If it still occurs, a more detailed reproducer would be greatly appreciated.

@maybe-sybr
Copy link
Contributor Author

maybe-sybr commented Aug 6, 2020 •
edited
Loading

@mheon @AkihiroSuda It looks like you might be right, the symptoms in #7016 sound almost exactly like what I'm observing. The intermittent failures of my nginxhello setup didn't match the behaviours of my more complex system, which is what I'm more concerned about at the moment. I'll bump to 2.0.4 and do some work today to see if the problems with the complex pod go away.

For reference, the repro I was trying to make with nginxhello looks like:

# k8s.yml
apiVersion: v1
kind: Pod
metadata:
  name: hello-world
spec:
  containers:
  -
    image: docker.io/nginxdemos/hello:plain-text
    name: nginx
    ports:
    - containerPort: 80
      hostPort: 8000
      protocol: TCP
  -
    image: docker.io/plavignotte/continuous-curl
    name: curl
    command: ["http://localhost/"]
    env:
    - name: "DEBUG"
      value: 1
    - name: "LOOP"
      value: 1

Then run:

$ podman play kube k8s.yml
$ podman logs -f hello-world-nginx
# ... watch for request lines from the curl container

in another term

$ watch -n 0.1 curl -vv http://localhost:8000
...

I noticed which running all of that, the curl loop running outside the pod will get a recv error and then block for a little while before resuming normal operation after some period of time ~10-20s. The curl loop in the pod appears to continue working fine as seen by the following excerpts from the log output.

Expand log output. curl/7.69 is host, curl/7.49 is the curl container. Note that 7.69 disappears from the log between 7:21:04 and 7:21:18. 
127.0.0.1 - - [05/Aug/2020:07:20:58 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:20:59 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:20:59 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:20:59 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:20:59 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:20:59 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:20:59 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:20:59 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:20:59 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:00 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:00 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:00 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:00 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:00 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:00 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:00 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:00 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:00 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:01 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:01 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:01 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:01 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:01 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:01 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:01 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:01 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:01 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:02 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:02 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:02 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:02 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:02 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:02 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:02 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:02 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:02 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:03 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:03 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:03 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:03 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:03 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:03 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:03 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:03 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:03 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:04 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:04 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:04 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:04 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:04 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:04 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:04 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:04 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:05 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:06 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:07 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:08 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:09 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:10 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:11 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:12 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:13 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:15 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:16 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:17 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:18 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:18 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:18 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:18 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:19 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:19 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:19 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:19 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:19 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:19 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:19 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:19 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:19 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:20 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:20 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:20 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:20 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:20 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:20 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:20 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:20 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:20 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:21 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:21 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:21 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:21 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:21 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:21 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:21 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:21 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:21 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:21 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:22 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:22 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:22 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:22 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:22 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:22 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:22 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:22 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:22 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:22 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:23 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:23 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:23 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:23 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:23 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:23 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:23 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:23 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:23 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:23 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:24 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:24 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.49.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:24 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"
127.0.0.1 - - [05/Aug/2020:07:21:24 +0000] "GET / HTTP/1.1" 200 139 "-" "curl/7.69.1" "-"

@maybe-sybr
Copy link
Contributor Author

maybe-sybr commented Aug 6, 2020 •
edited
Loading

Okay, I can confirm it's almost definitely the issue fixed in rootless-containers/rootlesskit#157.

Trivial debug step which might be useful for any future reports:

$ pgrep -f 'rootlessport' | xargs -n 1 -I PID ls /proc/PID/fd/
...

One of the rootlessport procs had 1024 FDs which I assume gets ulimited in at some point in the container setup. Interestingly I saw the FD list go up quite high and then back down again (900ish back down to 400) which seems like it's doing some periodic reaping somehow:

Running that command during an eventual lockup: 
 user@host $ pgrep -f 'rootless' | xargs -n 1 -I PID ls /proc/PID/fd/
0  1  10  11  12  13  14  15  17  2  3  5  6  7  8  9
0  1  10  12  14  15  16  17  18  19  2  20  21  22  23  24  25  26  27  28  29  3  30  31  32  33  34  35  36  37  4  5  6  7  8  9
0  10  12  15  17  19  20  23  25  27  29  36  38  42  44  48  5  7  9
1  11  14  16  18  2   21  24  26  28  3   37  41  43  45  49  6  8
0    119  14   160  181  201  222  243  264  285  305  326  347  368  389  409  43   450  471  492  512  535  556  577  598  77  98
1    12   140  161  182  202  223  244  265  286  306  327  348  369  39   41   430  451  472  493  513  536  557  578  599  78  99
10   120  141  162  183  203  224  245  266  287  307  328  349  37   390  410  431  452  473  494  514  537  558  579  6    79
100  121  142  163  184  204  225  246  267  288  308  329  35   370  391  411  432  453  474  495  515  538  559  58   60   8
101  122  143  164  185  205  226  247  268  289  309  33   350  371  392  412  433  454  475  496  516  539  56   580  600  80
102  123  144  165  186  206  227  248  269  29   31   330  351  372  393  413  434  455  476  497  517  54   560  581  601  81
103  124  145  166  187  207  228  249  27   290  310  331  352  373  394  414  435  456  477  498  519  540  561  582  61   82
104  125  146  167  188  208  229  25   270  291  311  332  353  374  395  415  436  457  478  499  52   541  562  583  62   83
105  126  147  168  189  209  23   250  271  292  312  333  354  375  396  416  437  458  479  5    520  542  563  584  63   84
106  127  148  169  19   21   230  251  272  293  313  334  355  376  397  417  438  459  48   50   521  543  564  585  64   85
107  128  149  17   190  210  231  252  273  294  314  335  356  377  398  418  439  46   480  500  522  544  565  586  65   86
108  129  15   170  191  211  232  253  274  295  315  336  357  378  399  419  44   460  481  501  523  545  566  587  66   87
109  13   150  171  192  212  233  254  275  296  316  337  358  379  4    42   440  461  482  502  524  546  567  588  67   88
11   130  151  172  193  213  234  255  276  297  317  338  359  38   40   420  441  462  483  503  525  547  568  589  68   89
110  131  152  173  194  214  235  256  277  298  318  339  36   380  400  421  442  463  484  504  526  548  569  59   69   9
111  132  153  174  195  215  236  257  278  299  319  34   360  381  401  422  443  464  485  505  527  549  57   590  7    90
112  133  154  175  196  216  237  258  279  3    32   340  361  382  402  423  444  465  486  506  528  55   570  591  70   91
113  134  155  176  197  217  238  259  28   30   320  341  362  383  403  424  445  466  487  507  529  550  571  592  71   92
114  135  156  177  198  218  239  26   280  300  321  342  363  384  404  425  446  467  488  508  53   551  572  593  72   93
115  136  157  178  199  219  24   260  281  301  322  343  364  385  405  426  447  468  489  509  530  552  573  594  73   94
116  137  158  179  2    22   240  261  282  302  323  344  365  386  406  427  448  469  49   51   531  553  574  595  74   95
117  138  159  18   20   220  241  262  283  303  324  345  366  387  407  428  449  47   490  510  533  554  575  596  75   96
118  139  16   180  200  221  242  263  284  304  325  346  367  388  408  429  45   470  491  511  534  555  576  597  76   97
 user@host $ pgrep -f 'rootless' | xargs -n 1 -I PID ls /proc/PID/fd/
0  1  10  11  12  13  14  15  17  2  3  5  6  7  8  9
0  1  10  12  14  15  16  17  18  19  2  20  21  22  23  24  25  26  27  28  29  3  30  31  32  33  34  35  36  37  4  5  6  7  8  9
0  10  12  14  16  18  2   21  23  25  28  30  33  35  37  4   41  43  52  6  8
1  11  13  15  17  19  20  22  24  26  3   31  34  36  39  40  42  5   53  7  9
0    128  158  188  217  247  277  306  336  366  396  425  455  485  514  544  574  603  633  663  693  722  752  782  811  841  873
1    129  159  189  218  248  278  307  337  367  397  426  456  486  515  545  575  604  634  664  694  723  753  783  812  842  874
10   13   16   19   219  249  279  308  338  368  398  427  457  487  516  546  576  605  635  665  695  724  754  784  813  843  875
100  130  160  190  22   25   28   309  339  369  399  428  458  488  517  547  577  606  636  666  696  725  755  785  814  844  876
101  131  161  191  220  250  280  31   34   37   4    429  459  489  518  548  578  607  637  667  697  726  756  786  815  845  877
102  132  162  192  221  251  281  310  340  370  40   43   46   49   519  549  579  608  638  668  698  727  757  787  816  846  878
103  133  163  193  222  252  282  311  341  371  400  430  460  490  52   55   58   609  639  669  699  728  758  788  817  847  879
104  134  164  194  223  253  283  312  342  372  401  431  461  491  520  550  580  61   64   67   7    729  759  789  818  849  88
105  135  165  195  224  254  284  313  343  373  402  432  462  492  521  551  581  610  640  670  70   73   76   79   819  85   880
106  136  166  196  225  255  285  314  344  374  403  433  463  493  522  552  582  611  641  671  700  730  760  790  82   850  881
107  137  167  197  226  256  286  315  345  375  404  434  464  494  523  553  583  612  642  672  701  731  761  791  820  851  89
108  138  168  198  227  257  287  316  346  376  405  435  465  495  524  554  584  613  643  673  702  732  762  792  821  852  9
109  139  169  199  228  258  288  317  347  377  406  436  466  496  525  555  585  614  644  674  703  733  763  793  822  853  90
11   14   17   2    229  259  289  318  348  378  407  437  467  497  526  556  586  615  645  675  704  734  764  794  823  854  91
110  140  170  20   23   26   29   319  349  379  408  438  468  498  527  557  587  616  646  676  705  735  765  795  824  855  92
111  141  171  200  230  260  290  32   35   38   409  439  469  499  528  558  588  617  647  677  706  736  766  796  825  856  93
112  142  172  201  231  261  291  320  350  380  41   44   47   5    529  559  589  618  648  678  707  737  767  797  826  857  94
113  143  173  202  232  262  292  321  351  381  410  440  470  50   53   56   59   619  649  679  708  738  768  798  827  858  95
114  144  174  203  233  263  293  322  352  382  411  441  471  500  530  560  590  62   65   68   709  739  769  799  828  859  96
115  145  175  204  234  264  294  323  353  383  412  442  472  501  531  561  591  620  650  680  71   74   77   8    829  86   97
116  146  176  205  235  265  295  324  354  384  413  443  473  502  532  562  592  621  651  681  710  740  770  80   83   860  98
117  147  177  206  236  266  296  325  355  385  414  444  474  503  533  563  593  622  652  682  711  741  771  800  830  861  99
118  148  178  207  237  267  297  326  356  386  415  445  475  504  534  564  594  623  653  683  712  742  772  801  831  862
119  149  179  208  238  268  298  327  357  387  416  446  476  505  535  565  595  624  654  684  713  743  773  802  832  863
12   15   18   209  239  269  299  328  358  388  417  447  477  506  536  566  596  625  655  685  714  744  774  803  833  864
120  150  180  21   24   27   3    329  359  389  418  448  478  507  537  567  597  626  656  686  715  745  775  804  834  865
121  151  181  210  240  270  30   33   36   39   419  449  479  508  538  568  598  627  657  687  716  746  776  805  835  866
122  152  182  211  241  271  300  330  360  390  42   45   48   509  539  569  599  628  658  688  717  747  777  806  836  867
123  153  183  212  242  272  301  331  361  391  420  450  480  51   54   57   6    629  659  689  718  748  778  807  837  868
124  154  184  213  243  273  302  332  362  392  421  451  481  510  540  570  60   63   66   69   719  749  779  808  838  869
125  155  185  214  244  274  303  333  363  393  422  452  482  511  541  571  600  630  660  690  72   75   78   809  839  87
126  156  186  215  245  275  304  334  364  394  423  453  483  512  542  572  601  631  661  691  720  750  780  81   84   871
127  157  187  216  246  276  305  335  365  395  424  454  484  513  543  573  602  632  662  692  721  751  781  810  840  872
 user@host $ pgrep -f 'rootless' | xargs -n 1 -I PID ls /proc/PID/fd/
0  1  10  11  12  13  14  15  17  2  3  5  6  7  8  9
0  1  10  12  14  15  16  17  18  19  2  20  21  22  23  24  25  26  27  28  29  3  30  31  32  33  34  35  36  37  4  5  6  7  8  9
0  10  12  14  16  18  2   21  23  26  29  30  32  34  36  38  4   41  43  45  47  49  50  52  54  56  58  6   61  8
1  11  13  15  17  19  20  22  24  28  3   31  33  35  37  39  40  42  44  46  48  5   51  53  55  57  59  60  7   9
0    111  125  139  152  166  18   193  206  22   233  247  260  274  288  300  314  328  341  355  369  382  396  409  49  62  76   886
1    112  126  14   153  167  180  194  207  220  234  248  261  275  289  301  315  329  342  356  37   383  397  41   5   63  77   89
10   113  127  140  154  168  181  195  208  221  235  249  262  276  29   302  316  33   343  357  370  384  398  410  50  64  78   9
100  114  128  141  155  169  182  196  209  222  236  25   263  277  290  303  317  330  344  358  371  385  399  411  51  65  79   90
101  115  129  142  156  17   183  197  21   223  237  250  264  278  291  304  318  331  345  359  372  386  4    412  52  66  8    91
102  116  13   143  157  170  184  198  210  224  238  251  265  279  292  305  319  332  346  36   373  387  40   413  53  67  80   92
103  117  130  144  158  171  185  199  211  225  239  252  266  28   293  306  32   333  347  360  374  388  400  414  54  68  81   93
104  118  131  145  159  172  186  2    212  226  24   253  267  280  294  307  320  334  348  361  375  389  401  415  55  69  82   94
105  119  132  146  16   173  187  20   213  227  240  254  268  281  295  308  321  335  349  362  376  39   402  42   56  7   83   95
106  12   133  147  160  174  188  200  214  228  241  255  269  282  296  309  322  336  35   363  377  390  403  43   57  70  84   96
107  120  134  148  161  175  189  201  215  229  242  256  27   283  297  31   323  337  350  364  378  391  404  44   58  71  848  97
108  121  135  149  162  176  19   202  216  23   243  257  270  284  298  310  324  338  351  365  379  392  405  45   59  72  85   98
109  122  136  15   163  177  190  203  217  230  244  258  271  285  299  311  325  339  352  366  38   393  406  46   6   73  86   99
11   123  137  150  164  178  191  204  218  231  245  259  272  286  3    312  326  34   353  367  380  394  407  47   60  74  87
110  124  138  151  165  179  192  205  219  232  246  26   273  287  30   313  327  340  354  368  381  395  408  48   61  75  88
 user@host $ pgrep -f 'rootless' | xargs -n 1 -I PID ls /proc/PID/fd/
0  1  10  11  12  13  14  15  17  2  3  5  6  7  8  9
0  1  10  12  14  15  16  17  18  19  2  20  21  22  23  24  25  26  27  28  29  3  30  31  32  33  34  35  36  37  4  5  6  7  8  9
0  10  12  14  16  18  2   21  23  25  27  29  30  32  34  4   5  63  8
1  11  13  15  17  19  20  22  24  26  28  3   31  33  35  43  6  7   9
0     111  146  180  214  249  283  317  351  386  42   454  489  522  557  591  625  66   694  728  762  797  830  865  9    933  968
1     112  147  181  215  25   284  318  352  387  420  455  49   523  558  592  626  660  695  729  763  798  831  866  90   934  969
10    113  148  182  216  250  285  319  353  388  421  456  490  524  559  593  627  661  696  73   764  799  832  867  900  935  97
100   114  149  183  217  251  286  32   354  389  422  457  491  525  56   594  628  662  697  730  765  8    833  868  901  936  970
1000  115  15   184  218  252  287  320  355  39   423  458  492  526  560  595  629  663  698  731  766  80   834  869  902  937  971
1001  116  150  185  219  253  288  321  356  390  424  459  493  527  561  596  63   664  699  732  767  800  835  87   903  938  972
1002  117  151  186  22   254  289  322  357  391  425  46   494  528  562  597  630  665  7    733  768  801  836  870  904  939  973
1003  118  152  187  220  255  29   323  358  392  426  460  495  529  563  598  631  666  70   734  769  802  837  871  905  94   974
1004  119  153  188  221  256  290  324  359  393  427  461  496  53   564  599  632  667  700  735  77   803  838  872  906  940  975
1005  12   154  189  222  257  291  325  36   394  428  462  497  530  565  6    633  668  701  736  770  804  839  873  907  941  976
1006  120  155  19   223  258  292  326  360  395  429  463  498  531  566  60   634  669  702  737  771  805  84   874  908  942  977
1007  121  156  190  224  259  293  327  361  396  43   464  499  532  567  600  635  67   703  738  772  806  840  875  909  943  978
1008  122  157  191  225  26   294  328  362  397  430  465  5    533  568  601  636  670  704  739  773  807  841  876  91   944  979
1009  123  158  192  226  260  295  329  363  398  431  466  50   534  569  602  637  671  705  74   774  808  842  877  910  945  98
101   124  159  193  227  261  296  33   364  399  432  467  500  535  57   603  638  672  706  740  775  809  843  878  911  946  980
1010  125  16   194  228  262  297  330  365  4    433  468  501  536  570  604  639  673  707  741  776  81   844  879  912  947  981
1011  126  160  195  229  263  298  331  366  40   434  469  502  537  571  605  64   674  708  742  777  810  845  88   913  948  982
1012  127  161  196  23   264  299  332  367  400  435  47   503  538  572  606  640  675  709  743  778  811  846  880  914  949  983
1013  128  162  197  230  265  3    333  368  401  436  470  504  539  573  607  641  676  71   744  779  812  847  881  915  95   984
1014  129  163  198  231  266  30   334  369  402  437  471  505  54   574  608  642  677  710  745  78   813  848  882  916  950  985
1015  13   164  199  232  267  300  335  37   403  438  472  506  540  575  609  643  678  711  746  780  814  849  883  917  951  986
1016  130  165  2    233  268  301  336  370  404  439  473  507  541  576  61   644  679  712  747  781  815  85   884  918  952  987
1017  131  166  20   234  269  302  337  371  405  44   474  508  542  577  610  645  68   713  748  782  816  850  885  919  953  989
1018  132  167  200  235  27   303  338  372  406  440  475  509  543  578  611  646  680  714  749  783  817  851  886  92   954  99
1019  133  168  201  236  270  304  339  373  407  441  476  51   544  579  612  647  681  715  75   784  818  852  887  920  955  990
102   134  169  202  237  271  305  34   374  408  442  477  510  545  58   613  648  682  716  750  785  819  853  888  921  956  991
1020  135  17   203  238  272  306  340  375  409  443  478  511  546  580  614  649  683  717  751  786  82   854  889  922  957  992
1022  136  170  204  239  273  307  341  376  41   444  479  512  547  581  615  65   684  718  752  787  820  855  89   923  958  993
1023  137  171  205  24   274  308  342  377  410  445  48   513  548  582  616  650  685  719  753  788  821  856  890  924  959  994
103   138  172  206  240  275  309  343  378  411  446  480  514  549  583  617  651  686  72   754  789  822  857  891  925  96   995
104   139  173  207  241  276  31   344  379  412  447  481  515  55   584  618  652  687  720  755  79   823  858  892  926  960  996
105   14   174  208  242  277  310  345  38   413  448  482  516  550  585  619  653  688  721  756  790  824  859  893  927  961  997
106   140  175  209  243  278  311  346  380  414  449  483  517  551  586  62   654  689  722  757  791  825  86   894  928  962  998
107   141  176  21   244  279  312  347  381  415  45   484  518  552  587  620  655  69   723  758  792  826  860  895  929  963  999
108   142  177  210  245  28   313  348  382  416  450  485  519  553  588  621  656  690  724  759  793  827  861  896  93   964
109   143  178  211  246  280  314  349  383  417  451  486  52   554  589  622  657  691  725  76   794  828  862  897  930  965
11    144  179  212  247  281  315  35   384  418  452  487  520  555  59   623  658  692  726  760  795  829  863  898  931  966
110   145  18   213  248  282  316  350  385  419  453  488  521  556  590  624  659  693  727  761  796  83   864  899  932  967
 user@host $ pgrep -f 'rootless' | xargs -n 1 -I PID ls /proc/PID/fd/
0  1  10  11  12  13  14  15  17  2  3  5  6  7  8  9
0  1  10  12  14  15  16  17  18  19  2  20  21  22  23  24  25  26  27  28  29  3  30  31  32  33  34  35  36  37  4  5  6  7  8  9
0  10  12  14  16  18  2   21  23  25  27  29  30  32  34  4   5  63  8
1  11  13  15  17  19  20  22  24  26  28  3   31  33  35  43  6  7   9
0     111  146  180  214  249  283  317  351  386  42   454  489  522  557  591  625  66   694  728  762  797  830  865  9    933  968
1     112  147  181  215  25   284  318  352  387  420  455  49   523  558  592  626  660  695  729  763  798  831  866  90   934  969
10    113  148  182  216  250  285  319  353  388  421  456  490  524  559  593  627  661  696  73   764  799  832  867  900  935  97
100   114  149  183  217  251  286  32   354  389  422  457  491  525  56   594  628  662  697  730  765  8    833  868  901  936  970
1000  115  15   184  218  252  287  320  355  39   423  458  492  526  560  595  629  663  698  731  766  80   834  869  902  937  971
1001  116  150  185  219  253  288  321  356  390  424  459  493  527  561  596  63   664  699  732  767  800  835  87   903  938  972
1002  117  151  186  22   254  289  322  357  391  425  46   494  528  562  597  630  665  7    733  768  801  836  870  904  939  973
1003  118  152  187  220  255  29   323  358  392  426  460  495  529  563  598  631  666  70   734  769  802  837  871  905  94   974
1004  119  153  188  221  256  290  324  359  393  427  461  496  53   564  599  632  667  700  735  77   803  838  872  906  940  975
1005  12   154  189  222  257  291  325  36   394  428  462  497  530  565  6    633  668  701  736  770  804  839  873  907  941  976
1006  120  155  19   223  258  292  326  360  395  429  463  498  531  566  60   634  669  702  737  771  805  84   874  908  942  977
1007  121  156  190  224  259  293  327  361  396  43   464  499  532  567  600  635  67   703  738  772  806  840  875  909  943  978
1008  122  157  191  225  26   294  328  362  397  430  465  5    533  568  601  636  670  704  739  773  807  841  876  91   944  979
1009  123  158  192  226  260  295  329  363  398  431  466  50   534  569  602  637  671  705  74   774  808  842  877  910  945  98
101   124  159  193  227  261  296  33   364  399  432  467  500  535  57   603  638  672  706  740  775  809  843  878  911  946  980
1010  125  16   194  228  262  297  330  365  4    433  468  501  536  570  604  639  673  707  741  776  81   844  879  912  947  981
1011  126  160  195  229  263  298  331  366  40   434  469  502  537  571  605  64   674  708  742  777  810  845  88   913  948  982
1012  127  161  196  23   264  299  332  367  400  435  47   503  538  572  606  640  675  709  743  778  811  846  880  914  949  983
1013  128  162  197  230  265  3    333  368  401  436  470  504  539  573  607  641  676  71   744  779  812  847  881  915  95   984
1014  129  163  198  231  266  30   334  369  402  437  471  505  54   574  608  642  677  710  745  78   813  848  882  916  950  985
1015  13   164  199  232  267  300  335  37   403  438  472  506  540  575  609  643  678  711  746  780  814  849  883  917  951  986
1016  130  165  2    233  268  301  336  370  404  439  473  507  541  576  61   644  679  712  747  781  815  85   884  918  952  987
1017  131  166  20   234  269  302  337  371  405  44   474  508  542  577  610  645  68   713  748  782  816  850  885  919  953  989
1018  132  167  200  235  27   303  338  372  406  440  475  509  543  578  611  646  680  714  749  783  817  851  886  92   954  99
1019  133  168  201  236  270  304  339  373  407  441  476  51   544  579  612  647  681  715  75   784  818  852  887  920  955  990
102   134  169  202  237  271  305  34   374  408  442  477  510  545  58   613  648  682  716  750  785  819  853  888  921  956  991
1020  135  17   203  238  272  306  340  375  409  443  478  511  546  580  614  649  683  717  751  786  82   854  889  922  957  992
1022  136  170  204  239  273  307  341  376  41   444  479  512  547  581  615  65   684  718  752  787  820  855  89   923  958  993
1023  137  171  205  24   274  308  342  377  410  445  48   513  548  582  616  650  685  719  753  788  821  856  890  924  959  994
103   138  172  206  240  275  309  343  378  411  446  480  514  549  583  617  651  686  72   754  789  822  857  891  925  96   995
104   139  173  207  241  276  31   344  379  412  447  481  515  55   584  618  652  687  720  755  79   823  858  892  926  960  996
105   14   174  208  242  277  310  345  38   413  448  482  516  550  585  619  653  688  721  756  790  824  859  893  927  961  997
106   140  175  209  243  278  311  346  380  414  449  483  517  551  586  62   654  689  722  757  791  825  86   894  928  962  998
107   141  176  21   244  279  312  347  381  415  45   484  518  552  587  620  655  69   723  758  792  826  860  895  929  963  999
108   142  177  210  245  28   313  348  382  416  450  485  519  553  588  621  656  690  724  759  793  827  861  896  93   964
109   143  178  211  246  280  314  349  383  417  451  486  52   554  589  622  657  691  725  76   794  828  862  897  930  965
11    144  179  212  247  281  315  35   384  418  452  487  520  555  59   623  658  692  726  760  795  829  863  898  931  966
110   145  18   213  248  282  316  350  385  419  453  488  521  556  590  624  659  693  727  761  796  83   864  899  932  967
 user@host $ pgrep -f 'rootless' | xargs -n 1 -I PID ls /proc/PID/fd/
0  1  10  11  12  14  15  17  2  3  5  6  7  8  9
0  1  10  12  2  3  4  5  6  7  8  9
0  10  12  14  16  18  2   21  23  25  27  29  30  32  34  36  4   5  63  8
1  11  13  15  17  19  20  22  24  26  28  3   31  33  35  37  43  6  7   9
0     111  146  180  214  249  283  317  351  386  42   454  489  522  557  591  625  66   694  728  762  797  830  865  9    933  968
1     112  147  181  215  25   284  318  352  387  420  455  49   523  558  592  626  660  695  729  763  798  831  866  90   934  969
10    113  148  182  216  250  285  319  353  388  421  456  490  524  559  593  627  661  696  73   764  799  832  867  900  935  97
100   114  149  183  217  251  286  32   354  389  422  457  491  525  56   594  628  662  697  730  765  8    833  868  901  936  970
1000  115  15   184  218  252  287  320  355  39   423  458  492  526  560  595  629  663  698  731  766  80   834  869  902  937  971
1001  116  150  185  219  253  288  321  356  390  424  459  493  527  561  596  63   664  699  732  767  800  835  87   903  938  972
1002  117  151  186  22   254  289  322  357  391  425  46   494  528  562  597  630  665  7    733  768  801  836  870  904  939  973
1003  118  152  187  220  255  29   323  358  392  426  460  495  529  563  598  631  666  70   734  769  802  837  871  905  94   974
1004  119  153  188  221  256  290  324  359  393  427  461  496  53   564  599  632  667  700  735  77   803  838  872  906  940  975
1005  12   154  189  222  257  291  325  36   394  428  462  497  530  565  6    633  668  701  736  770  804  839  873  907  941  976
1006  120  155  19   223  258  292  326  360  395  429  463  498  531  566  60   634  669  702  737  771  805  84   874  908  942  977
1007  121  156  190  224  259  293  327  361  396  43   464  499  532  567  600  635  67   703  738  772  806  840  875  909  943  978
1008  122  157  191  225  26   294  328  362  397  430  465  5    533  568  601  636  670  704  739  773  807  841  876  91   944  979
1009  123  158  192  226  260  295  329  363  398  431  466  50   534  569  602  637  671  705  74   774  808  842  877  910  945  98
101   124  159  193  227  261  296  33   364  399  432  467  500  535  57   603  638  672  706  740  775  809  843  878  911  946  980
1010  125  16   194  228  262  297  330  365  4    433  468  501  536  570  604  639  673  707  741  776  81   844  879  912  947  981
1011  126  160  195  229  263  298  331  366  40   434  469  502  537  571  605  64   674  708  742  777  810  845  88   913  948  982
1012  127  161  196  23   264  299  332  367  400  435  47   503  538  572  606  640  675  709  743  778  811  846  880  914  949  983
1013  128  162  197  230  265  3    333  368  401  436  470  504  539  573  607  641  676  71   744  779  812  847  881  915  95   984
1014  129  163  198  231  266  30   334  369  402  437  471  505  54   574  608  642  677  710  745  78   813  848  882  916  950  985
1015  13   164  199  232  267  300  335  37   403  438  472  506  540  575  609  643  678  711  746  780  814  849  883  917  951  986
1016  130  165  2    233  268  301  336  370  404  439  473  507  541  576  61   644  679  712  747  781  815  85   884  918  952  987
1017  131  166  20   234  269  302  337  371  405  44   474  508  542  577  610  645  68   713  748  782  816  850  885  919  953  989
1018  132  167  200  235  27   303  338  372  406  440  475  509  543  578  611  646  680  714  749  783  817  851  886  92   954  99
1019  133  168  201  236  270  304  339  373  407  441  476  51   544  579  612  647  681  715  75   784  818  852  887  920  955  990
102   134  169  202  237  271  305  34   374  408  442  477  510  545  58   613  648  682  716  750  785  819  853  888  921  956  991
1020  135  17   203  238  272  306  340  375  409  443  478  511  546  580  614  649  683  717  751  786  82   854  889  922  957  992
1022  136  170  204  239  273  307  341  376  41   444  479  512  547  581  615  65   684  718  752  787  820  855  89   923  958  993
1023  137  171  205  24   274  308  342  377  410  445  48   513  548  582  616  650  685  719  753  788  821  856  890  924  959  994
103   138  172  206  240  275  309  343  378  411  446  480  514  549  583  617  651  686  72   754  789  822  857  891  925  96   995
104   139  173  207  241  276  31   344  379  412  447  481  515  55   584  618  652  687  720  755  79   823  858  892  926  960  996
105   14   174  208  242  277  310  345  38   413  448  482  516  550  585  619  653  688  721  756  790  824  859  893  927  961  997
106   140  175  209  243  278  311  346  380  414  449  483  517  551  586  62   654  689  722  757  791  825  86   894  928  962  998
107   141  176  21   244  279  312  347  381  415  45   484  518  552  587  620  655  69   723  758  792  826  860  895  929  963  999
108   142  177  210  245  28   313  348  382  416  450  485  519  553  588  621  656  690  724  759  793  827  861  896  93   964
109   143  178  211  246  280  314  349  383  417  451  486  52   554  589  622  657  691  725  76   794  828  862  897  930  965
11    144  179  212  247  281  315  35   384  418  452  487  520  555  59   623  658  692  726  760  795  829  863  898  931  966
110   145  18   213  248  282  316  350  385  419  453  488  521  556  590  624  659  693  727  761  796  83   864  899  932  967

Perhaps that reaping is how my nginxhello sample manages to recover.

Bumping to 2.0.4 appears to have fixed the issue without a reboot, as we'd expect. I'm going to close this as a dupe of #7016.

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mheon @AkihiroSuda @openshift-ci-robot @maybe-sybr