Published port "connection refused" permanently if connected before listening

[jmou]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I have been having this issue for a while and finally found an isolated reproduction. In certain situations, a published port may refuse connections until the container is restarted, even if there is a container process listening. This appears to happen if connections are attempted to the port before there is a listener. In my testing, at least two connections must be attempted to the port before the listener connects.

Steps to reproduce the issue:

1. In one terminal:

$ podman run --rm --name nc -it --entrypoint /bin/sh -p 2000:2000 toolbelt/netcat -c 'echo sleeping; sleep 5; echo listening; nc -lp 2000'

2. In a second terminal:

$ nc localhost 2000 <<< hi  # run this while sleeping
Ncat: Connection reset by peer.
$ nc localhost 2000 <<< hi  # also run this while sleeping
Ncat: Connection refused.
$ nc localhost 2000 <<< hi  # run this while listening
Ncat: Connection refused.
$ podman kill nc  # first terminal appears unresponsive at this point

Describe the results you received:

All netcats fail, no output on first terminal, first terminal does not quit on Ctrl-C.

Describe the results you expected:

First two netcats are expected to fail because the first terminal's netcat is not yet listening. However the third netcat should succeed, and hi should be output.

I have tested this with Docker and the results are similar to what I expect, but I would expect the first two netcats to fail (I only see output on the third netcat):

$ nc -v localhost 2000 <<< hi  # sleeping
Connection to localhost 2000 port [tcp/cisco-sccp] succeeded!
$ nc -v localhost 2000 <<< hi  # sleeping
Connection to localhost 2000 port [tcp/cisco-sccp] succeeded!
$ nc -v localhost 2000 <<< hi  # listening, output occurs here
Connection to localhost 2000 port [tcp/cisco-sccp] succeeded!

Additional information you deem important (e.g. issue happens only occasionally):

I am only able to reproduce this if at least two of the netcats are run while the first terminal is sleeping. If only one (or zero) netcats are run while sleeping, everything behaves as expected. Also perhaps of note is the first error is connection reset and the second and subsequent errors are connection refused.

I know slirp4netns is related to podman networking, but I don't know enough about the internals to be sure where the bug is, so I reported it here. I am using rootless podman.

Output of podman version:

Version:            1.8.0
RemoteAPI Version:  1
Go Version:         go1.14rc1
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.14rc1
  podman version: 1.8.0
host:
  BuildahVersion: 1.13.1
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.11-0.6.dev.git86aa80b.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.11-dev, commit: b6ad06ed817c8b9d68bb7b92f32b7fb6d08bbb8f'
  Distribution:
    distribution: fedora
    version: "32"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 1574301696
  MemTotal: 8027217920
  OCIRuntime:
    name: crun
    package: crun-0.12.2.1-1.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.12.2.1
      commit: cd7cea7114db5f6aa35fbb69fa307c19c2728a31
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 8162111488
  SwapTotal: 8162111488
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: sparky
  kernel: 5.6.0-0.rc5.git0.2.fc32.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.3-6.0.dev.gita8414d1.fc32.x86_64
    Version: |-
      slirp4netns version 0.4.3+dev
      commit: a8414d1d1629f6f7a93b60b55e183a93d10d9a1c
  uptime: 13h 42m 40.39s (Approximately 0.54 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  ConfigFile: /var/home/joe/.config/containers/storage.conf
  ContainerStore:
    number: 14
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.6-2.0.dev.gitce61c7e.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.0
        fuse-overlayfs: version 0.7.6
        FUSE library version 3.9.0
        using FUSE kernel interface version 7.31
  GraphRoot: /var/home/joe/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 13
  RunRoot: /run/user/1000/containers
  VolumePath: /var/home/joe/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.8.0-5.fc32.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

physical

[Gelbpunkt]

I've had this issue too, only in rootless. Restarting the container usually helps, but not always. It also applies to pods themselves. To fix it in a pod, I have to restart the pod.

Edit:

debug:
  compiler: gc
  git commit: ""
  go version: go1.13.6
  podman version: 1.8.2
host:
  BuildahVersion: 1.14.3
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.14-1.fc31.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.14, commit: 083a0be12178013d44ff51ceda3090ea741b6516'
  Distribution:
    distribution: fedora
    version: "31"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  MemFree: 182669516800
  MemTotal: 270157025280
  OCIRuntime:
    name: crun
    package: crun-0.13-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.13
      commit: e79e4de4ac16da0ce48777afb72c6241de870525
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 34324082688
  SwapTotal: 34324082688
  arch: amd64
  cpus: 12
  eventlogger: journald
  hostname: xena
  kernel: 5.5.11-200.fc31.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.0-20.1.dev.gitbbd6f25.fc31.x86_64
    Version: |-
      slirp4netns version 0.4.0-beta.3+dev
      commit: bbd6f25c70d5db2a1cd3bfb0416a8db99a75ed7e
  uptime: 120h 1m 19.5s (Approximately 5.00 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - quay.io
store:
  ConfigFile: /home/jens/.config/containers/storage.conf
  ContainerStore:
    number: 56
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.8-1.fc31.x86_64
      Version: |-
        fusermount3 version: 3.6.2
        fuse-overlayfs: version 0.7.8
        FUSE library version 3.6.2
        using FUSE kernel interface version 7.29
  GraphRoot: /home/jens/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 61
  RunRoot: /run/user/1001
  VolumePath: /home/jens/.local/share/containers/storage/volumes

[mheon]

@giuseppe @AkihiroSuda I feel like we may have fixed this one already?

[ingobecker]

Having the same issue. I'm staring a shell in an interactive container($ podman run -it ...) and from there starting the development webserver of my webframework. If i have to kill the development server (strg+c) and restart it, it's impossible to connect to the server. The only thing that helps is to restart the container.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@giuseppe @AkihiroSuda Any comment on this one?

[AkihiroSuda]

Maybe fixed in #5183 ?

[giuseppe]

yes I think it is fixed upstream, there were few fixes in rootlessport related to it.

@ingobecker @jmou could you verify if the issue still persists in 1.9?

[ingobecker]

It's fixed. Thanks for the reply.

[jmou]

It's fixed for me, thanks!