Document port forwarding behaviors between different port forwarders

[io7m]

Issue Description

I maintain a small Java library used to start containers using podman for the purposes of integration testing.

Yesterday, I inadvertently upgraded to podman 5.0.0 as part of a system upgrade, and suddenly multiple tests are failing in the test suites of various projects. I've tracked it down to something I'm finding completely inexplicable:

I can start podman from a Java program, and be unable to connect to the container from the Java program. However, at the same time the program and container are running, I can connect to the container just fine using nc. I can even have the Java program start nc and tell it to connect to the container and it will work fine.

I've managed to put together a repro case that can be found here: https://github.com/io7m/podmanbug-20240330/blob/master/src/main/java/com/io7m/podmanbug/Main.java

Assuming you have a JDK 21+ implementation installed, you can:

$ java src/main/java/com/io7m/podmanbug/Main.java

This has been working fine for months, and the only thing that changed in the setup was a move from Podman 4.* to Podman 5.0.0. I was advised in the Matrix channel to report it here.

Steps to reproduce the issue

Steps to reproduce the issue

1. Run a Java program that starts a container with podman run.
2. Try to connect to the running container.
3. Note the repeated connection failures.
4. Bonus step: Press the return key to spawn an nc process that will connect just fine!

Describe the results you received

I consistently see connection failures from the Java program:

Couldn't connect: Connection refused
Couldn't connect: Connection refused
Couldn't connect: Connection refused

And yet, with nc:

Connection to :: 5432 port [tcp/postgresql] succeeded!

Describe the results you expected

I should be able to connect to a TCP port bound to [::].

podman info output

host:
  arch: amd64
  buildahVersion: 1.35.1
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.10-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: 2dcd736e46ded79a53339462bc251694b150f870'
  cpuUtilization:
    idlePercent: 97.74
    systemPercent: 0.46
    userPercent: 1.79
  cpus: 12
  databaseBackend: boltdb
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2028
  hostname: workstation01
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 184135848
      size: 1
    - container_id: 1
      host_id: 200000000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 184135848
      size: 1
    - container_id: 1
      host_id: 200000000
      size: 65536
  kernel: 6.8.2-arch2-1
  linkmode: dynamic
  logDriver: journald
  memFree: 9231446016
  memTotal: 33300709376
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
    package: /usr/lib/podman/netavark is owned by netavark 1.10.3-1
    path: /usr/lib/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.14.4-1
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/user/184135848/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: /usr/bin/pasta is owned by passt 2024_03_26.4988e2b-1
    version: |
      pasta 2024_03_26.4988e2b
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/184135848/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.3-1
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 5h 41m 47.00s (Approximately 0.21 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  registry.int.arc7.info:5000:
    Blocked: false
    Insecure: true
    Location: registry.int.arc7.info:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: registry.int.arc7.info:5000
    PullFromMirror: ""
store:
  configFile: /home/rm/etc/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 0
    stopped: 4
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/rm/local/containers/storage
  graphRootAllocated: 982821621760
  graphRootUsed: 593637048320
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 718
  runRoot: /run/user/184135848/containers
  transientStore: false
  volumePath: /home/rm/local/containers/storage/volumes
version:
  APIVersion: 5.0.0
  Built: 1711060217
  BuiltTime: Thu Mar 21 22:30:17 2024
  GitCommit: e71ec6f1d94d2d97fb3afe08aae0d8adaf8bddf0-dirty
  GoVersion: go1.22.1
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Local machine running Arch Linux:

Linux workstation01 6.8.2-arch2-1 #1 SMP PREEMPT_DYNAMIC Thu, 28 Mar 2024 17:06:35 +0000 x86_64 GNU/Linux

Additional information

No response

[io7m]

I've just been advised to run podman system reset. It made no difference.

[io7m]

After doing some packet capture, it turns out to be something I didn't expect.

From what I can make out, either intentionally or unintentionally, --publish [::]:5432:5432/tcp used to mean "bind to all addresses". Now it seems to mean "bind on all IPv6 addresses". This wouldn't normally matter but it seems that, for some reason I don't understand, Java's Socket.connect() is preferring IPv4 (despite java.net.preferIPv4Stack=true not being set, and despite the argument to connect() ostensibly being an IPv6 address).

This is likely something that was always wrong on the Java side, but that used to accidentally work on pre-5.0.0 Podman.

If I switch to just using --publish 5432:5432/tcp, I get the old behaviour.

[io7m]

The podman documentation actually only mentions using 0.0.0.0 for the IP argument, so arguably the documented behaviour hasn't changed. 😉

[rhatdan]

The big switch in podman 5.0 is from slirp4netns to pasta for the default rootless network stack. If you switch containers.conf to point at slirp4netns, does your problem go away?

@Luap99 PTAL

[io7m]

If you switch containers.conf to point at slirp4netns, does your problem go away?

It does, yes. Setting slirp4netns causes a --publish option that specifies [::] to bind to all addresses. Under pasta, it only binds IPv6 addresses. Specifying no address at all binds to both IPv4 and IPv6.

[io7m]

The new behaviour is better (more expressive), for what it's worth. I would consider the old behaviour to be a bug.

[sbrivio-rh]

It does, yes. Setting slirp4netns causes a --publish option that specifies [::] to bind to all addresses.

By the way, as far as I know, the current version of slirp4netns can't actually forward IPv6 ports, see rootless-containers/slirp4netns#253 -- so that might be the reason.

[Luap99]

We do not use the slirp4netns port forwarder by default with slirpo4netns, instead we use rootlessport
And given it is go the go std lib has the weird special case that binding 0.0.0.0 or :: means dual stack unless tcp4/tcp6 is used explicitly as protocol. We do not do that there. There is also the special case that rootlessport will map incoming ipv6 to ipv4 inside the container.

[sbrivio-rh]

We do not use the slirp4netns port forwarder by default with slirpo4netns, instead we use rootlessport And given it is go the go std lib has the weird special case that binding 0.0.0.0 or :: means dual stack unless tcp4/tcp6 is used explicitly as protocol.

Aah, right.

Anyway, should we leave the current behaviour with pasta as it is? I also think it's desirable (same as #22221 (comment)).

It has the downside of being inconsistent with rootlessport, but then again, slirp4netns and rootlessport behave differently anyway with e.g. NAT and changing the source address. I would be tempted to say that it's not entirely Podman's job to define what :: is.

[Luap99]

yes what pasta is doing is good, consistent and IMO the least surprising behaviour.

It is a shame that these things ever work so slightly different in these corner cases but I don't think it is worth trying to make them behave the same. But these rootlessport quirks are definitely something we should document in some form.

[io7m]

I agree, what pasta is doing right now is the right behaviour as far as I'm concerned. The behaviour as it is now allows for expressing all of the following:

• All IPv4 and IPv6 addresses (don't specify an IP at all for --publish)
• All IPv4 addresses, but not IPv6 (0.0.0.0 for --publish)
• All IPv6 addresses, but not IPv4 (::)

This evidently wasn't quite doable before. You might get more or less than you asked for depending on how you asked.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[dgibson]

@io7m is there anything we still want to change here? Or can this ticket be closed?

[Luap99]

We need to update the documentation on how port forwarding works with no ip given, 0.0.0.0 or :: set...
But in order to do so I first must test the behaviour of all our ways to do port forwarding and I don't really have time to deal with that.

And to be clear it is a lot to test:
rootless: pasta slirp4netns:port_handler=slirp4netns slirp4netns:port_handler=rootlesskit
rootful: netavark using iptables or nftables driver (checking podman bind logic I know this is broken for v6 #17782) and the difference between networks that are v4/v6 only or dual stack. I know there are a lot of problem there...

This all needs to get tested and then work out which things are considered bugs we should fix and then document the findings in the man page. I retitle the issue to make this clear.

[io7m]

@dgibson I think the actual functionality is working well in 5.x.x. I assume the only work left is the bookkeeping @Luap99 mentioned. 👍