No Connectivity to Published Ports for Containers on Rootless User-Defined CNI Network

[usury]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Unable to access services in containers connected to rootless user-defined CNI network despite proper port publishing

Steps to reproduce the issue:

normaluser@containerhost:~ $ podman rm -af      # fresh set of examples
normaluser@containerhost:~ $ podman network create myCNI
normaluser@containerhost:~ $ podman run --name myProxy  --network myCNI -d --publish 8080:80  nginx:alpine
normaluser@containerhost:~ $ podman run --name myNginx1 --network myCNI -d nginx:alpine

## proper connectivity between containers on "myCNI"
normaluser@containerhost:~ $ podman exec myProxy ping -c 1 myNginx1
    PING myNginx1 (10.88.2.3): 56 data bytes
    64 bytes from 10.88.2.3: seq=0 ttl=42 time=0.088 ms

normaluser@containerhost:~ $ podman exec myNginx1 ping -c 1 myProxy
    PING myProxy (10.88.2.2): 56 data bytes
    64 bytes from 10.88.2.2: seq=0 ttl=42 time=0.057 ms

## CANNOT ACCESS CONTAINER'S PUBLISHED PORT
normaluser@containerhost:~ $ curl --head containerhost:8080
    curl: (56) Recv failure: Connection reset by peer

## Services are properly running **inside** their containers
## - "myProxy" can access itself on its own port 80
normaluser@containerhost:~ $ podman exec myProxy curl --head localhost:80
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/1.1 200 OK
      0   615    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
    Server: nginx/1.21.3
    Date: Sun, 17 Oct 2021 21:48:34 GMT
    Content-Type: text/html
    Content-Length: 615
    Last-Modified: Tue, 07 Sep 2021 15:50:58 GMT
    Connection: keep-alive
    ETag: "61378a62-267"
    Accept-Ranges: bytes

## Services are properly running **inside** their container
## - "myProxy" can access "myNginx1" on its port 80
normaluser@containerhost:~ $ podman exec myProxy curl --head myNginx1:80
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/1.1 200 OK
      0   615    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
    Server: nginx/1.21.3
    Date: Sun, 17 Oct 2021 21:51:33 GMT
    Content-Type: text/html
    Content-Length: 615
    Last-Modified: Tue, 07 Sep 2021 15:50:58 GMT
    Connection: keep-alive
    ETag: "61378a62-267"
    Accept-Ranges: bytes

Describe the results you received:
Cannot access container's published port

Describe the results you expected:
Successful connection to container's published port

Additional information you deem important (e.g. issue happens only occasionally):
Happens consistently and reproducible

Output of podman version:

    normaluser@containerhost:~ $ podman version
        Version:      3.0.1
        API Version:  3.0.0
        Go Version:   go1.14
        Built:        Wed Dec 31 16:00:00 1969
        OS/Arch:      linux/amd64

Output of podman info --debug:

normaluser@containerhost:~ $ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.19.4
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 4
  distribution:
    distribution: debian
    version: "10"
  eventLogger: journald
  hostname: arachne
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 2000
      size: 1
    - container_id: 1
      host_id: 100001
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 2000
      size: 1
    - container_id: 1
      host_id: 100001
      size: 65536
  kernel: 4.19.0-10-amd64
  linkmode: dynamic
  memFree: 392736768
  memTotal: 2091732992
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.18.1-7931a-dirty
      commit: 7931a1eab0590eff4041c1f74e2844b297c31cea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/2000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.4
  swapFree: 670822400
  swapTotal: 1073737728
  uptime: 28h 9m 28.27s (Approximately 1.17 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/ddieringer/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 3
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.4.1
        fuse-overlayfs: version 1.4
        FUSE library version 3.4.1
        using FUSE kernel interface version 7.27
  graphRoot: /home/ddieringer/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 9
  runRoot: /run/user/2000/containers
  volumePath: /home/ddieringer/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Wed Dec 31 16:00:00 1969
  GitCommit: ""
  GoVersion: go1.14
  OsArch: linux/amd64
  Version: 3.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

normaluser@containerhost:~ $ apt list podman
Listing... Done
podman/unknown,now 100:3.0.1-2 amd64 [installed]
podman/unknown 100:3.0.1-2 arm64
podman/unknown 100:3.0.1-2 armhf
podman/unknown 100:3.0.1-2 ppc64el
podman/unknown 100:3.0.1-2 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

This is the latest version of podman in debian repos I know about

Additional environment details (AWS, VirtualBox, physical, etc.):

Containerhost
Container host is a Debian 10 Buster VirtualBox VM running on Fedora 34
podman packages are latest available from OpenSuse repo (as of 17 Oct 2021)

    normaluser@containerhost:~ $ cat /etc/os-release 
        PRETTY_NAME="Debian GNU/Linux 10 (buster)"

    normaluser@containerhost:~ $ cat "/etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list"
        ## "Kubic" repo for "podman" packages since they aren't in Debian 10 (buster) repos
        deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Debian_10/ /

OTHER INFO

root@containerhost:~ # iptables -t nat -L
    Chain PREROUTING (policy ACCEPT)
    target     prot opt source               destination         
    CNI-HOSTPORT-DNAT  all  --  anywhere     anywhere             ADDRTYPE match dst-type LOCAL

    Chain INPUT (policy ACCEPT)
    target     prot opt source               destination         

    Chain POSTROUTING (policy ACCEPT)
    target     prot opt source               destination         
    CNI-HOSTPORT-MASQ  all  --  anywhere     anywhere             /* CNI portfwd requiring masquerade */

    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination         
    CNI-HOSTPORT-DNAT  all  --  anywhere     anywhere             ADDRTYPE match dst-type LOCAL

    Chain CNI-HOSTPORT-SETMARK (0 references)
    target     prot opt source               destination         
    MARK       all  --  anywhere             anywhere             /* CNI portfwd masquerade mark */ MARK or 0x2000

    Chain CNI-HOSTPORT-MASQ (1 references)
    target     prot opt source               destination         
    MASQUERADE  all  --  anywhere            anywhere             mark match 0x2000/0x2000

    Chain CNI-HOSTPORT-DNAT (2 references)
    target     prot opt source               destination

This problem seems to be related to the following other post:
#11396
- talks about problem still present in 3.3.x?
- talks about even higher version numbers being available, like 3.4?

I have been under the impression that the "opensuse" repo is the correct way for Debian users to obtain podman packages.
Is that actually correct? It seems the "opensuse" repo has only v3.0.1, which clearly isn't latest one being discussed.
Should Debian users be getting podman packages from somewhere else?

I had my environment working properly until I upgraded from podman v2.x to v3.0.1, where I've encountered this problem. Though this bug report provides a minimal set of steps to reproduce the issue (above), my actual use case involves a reverse proxy with a published port and multiple additional containers only accessible from the reverse proxy, all within a pod, all rootless.

If there is a better way for Debian systems to install a more recent version of podman, I'd like to know that. Otherwise, I'd sure love a work-around for this issue.

Thank You

[usury]

Even though podman v3.0.x seems to be the latest podman in the common opensuse repo for Debian 10 (buster), I checked a few other systems available to me.

A fresh Debian 11 (bullseye) system

$> root@bullseye:~ # apt list podman
Listing... Done
podman/stable 3.0.1+dfsg1-3+b2 amd64

A Raspian 10 (buster) system

root@raspian:~ # apt list podman
Listing... Done
podman/unstable 3.3.1+ds2-1 armhf

So, it seems a more recent version of podman is available in Debian "unstable".

Sure enough. I temporarily added the following to /etc/apt/sources.list

deb https://deb.debian.org/debian/ unstable

... then ...

apt-get update
apt-get remove podman #removes current version
apt-get install podman  #installs new version

It pulls in a lot of other dependencies or updated versions of other things. Sure enough, new version fixed the problem I reported.

However, it is unfortunate that the version of podman available in "opensuse" (and "debian 11" for that matter) is one with this particular bug. It would certainly be confounding to users unfamiliar with containers or unfamiliar with podman. As in, they don't know they are fighting an already-fixed bug.

Also, pulling in podman from "unstable" may be perfectly fine for a development box or someone's home use (like me), but probably not appropriate for production systems that can't have their packages widely updated.

[Luap99]

We do not have control over debian packaging. Generally speaking we only support the latest version upstream. If you have problems with an older version in your distro you should report the bug in their bug tracker.

[usury]

We do not have control over debian packaging. Generally speaking we only support the latest version upstream. If you have problems with an older version in your distro you should report the bug in their bug tracker.

I get that, for sure.

For the next person encountering what would appear to them as a problem with podman (or their understanding of how to use it), hopefully this bug report proves helpful.

After all, a section like "Hey, the package maintainer for your very popular distro might be stocking a version of podman containing bugs that we have already fixed" is missing from the Podman Troubleshooting Guide