Port forwarding with podman machine for 127.0.0.1 should work

[netvl]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When using podman machine (created via simple podman machine init/podman machine start) on macOS, port forwarding with explicit 127.0.0.1 for host does not work, most likely because the listen address is passed as is into the virtual machine, and then the proxy tool cannot connect to the virtual machine's port (which is visible by stray error log entries from tcpproxy).

Steps to reproduce the issue:

> podman machine init
> podman machine start
> podman run -d -p 18888:80 docker.io/library/nginx
8355e4fa8ce0161f7517207a34f5480359c123f3922e3600712c574af2a53e0b
> podman run -d -p 127.0.0.1:28888:80 docker.io/library/nginx
713a04a35b03bf02747c11757de7382f3ef773d4c2ffd38bbcb931a5e2035dac
> curl -s -I localhost:18888 | head -n1
HTTP/1.1 200 OK
> curl -s -I localhost:28888 | head -n1
2021/09/10 17:01:54 tcpproxy: for incoming conn 127.0.0.1:53866, error dialing "192.168.127.2:28888": connect tcp 192.168.127.2:28888: connection was refused

Describe the results you received:

cURL call to 28888 in the above example fails, while the same call to 18888 succeeds.

Describe the results you expected:

Both calls succeed.

Additional information you deem important (e.g. issue happens only occasionally):

Discovered this when trying to use podman instead of Docker Deskop together with some internal script which always uses 127.0.0.1 for listen host (and cannot be configured not to do so).

Note that Docker for Mac supports 127.0.0.1 as expected, so the above commands reproduced with Docker work.

Output of podman version:

Client:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.17
Built:        Mon Aug 30 12:15:26 2021
OS/Arch:      darwin/amd64

Server:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.16.6
Built:        Mon Aug 30 13:46:36 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.29-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: localhost
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.13.13-200.fc34.x86_64
  linkmode: dynamic
  memFree: 126861312
  memTotal: 2061852672
  ociRuntime:
    name: crun
    package: crun-1.0-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.0
      commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc34.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 0
  swapTotal: 0
  uptime: 50m 39.36s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 3
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 3
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 1630356396
  BuiltTime: Mon Aug 30 20:46:36 2021
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.3.1

Package info (e.g. output of rpm -q podman or apt list podman):

> brew info podman
podman: stable 3.3.1 (bottled), HEAD
<snip>

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes (3.3.1 seems like the latest version now)

Additional environment details (AWS, VirtualBox, physical, etc.):

Running under macOS Big Sur 11.5.2

[Luap99]

Thanks for the issue. Fixing this is unfortunately not as easy as I would like.
We have to drop the host ip part of the port mapping on the VM side. At the same time, podman-machine-cni has to get the correct host ip and send it to gvproxy (see containers/podman-machine-cni#7).

[simnalamburt]

FYI, podman will use CNI as a default rootless_networking method in macOS from the next release.

• Switch default Rootless Networking to "CNI" for OSX common#754
• bump c/common to v0.44.0 #11552

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@Luap99 @mheon @baude is this still an issue?

[Luap99]

Yes I started to work on this but other things have higher priorities.

[rhatdan]

No problem

[convexset]

Would this be the same problem as the port mapping for pods happening inside the podman machine VM instead of on the host on OS X?

See below:

❯ podman -v
podman version 3.4.1

❯ uname -a
Darwin hostname-here 21.1.0 Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:23 PDT 2021; root:xnu-8019.41.5~1/RELEASE_X86_64 x86_64

Creating a container on the host. Port mapping is fine:

❯ podman run -d -p 8080:80 --name myapp docker.io/library/nginx
7f6fd2903c28bf187dc4e88f9eceedafff5315b48a972999236ab35570f26d0c

❯ curl localhost:8080
<h1>Welcome to nginx!</h1>

❯ podman rm -f $(podman ps -a --quiet)
7f6fd2903c28

Creating a pod on the host. Port mapping is messed up (accessible from within the podman machine VM):

❯ podman pod create --name mypod -p 8080:80
307e963bbfec30d029c0ce7c0292b89c445956231e02a9b3f931dfe202a8cb9d

❯ podman create --pod mypod --name myapp docker.io/library/nginx
975a7a252cc64ef7e9d35ec0a67daaa2d2099d9e29476d1b0d7c64b052f6ae93

❯ podman pod start mypod
307e963bbfec30d029c0ce7c0292b89c445956231e02a9b3f931dfe202a8cb9d

❯ curl localhost:8080
curl: (7) Failed to connect to localhost port 8080: Connection refused

❯ podman machine ssh

Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
Fedora CoreOS 34.20211031.2.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/c/server/coreos/

Last login: Sun Nov  7 19:01:45 2021 from 192.168.127.1
[core@localhost ~]$ curl localhost:8080
<h1>Welcome to nginx!</h1>

[Luap99]

@convexset That looks like a separate issue, please open a new one.

[ncdc]

FYI, I think once this is fixed, https://github.com/kubernetes-sigs/kind should be able to work OOTB with podman!

[rhatdan]

Should we close this issue?

[ncdc]

@rhatdan why?

[Luap99]

I am working on it right now

[rhatdan]

@ncdc sorry I read:
FYI, I think once this is fixed ...
But my brain saw
FYI, I think this one is fixed ...

[colinbendell]

Docker supports the notion of -p 127.0.0.1:2000-2100:2000-2100 as a shorthand for publishing a range of ports (also for exposing). I'll open a parallel issue if you don't want to tackle it in this PR, but wanted to draw your attention to the need to support port ranges in the future.

[Luap99]

@colinbendell Port ranges are already supported

[colinbendell]

@colinbendell Port ranges are already supported

Digging deeper, it looks like an EXPOSE problem where port ranges aren't supported. All good; shouldn't be a concern then for this PR.