[podman machine] Port auto-forwarding does not work in rootless mode and macOS #11396

tnk4on · 2021-09-01T17:47:52Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When I run the podman machine with Podman v3.3.x and gvproxy.
In rootless mode on Linux or macOS, automatic port forwarding by gvproxy does not work when I start the container and expose the port.
In root mode on Linux, automatic port forwarding does work.

Steps to reproduce the issue:

(rootless mode on Linux)

Initialize and start the podman machine.

$ podman machine init; podman machine start

Pull the container image and start it.

$ podman -r pull docker.io/library/nginx
$ podman -r run --rm -d --name nginx -p 8888:80 nginx

Checking the port number, there are no published ports.

$ podman -r ps
CONTAINER ID  IMAGE                           COMMAND               CREATED        STATUS            PORTS                 NAMES
4c9564b5d1aa  docker.io/library/nginx:latest  nginx -g daemon o...  2 minutes ago  Up 2 minutes ago  0.0.0.0:8888->80/tcp  nginx
$ curl http://localhost:8888
curl: (7) Failed to connect to localhost port 8888: Connection refused
$ ss -ltnup | grep 8888

Describe the results you received:

$ podman -r ps
CONTAINER ID  IMAGE                           COMMAND               CREATED        STATUS            PORTS                 NAMES
4c9564b5d1aa  docker.io/library/nginx:latest  nginx -g daemon o...  2 minutes ago  Up 2 minutes ago  0.0.0.0:8888->80/tcp  nginx
$ ss -ltnup | grep 8888
$ curl http://localhost:8888
curl: (7) Failed to connect to localhost port 8888: Connection refused

Describe the results you expected:

Automatic port forwarding works, allowing access to exposed ports.

Additional information you deem important (e.g. issue happens only occasionally):

Automatic port forwarding does not work on macOS as well as rootless mode on Linux.
It works fine in root mode on Linux.

# podman machine init; podman machine start
# podman -r pull docker.io/library/nginx
# podman -r run --rm -d --name nginx -p 8888:80 nginx
# podman -r ps
CONTAINER ID  IMAGE                           COMMAND               CREATED         STATUS             PORTS                 NAMES
b75a5a32c09a  docker.io/library/nginx:latest  nginx -g daemon o...  30 seconds ago  Up 30 seconds ago  0.0.0.0:8888->80/tcp  nginx
# ss -ltnup | grep 8888
tcp   LISTEN 0      4096               *:8888             *:*    users:(("gvproxy",pid=8857,fd=31))
# curl http://localhost:8888
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
</html>

Output of podman version:

$ podman version
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.16.6
Built:        Tue Aug 31 05:46:36 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.29-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: '
  cpus: 2
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: fedora34
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.13.12-200.fc34.x86_64
  linkmode: dynamic
  memFree: 137220096
  memTotal: 4103892992
  ociRuntime:
    name: crun
    package: crun-0.21-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.21
      commit: c4c3cdf2ce408ed44a9e027c618473e6485c635b
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc34.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 4094160896
  swapTotal: 4103073792
  uptime: 25h 24m 58.8s (Approximately 1.04 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.7.1-1.fc34.x86_64
      Version: |-
        fusermount3 version: 3.10.4
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.10.4
        using FUSE kernel interface version 7.31
  graphRoot: /home/user/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 16
  runRoot: /run/user/1000/containers
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 1630356396
  BuiltTime: Tue Aug 31 05:46:36 2021
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.3.1

Package info (e.g. output of rpm -q podman or apt list podman):

$ rpm -q podman
podman-3.3.1-1.fc34.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Fedora release 34 (Thirty Four)
VM on VMware ESXi

The text was updated successfully, but these errors were encountered:

Luap99 · 2021-09-01T18:05:20Z

Can you try running with --network bridge and see if that works

tnk4on · 2021-09-01T18:29:22Z

@Luap99 Wow, It works fine!

(rootless mode on Linux)

$ podman -r run --rm -d --name nginx -p 8888:80 --network bridge nginx
$ ss -ltnup |grep 8888
tcp   LISTEN 0      4096               *:8888             *:*    users:(("gvproxy",pid=9725,fd=17))

(macOS)

$ podman network create nginx
$ podman run --rm -d --name nginx -p 8888:80 --network nginx nginx
$ netstat -an |grep 8888
tcp46      0      0  *.8888                 *.*                    LISTEN

It also works when using --network bridge on macOS.

mheon · 2021-09-01T18:49:26Z

This one's probably me, we should be doing that automatically for machine VMs. Self-assigning.

Luap99 · 2021-09-01T18:53:00Z

@mheon The problem is that we parse this into specgen on the client, so rootless_networking = "cni" must be set on the client I think.

mheon · 2021-09-01T18:54:39Z

Argh. Alright, makes sense. Maybe we should make that the default for OS X and Windows builds of c/common and containers.conf?

Luap99 · 2021-09-01T18:58:01Z

Yes I think that would be the best way to fix it

AverageMarcus · 2021-09-01T18:59:08Z

Just confirming that adding rootless_networking = "cni" under the [containers] section of ~/.config/containers/containers.conf does also fix this on MacOS :)

rhatdan · 2021-09-01T19:14:59Z

@ashley-cui @jnovy PTAL, is this something we do in a brew build somewhere?

ashley-cui · 2021-09-01T19:18:40Z

Probably shouldn't be done in brew, maybe in the our makefile or somewhere in podman?

rhatdan · 2021-09-01T19:20:31Z

I am talking about where the containers.conf comes from that we ship for MAC.

ashley-cui · 2021-09-01T19:31:43Z

Ah, no, the containers.conf isn't touched or installed by homebrew.

mheon · 2021-09-01T19:47:28Z

Yes, this should be compiled-in defaults only

This should better support rootless CNI usescases. Fixes containers/podman#11396 Signed-off-by: Matthew Heon <[email protected]>

baude · 2021-09-01T21:00:23Z

this works in 3.3.1 and later. if you have 3.3.1, it should have installed gvproxy ... then simply create a new network and run the container like you did earlier.

jef · 2021-09-02T20:06:56Z

Just confirming that adding rootless_networking = "cni" under the [containers] section of ~/.config/containers/containers.conf does also fix this on MacOS :)

FWIW, I installed Podman via Homebrew today and I had to add the above to work with forwarding.

Followed the instructions for macOS as well.

mheon · 2021-09-02T20:19:30Z

The next release (probably 3.3.2?) should include the changed default. I don't have a solid date on when that's coming out, probably in the next two weeks.

This should better support rootless CNI usescases. Fixes containers/podman#11396 Signed-off-by: Matthew Heon <[email protected]>

mamuf · 2021-09-15T15:13:59Z

None of the suggested workarounds fixed it for me. I'm on macOS, installed podman 3.3.1 via Homebrew, set the rootless_networking = "cni" in the local containers.conf, even tried to use the --network bridge. Well, it worked for the nginx test, but it doesn't work for my scripts:

podman volume create tpm-local-oradata
podman run \
    --name tpm-local-oradata \
    -v tpm-local-oradata:/opt/oracle/oradata \
    --network bridge \
    "${TPM_ORACLE_IMAGE}"

(The image variable is set before in the script to a correct path to our registry. The whole thing is just to run local Oracle DB for software development, nothing more. There's a second podman run command after that that creates the actual container, but it already fails on this one.)

The podman run command fails with the following error message:

Error: error preparing container 2f349ef29f9c20cbfb67b70503356eee751964bda4b619117c5f083a1f57c49e for attach: error configuring network namespace for container 2f349ef29f9c20cbfb67b70503356eee751964bda4b619117c5f083a1f57c49e: error adding pod tpm-local-oradata_tpm-local-oradata to CNI network "podman": unexpected end of JSON input

Note that it fails even if I don't use the --network option.

mheon · 2021-09-15T15:17:13Z

This is fixed already, but requires a new podman-machine CNI plugin (released upstream, hasn't made its way into the FCOS OS image yet).

mheon · 2021-09-15T15:17:43Z

We're putting together a Podman 3.3.2 later this week that should have all the fixes bundled together and should Just Work.

mamuf · 2021-09-15T15:18:11Z

Ok, thanks. Looking forward to it! :-)

hawkeng · 2021-09-26T23:08:07Z

any updates on this?

mheon · 2021-09-26T23:38:56Z

We decided against 3.3.2, and instead are releasing 3.4.0 on Wednesday. That should have all the bugfixes we've accumulated. 3.4.0-RC2 is presently available, though I don't know if we've built the RC for OS X.

…

On Sun, Sep 26, 2021 at 7:08 PM hawkeng ***@***.***> wrote: any updates on this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11396 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3AOCD53WDEDWCBY6YMDYTUD6RWHANCNFSM5DHAGSJA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

jrwren · 2021-10-01T19:13:42Z

This doesn't work for me. Instead I get this:

$ podman run -dt -p 9999:80 --network bridge alpine_nginx
Error: error configuring network namespace for container 475324f18865edfd5a06a69de07c808f1c68653281fda8975592f92c34a1a832: error adding pod charming_babbage_charming_babbage to CNI network "podman": Post "http://host.crc.testing:7777/services/forwarder/expose": dial tcp: lookup host.crc.testing: no such host

mheon · 2021-10-01T19:46:50Z

Please open a new issue.

HarikrishnanBalagopal · 2021-10-13T15:11:56Z

We decided against 3.3.2, and instead are releasing 3.4.0 on Wednesday. That should have all the bugfixes we've accumulated. 3.4.0-RC2 is presently available, though I don't know if we've built the RC for OS X.
…
On Sun, Sep 26, 2021 at 7:08 PM hawkeng @.***> wrote: any updates on this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11396 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB3AOCD53WDEDWCBY6YMDYTUD6RWHANCNFSM5DHAGSJA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Hi @mheon , the new v3.4.0 release doesn't mention this issue #11396
https://github.com/containers/podman/releases/tag/v3.4.0

mheon · 2021-10-13T15:13:20Z

It should be in the release notes of the containers/common library as the fix was there. It is fixed in 3.4.0.

ocafebabe · 2021-10-26T01:04:02Z

We've been experiencing the same error under WSL2 (Ubuntu) and podman.

ocafebabe · 2021-10-26T01:21:46Z

I've just tried with the development version of podman (3.4.0) and I get the same results under WSL2!

This issue should be reopened as the bug is still present in the latest release...

Luap99 · 2021-10-26T07:59:57Z

WSL2 has nothing to with this, this issue is about podman machine port forwarding.

ocafebabe · 2021-10-26T16:03:01Z

@Luap99 sorry, at first I thought it was the same issue that I'm observing with my application but only when I deploy on podman under WSL2.

fl00r · 2021-10-29T10:39:24Z

I am not sure if this is connected. But I am facing the same issue if I am running a container within a pod

podman pod create --name pgtest -p 5432:5432
podman run --pod pgtest -e POSTGRES_USER=test -e POSTGRES_DB=test --name postgres -d postgres:latest

telnet 0 5432
Trying 0.0.0.0...
telnet: connect to address 0.0.0.0: Connection refused
telnet: Unable to connect to remote host

having --network=bridge fixes the problem

podman pod create --name pgtest -p 5432:5432 --network=bridge
podman run --pod pgtest -e POSTGRES_USER=test -e POSTGRES_DB=test --name postgres -d postgres:latest

telnet 0 5432
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
Connection closed by foreign host.

mheon · 2021-10-29T19:57:42Z

What Podman version?

fl00r · 2021-10-30T10:03:03Z

My bad, sorry

podman --version
podman version 3.4.1```

MacOs 11.6

mheon · 2021-11-01T14:09:17Z

Please open a fresh issue, that version should have all the issues we know about with port forwarding worked out.

openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Sep 1, 2021

mheon self-assigned this Sep 1, 2021

mheon added a commit to mheon/common that referenced this issue Sep 1, 2021

Switch default Rootless Networking to "CNI" for OSX

9b7e72a

This should better support rootless CNI usescases. Fixes containers/podman#11396 Signed-off-by: Matthew Heon <[email protected]>

mheon mentioned this issue Sep 1, 2021

Switch default Rootless Networking to "CNI" for OSX containers/common#754

Merged

mheon added a commit to mheon/common that referenced this issue Sep 1, 2021

Switch default Rootless Networking to "CNI" for OSX

854253d

This should better support rootless CNI usescases. Fixes containers/podman#11396 Signed-off-by: Matthew Heon <[email protected]>

openshift-merge-robot closed this as completed in containers/common#754 Sep 2, 2021

This was referenced Sep 3, 2021

Port forwarding to host OS does not work with podman machine #11429

Closed

macOS / podman-machine based container does not expose ports #11456

Closed

Unable to access httpd container with port forwarding on Intel Mac #11500

Closed

This was referenced Sep 10, 2021

Use "cni" as a default rootless_networking method at least in macOS simnalamburt/podman#4

Open

Upstream podman's Apple Silicon support status simnalamburt/podman#5

Closed

Luap99 pushed a commit to Luap99/common that referenced this issue Sep 14, 2021

Switch default Rootless Networking to "CNI" for OSX

ad7d21a

This should better support rootless CNI usescases. Fixes containers/podman#11396 Signed-off-by: Matthew Heon <[email protected]>

HarikrishnanBalagopal mentioned this issue Sep 29, 2021

UI image does not work with Podman konveyor/move2kube#594

Closed

moinessim mentioned this issue Oct 14, 2021

Post "http://host.crc.testing:7777/services/forwarder/expose": dial tcp: lookup host.crc.testing: no such host #11981

Closed

usury mentioned this issue Oct 17, 2021

No Connectivity to Published Ports for Containers on Rootless User-Defined CNI Network #12006

Closed

christopherchong94 mentioned this issue Nov 10, 2021

[podman machine] Port auto-forwarding does not work in macOS when used with podman play kube #12248

Closed

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 21, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[podman machine] Port auto-forwarding does not work in rootless mode and macOS #11396

[podman machine] Port auto-forwarding does not work in rootless mode and macOS #11396

tnk4on commented Sep 1, 2021

Luap99 commented Sep 1, 2021

tnk4on commented Sep 1, 2021 •

edited

Loading

mheon commented Sep 1, 2021

Luap99 commented Sep 1, 2021

mheon commented Sep 1, 2021

Luap99 commented Sep 1, 2021

AverageMarcus commented Sep 1, 2021

rhatdan commented Sep 1, 2021

ashley-cui commented Sep 1, 2021 •

edited

Loading

rhatdan commented Sep 1, 2021

ashley-cui commented Sep 1, 2021

mheon commented Sep 1, 2021

baude commented Sep 1, 2021

jef commented Sep 2, 2021 •

edited

Loading

mheon commented Sep 2, 2021

mamuf commented Sep 15, 2021 •

edited

Loading

mheon commented Sep 15, 2021

mheon commented Sep 15, 2021

mamuf commented Sep 15, 2021

hawkeng commented Sep 26, 2021

mheon commented Sep 26, 2021 via email

jrwren commented Oct 1, 2021

mheon commented Oct 1, 2021

HarikrishnanBalagopal commented Oct 13, 2021 •

edited

Loading

mheon commented Oct 13, 2021

ocafebabe commented Oct 26, 2021

ocafebabe commented Oct 26, 2021 •

edited

Loading

Luap99 commented Oct 26, 2021

ocafebabe commented Oct 26, 2021

fl00r commented Oct 29, 2021 •

edited

Loading

mheon commented Oct 29, 2021

fl00r commented Oct 30, 2021 •

edited

Loading

mheon commented Nov 1, 2021

[podman machine] Port auto-forwarding does not work in rootless mode and macOS #11396

[podman machine] Port auto-forwarding does not work in rootless mode and macOS #11396

Comments

tnk4on commented Sep 1, 2021

Luap99 commented Sep 1, 2021

tnk4on commented Sep 1, 2021 • edited Loading

mheon commented Sep 1, 2021

Luap99 commented Sep 1, 2021

mheon commented Sep 1, 2021

Luap99 commented Sep 1, 2021

AverageMarcus commented Sep 1, 2021

rhatdan commented Sep 1, 2021

ashley-cui commented Sep 1, 2021 • edited Loading

rhatdan commented Sep 1, 2021

ashley-cui commented Sep 1, 2021

mheon commented Sep 1, 2021

baude commented Sep 1, 2021

jef commented Sep 2, 2021 • edited Loading

mheon commented Sep 2, 2021

mamuf commented Sep 15, 2021 • edited Loading

mheon commented Sep 15, 2021

mheon commented Sep 15, 2021

mamuf commented Sep 15, 2021

hawkeng commented Sep 26, 2021

mheon commented Sep 26, 2021 via email

jrwren commented Oct 1, 2021

mheon commented Oct 1, 2021

HarikrishnanBalagopal commented Oct 13, 2021 • edited Loading

mheon commented Oct 13, 2021

ocafebabe commented Oct 26, 2021

ocafebabe commented Oct 26, 2021 • edited Loading

Luap99 commented Oct 26, 2021

ocafebabe commented Oct 26, 2021

fl00r commented Oct 29, 2021 • edited Loading

mheon commented Oct 29, 2021

fl00r commented Oct 30, 2021 • edited Loading

mheon commented Nov 1, 2021

tnk4on commented Sep 1, 2021 •

edited

Loading

ashley-cui commented Sep 1, 2021 •

edited

Loading

jef commented Sep 2, 2021 •

edited

Loading

mamuf commented Sep 15, 2021 •

edited

Loading

HarikrishnanBalagopal commented Oct 13, 2021 •

edited

Loading

ocafebabe commented Oct 26, 2021 •

edited

Loading

fl00r commented Oct 29, 2021 •

edited

Loading

fl00r commented Oct 30, 2021 •

edited

Loading