Pod cannot redirect ports if IP is 0.0.0.0 #7014

yajo · 2020-07-19T05:34:53Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When using a pod to redirect host ports to container ports, if the interface declared in the pod is 0.0.0.0, it will not work. This is unexpected, because it works fine on containers.

Steps to reproduce the issue:

Run this script:

sudo podman pod create --name test1 -p 10001:10001
sudo podman run -d --name test1_traefik --pod test1 traefik --log.level debug --entryPoints.test.address=:10001
sudo podman exec test1_traefik wget localhost:10001 --timeout 5 --tries 1
wget localhost:10001 --timeout 5 --tries 1

sudo podman pod create --name test2 -p 0.0.0.0:10002:10002
sudo podman run -d --name test2_traefik --pod test2 traefik --log.level debug --entryPoints.test.address=:10002
sudo podman exec test2_traefik wget localhost:10002 --timeout 5 --tries 1
wget localhost:10002 --timeout 5 --tries 1

Describe the results you received:

The last command (wget on host machine) works fine on test1 (returns 404 error... but that's OK for us), but fails on test2.

Describe the results you expected:

Both tests should pass equally.

Additional information you deem important (e.g. issue happens only occasionally):

I think this started happening on podman v2.

Output of podman version:

Version:      2.0.2
API Version:  1
Go Version:   go1.14.2
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.18, commit: '
  cpus: 12
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: file
  hostname: ubuntu-vm
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.4.0-40-generic
  linkmode: dynamic
  memFree: 293539840
  memTotal: 4125093888
  ociRuntime:
    name: runc
    package: 'runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: 'runc version spec: 1.0.1-dev'
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.4
      commit: unknown
      libslirp: 4.2.0
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 0
  swapTotal: 0
  uptime: 1h 16m 42.41s (Approximately 0.04 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/yajo/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/yajo/.local/share/containers/storage
  graphStatus: {}
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /home/yajo/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.2
  OsArch: linux/amd64
  Version: 2.0.2

Package info (e.g. output of rpm -q podman or apt list podman):

Listing... Done
podman/unknown,now 2.0.2~2 amd64 [installed]
podman/unknown 2.0.2~2 arm64
podman/unknown 2.0.2~2 armhf
podman/unknown 2.0.2~2 s390x

Additional environment details (AWS, VirtualBox, physical, etc.):
GNOME Boxes amd64 VM with Ubuntu 20.04.

The text was updated successfully, but these errors were encountered:

yajo · 2020-07-19T05:40:56Z

Oops I just realized the same is happening when opening direct ports from containers:

sudo podman run -d --name test3_traefik -p 10003:10003 traefik --log.level debug --entryPoints.test.address=:10003
sudo podman exec test3_traefik wget localhost:10003 --timeout 5 --tries 1
wget localhost:10003 --timeout 5 --tries 1

sudo podman run -d --name test4_traefik -p 0.0.0.0:10004:10004 traefik --log.level debug --entryPoints.test.address=:10004
sudo podman exec test4_traefik wget localhost:10004 --timeout 5 --tries 1
wget localhost:10004 --timeout 5 --tries 1

Results

yajo@ubuntu-vm:~$ sudo podman run -d --name test3_traefik -p 10003:10003 traefik --log.level debug --entryPoints.test.address=:10003
12faa2eeb544679e41f71892e4025802bcdd8cda99ef9d3863068f687d6d1b57
yajo@ubuntu-vm:~$ sudo podman exec test3_traefik wget localhost:10003 --timeout 5 --tries 1
Connecting to localhost:10003 (127.0.0.1:10003)
wget: server returned error: HTTP/1.1 404 Not Found
Error: exec session exited with non-zero exit code 1: OCI runtime error
yajo@ubuntu-vm:~$ wget localhost:10003 --timeout 5 --tries 1
--2020-07-19 05:39:28--  http://localhost:10003/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:10003... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-07-19 05:39:28 ERROR 404: Not Found.

yajo@ubuntu-vm:~$ 
yajo@ubuntu-vm:~$ sudo podman run -d --name test4_traefik -p 0.0.0.0:10004:10004 traefik --log.level debug --entryPoints.test.address=:10004
0a77c0c4bb3ee6018a098daf5dd72b1d881dae9587c71a65b49fd2969593d418
yajo@ubuntu-vm:~$ sudo podman exec test4_traefik wget localhost:10004 --timeout 5 --tries 1
Connecting to localhost:10004 (127.0.0.1:10004)
wget: server returned error: HTTP/1.1 404 Not Found
Error: exec session exited with non-zero exit code 1: OCI runtime error
yajo@ubuntu-vm:~$ wget localhost:10004 --timeout 5 --tries 1
--2020-07-19 05:39:29--  http://localhost:10004/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:10004... connected.
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Giving up.

(Remember 404 = 👍 in this test)

yajo · 2020-07-19T05:48:27Z

I think it might be interesting for you a proof on how this works fine with docker. Just run this test:

sudo docker run -d --name test5_traefik -p 10005:10005 traefik --log.level debug --entryPoints.test.address=:10005
sudo docker exec test5_traefik wget localhost:10005 --timeout 5 --tries 1
wget localhost:10005 --timeout 5 --tries 1

sudo docker run -d --name test6_traefik -p 0.0.0.0:10006:10006 traefik --log.level debug --entryPoints.test.address=:10006
sudo docker exec test6_traefik wget localhost:10006 --timeout 5 --tries 1
wget localhost:10006 --timeout 5 --tries 1

Results

yajo@ubuntu-vm:~$ sudo docker run -d --name test5_traefik -p 10005:10005 traefik --log.level debug --entryPoints.test.address=:10005
Unable to find image 'traefik:latest' locally
latest: Pulling from library/traefik
cbdbe7a5bc2a: Pull complete 
f16506d32a25: Pull complete 
0a75dc5adbaa: Pull complete 
c17b7d897907: Pull complete 
Digest: sha256:e8f625bda602ea5c7d370ff309ece9e124d2c35d68d0bb8f8ac75292836eda68
Status: Downloaded newer image for traefik:latest
25c813c496905995e039a6b2ff8a9dbbe912b3f03c09a8bc15aa23ca438244ca
yajo@ubuntu-vm:~$ sudo docker exec test5_traefik wget localhost:10005 --timeout 5 --tries 1
Connecting to localhost:10005 (127.0.0.1:10005)
wget: server returned error: HTTP/1.1 404 Not Found
yajo@ubuntu-vm:~$ wget localhost:10005 --timeout 5 --tries 1
--2020-07-19 05:46:36--  http://localhost:10005/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:10005... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-07-19 05:46:36 ERROR 404: Not Found.

yajo@ubuntu-vm:~$ 
yajo@ubuntu-vm:~$ sudo docker run -d --name test6_traefik -p 0.0.0.0:10006:10006 traefik --log.level debug --entryPoints.test.address=:10006
47db53c9137377de987b97e6250428910f843207a17d1f2af670f276cd9346e1
yajo@ubuntu-vm:~$ sudo docker exec test6_traefik wget localhost:10006 --timeout 5 --tries 1
Connecting to localhost:10006 (127.0.0.1:10006)
wget: server returned error: HTTP/1.1 404 Not Found
yajo@ubuntu-vm:~$ wget localhost:10006 --timeout 5 --tries 1
--2020-07-19 05:46:37--  http://localhost:10006/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:10006... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-07-19 05:46:37 ERROR 404: Not Found.

(Remember 404 = 👍 in this test)

mheon · 2020-08-03T17:24:40Z

This looks like a bug in CNI - the only different is that we're handing it a host IP of "0.0.0.0" instead of "" (which I imagine should be equivalent?).

From an IPTables perspective, the two generate almost identical rules, with one key exception - destination IP for "0.0.0.0" is literally "0.0.0.0" - whereas, if it's unset, it's "0.0.0.0/0".

At this point, I'm leaning towards detecting 0.0.0.0 and replacing is with "" when parsing port mappings.

Docker and CNI have very different ideas of what 0.0.0.0 means. Docker takes it to be 0.0.0.0/0 - that is, bind to every IPv4 address on the host. CNI (and, thus, root Podman) take it to mean the literal IP 0.0.0.0. Instead, CNI interprets the empty string ("") as "bind to all IPs". We could ask CNI to change, but given this is established behavior, that's unlikely. Instead, let's just catch 0.0.0.0 and turn it into "" when we parse ports. Fixes containers#7014 Signed-off-by: Matthew Heon <[email protected]>

yajo · 2020-08-03T20:45:59Z

Another difference that I'm not sure if is true could be if adding 0.0.0.0 opens the port only on ipv4 networks.

Docker and CNI have very different ideas of what 0.0.0.0 means. Docker takes it to be 0.0.0.0/0 - that is, bind to every IPv4 address on the host. CNI (and, thus, root Podman) take it to mean the literal IP 0.0.0.0. Instead, CNI interprets the empty string ("") as "bind to all IPs". We could ask CNI to change, but given this is established behavior, that's unlikely. Instead, let's just catch 0.0.0.0 and turn it into "" when we parse ports. Fixes containers#7014 Signed-off-by: Matthew Heon <[email protected]>

openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Jul 19, 2020

rhatdan assigned mheon Jul 20, 2020

mheon mentioned this issue Aug 3, 2020

Do not set host IP on ports when 0.0.0.0 requested #7201

Merged

openshift-merge-robot closed this as completed in #7201 Aug 4, 2020

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pod cannot redirect ports if IP is 0.0.0.0 #7014

Pod cannot redirect ports if IP is 0.0.0.0 #7014

yajo commented Jul 19, 2020

yajo commented Jul 19, 2020 •

edited

Loading

yajo commented Jul 19, 2020

mheon commented Aug 3, 2020

yajo commented Aug 3, 2020

Pod cannot redirect ports if IP is 0.0.0.0 #7014

Pod cannot redirect ports if IP is 0.0.0.0 #7014

Comments

yajo commented Jul 19, 2020

yajo commented Jul 19, 2020 • edited Loading

yajo commented Jul 19, 2020

mheon commented Aug 3, 2020

yajo commented Aug 3, 2020

yajo commented Jul 19, 2020 •

edited

Loading