Source IP always 127.0.0.1 in rootless Podman 1.8.0 #5138

basvdlei · 2020-02-09T22:06:10Z

/kind bug

Description

For a rootless container the source IP of incoming packets on a publish port is always 127.0.0.1. Even if the request is made from an external host.

Steps to reproduce the issue:

Start a NGINX container:

machine-1$ podman run -p 8888:80 docker.io/library/nginx:latest

Make a request from another node.

machine-2$ curl http://machine-1:8888

Look at the source ip of the request in NGINX stdout log:

127.0.0.1 - - [09/Feb/2020:21:54:17 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.66.0" "-"

Describe the results you received:

The logged source address is always 127.0.0.1

Describe the results you expected:

The logged source ip address to match the ip of the host the request was coming from.

Additional information you deem important (e.g. issue happens only occasionally):

In Podman 1.7 this worked as expected. And it's probably related to:

Rootless Podman now uses Rootlesskit for port forwarding, which should greatly improve performance and capabilities

Output of podman version:

Version:            1.8.0
RemoteAPI Version:  1
Go Version:         go1.13.6
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.13.6
  podman version: 1.8.0
host:
  BuildahVersion: 1.13.1
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.10-2.fc31.x86_64
    path: /usr/libexec/crio/conmon
    version: 'conmon version 2.0.10, commit: 6b526d9888abb86b9e7de7dfdeec0da98ad32ee0'
  Distribution:
    distribution: fedora
    version: "31"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 239222784
  MemTotal: 16487555072
  OCIRuntime:
    name: crun
    package: crun-0.12.1-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.12.1
      commit: df5f2b2369b3d9f36d175e1183b26e5cee55dd0a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 8187277312
  SwapTotal: 8329883648
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: prefect
  kernel: 5.4.17-200.fc31.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.0-20.1.dev.gitbbd6f25.fc31.x86_64
    Version: |-
      slirp4netns version 0.4.0-beta.3+dev
      commit: bbd6f25c70d5db2a1cd3bfb0416a8db99a75ed7e
  uptime: 1h 17m 29.71s (Approximately 0.04 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - quay.io
store:
  ConfigFile: /home/bas/.config/containers/storage.conf
  ContainerStore:
    number: 22
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.5-2.fc31.x86_64
      Version: |-
        fusermount3 version: 3.6.2
        fuse-overlayfs: version 0.7.5
        FUSE library version 3.6.2
        using FUSE kernel interface version 7.29
  GraphRoot: /var/home/bas/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 232
  RunRoot: /run/user/1000
  VolumePath: /var/home/bas/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.8.0-2.fc31.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

Silverblue 31.20200209.0 (Workstation Edition)

The text was updated successfully, but these errors were encountered:

mheon · 2020-02-09T22:30:37Z

This may be a side-effect of the swap to RootlessKit for port forwarding - @AkihiroSuda PTAL

AkihiroSuda · 2020-02-10T06:23:56Z

Intentional #4586 (comment)

basvdlei · 2020-02-10T09:19:12Z

Ah, I missed this was an intentional change. While I do see the benefits of the new solution, I'm wondering if we are losing something valuable.

Just for context, this change broke two setups I run with rootless Podman for local development/testing:

Custom server running in a Podman container where nodes from a Vagrant project connect to. This server relies on the source IP of the VM for identification.
Tests setup for an authentication server with ACL's that in include source IP range.

Since AFAIK rootless Podman has no network support, even container connections need to go through exposed ports, effectively making all traffic local. Any use-cases that cover either security or audibility relying on source IP will no longer be possible in rootless Podman.

It can also change the behavior of containers running as root or rootless. Since the incoming connections now arrive in on the loopback interface instead of the container's virtual interface. The process is required to listen on the loopback interface and may have different (security) policies there.

AkihiroSuda · 2020-02-10T09:42:19Z

Thanks for the context and sorry.
Maybe we should add a new flag like podman run --port-driver=(builtin|slirp4netns). Not sure podman flag or podman run flag.

@giuseppe

giuseppe · 2020-02-10T10:17:36Z

@AkihiroSuda we could probably add another network type:

podman run --network slirp4netns .... <- works as on Podman 1.7
podman run --network rootlesskit .... <- works as on Podman 1.8

What do you think?

AkihiroSuda · 2020-02-10T10:23:40Z

That may give people false sense that slirp4netns was not used at all when --network=rootlesskit

AkihiroSuda · 2020-02-10T10:25:15Z

Also, in future, rootlesskit MAY implement built-in slirp functionality using slirpnetstack as a Go library if there is a demand.

I think --network=rootlesskit should be reserved for that case.

giuseppe · 2020-02-10T11:34:10Z

I think --network=rootlesskit should be reserved for that case.

wouldn't that be an implementation detail for users?

So if that is the case, should we have slirp4netns-rootlesskit? I'd just like to avoid another option tailored for slirp and re-use what we have now

rhatdan · 2020-02-10T13:50:56Z

Yes, I would go along with @giuseppe Just use slirp4netns or rootlesskit. We can document the difference, most users will have no idea the difference,and will just go with whatever the distro or podman chooses as the default.

aleks-mariusz · 2020-04-23T00:48:39Z

i unfortunately ran into this today (using centos 7)..

looked at the --network parameter for podman pod create:

$ podman pod create --help
Create a new empty pod
[...]
      --network string         Connect a container to a network (default "slirp4netns")
[...]

but alas doesn't like it:

$ podman pod create --name=foo --share net --network slirp4netns 
ERRO[0000] Error freeing pod lock after failed creation: no such file or directory 
Error: unable to create pod: error adding Infra Container: cannot join CNI networks if running rootless: invalid argument

@AkihiroSuda @giuseppe @rhatdan so what actually is the right way/options to switch and be able to revert away from the default of using rootlesskit optimization (for example, i need the source address more than i need 27gbps and the 8gbps bandwidth using slirp4netns is more than sufficient for my use-case :-)

i also can't work-around this by using --net=host because the container is configured to bind to port 80, which is privileged, and centos 7 seems to lack the sysctl net.ipv4.ip_unprivileged_port_start :-/

giuseppe · 2020-04-23T07:55:03Z

I am just worried on the configuration side and have to maintain two different backends for minimal differences.

Would you like to open a PR to address it? We need to take pieces from da7595a that were removed and make it configurable.

We accept --net=slirp4netns now. We could extend it to have extra options like --net=slirp4netns:rootlessport and --net=slirp4netns:slirplisten.

What do you think?

aleks-mariusz · 2020-04-23T12:52:18Z

i tried --net=slirp4netns and still see source ip always 127.0.0.1.. :-(

it's my understanding that slirp4netns is the "traditional" way of doing rootless networking, and as of da7595a it defaults to rootlesskitport for rootless port forwarding.. so why does --net=slirp4netns not change anything?

also i'm unclear what the two values on either side of the colon are in slirp4netns:rootlessport vs slirp4netns:slirplisten will end up being when that param handling is updated?

giuseppe · 2020-04-23T13:00:13Z

it's my understanding that slirp4netns is the "traditional" way of doing rootless networking, and as of da7595a it defaults to rootlesskitport for rootless port forwarding.. so why does --net=slirp4netns not change anything?

it won't change anything in the current version since --net=slirp4netns is already the default mode for rootless, and that mode uses rootlesskitport for listening.

We'd need to add back the support for listening through slirp4netns, and I was thinking of a way to expose it to users.

aleks-mariusz · 2020-04-23T14:09:34Z

Ahh thanks for the explanation, so there's two components in play here.. setting up the networking capabilities for the rootless container AND doing port-forwarding.. slirp4netns can do both, but as of da7595a it only does the former, and the latter has been defaulted since 1.8.0 to be done by rootlessport, where the source always is 127.0.0.1 (any reason why?).. is that summary correct/accurate?

As for the PR to re-enable the ability to choose, the colon notation lets you pick the two different parts.. Were you asking for me to create this PR or for someone else (such as @AkihiroSuda ?).. me i can give it the old college-try, but go is not my language of preference (i'm more of a pythonista)

giuseppe · 2020-04-23T15:12:53Z

As for the PR to re-enable the ability to choose, the colon notation lets you pick the two different parts.. Were you asking for me to create this PR or for someone else (such as @AkihiroSuda ?).. me i can give it the old college-try, but go is not my language of preference (i'm more of a pythonista)

at the moment I've no time to look at it, but if you could give it a try and get to a point where we have both methods living in the code base (i.e. recover the existing method from da7595a) I could help with the remaining plumbing and have something faster.

Also please use the v1.9 stable branch as the base for development.

aleks-mariusz · 2020-04-28T18:51:08Z

I've put together PR #6025 to restore this according to your guidance @giuseppe please review and advise if/how it should be changed in order to be merged

(also is it worth re-opening this issue until the PR is accepted?)

…netns As of podman 1.8.0, because of commit da7595a, the default approach of providing port-forwarding in rootless mode has switched (and been hard-coded) to rootlessport, for the purpose of providing super performance. The side-effect of this switch is source within the container to the port-forwarded service always appears to originate from 127.0.0.1 (see issue containers#5138). This commit allows a user to specify if they want to revert to the previous approach of leveraging slirp4netns add_hostfwd() api which, although not as stellar performance, restores usefulness of seeing incoming traffic origin IP addresses. The change should be transparent; when not specified, rootlessport will continue to be used, however if specifying --net slirp4netns:slirplisten the old approach will be used. Note: the above may imply the restored port-forwarding via slirp4netns is not as performant as the new rootlessport approach, however the figures shared in the original commit that introduced rootlessport are as follows: slirp4netns: 8.3 Gbps, RootlessKit: 27.3 Gbps, which are more than sufficient for many use cases where the origin of traffic is more important than limits that cannot be reached due to bottlenecks elsewhere. Signed-off-by: Aleks Mariusz <[email protected]>

disaster123 · 2020-05-19T15:51:44Z

+1 this broke two of my setups as well also I find it very strange to see 127.0.0.1 as source IP for exposed ports

disaster123 · 2020-05-19T17:53:38Z

i think all users expect that the source IP is the real source ip even in rootless mode

mheon · 2020-05-19T18:17:27Z

I think that's a little hyperbolic - while we've seen complaints about the switch to rootlesskit port forwarder, for the most part it doesn't seem to have been a problem for our users.

We have no plans to revert the change of port forwarder, but @aleks-mariusz is working on a PR to allow optionally selecting the old port forwarding method for containers where the new approach does prove problematic.

disaster123 · 2020-05-19T18:21:17Z

may be ;-) at least i'm thinking of log files where remote ip is logged. I already tried the PR but src IP is still 127.0.0.1 - commented in the PR

As of podman 1.8.0, because of commit da7595a, the default approach of providing port-forwarding in rootless mode has switched (and been hard-coded) to rootlessport, for the purpose of providing super performance. The side-effect of this switch is source within the container to the port-forwarded service always appears to originate from 127.0.0.1 (see issue containers#5138). This commit allows a user to specify if they want to revert to the previous approach of leveraging slirp4netns add_hostfwd() api which, although not as stellar performance, restores usefulness of seeing incoming traffic origin IP addresses. The change should be transparent; when not specified, rootlessport will continue to be used, however if specifying --net slirp4netns:slirplisten the old approach will be used. Note: the above may imply the restored port-forwarding via slirp4netns is not as performant as the new rootlessport approach, however the figures shared in the original commit that introduced rootlessport are as follows: slirp4netns: 8.3 Gbps, RootlessKit: 27.3 Gbps, which are more than sufficient for many use cases where the origin of traffic is more important than limits that cannot be reached due to bottlenecks elsewhere. Signed-off-by: Aleks Mariusz <[email protected]>

As of podman 1.8.0, because of commit da7595a, the default approach of providing port-forwarding in rootless mode has switched (and been hard-coded) to rootlessport, for the purpose of providing super performance. The side-effect of this switch is source within the container to the port-forwarded service always appears to originate from 127.0.0.1 (see issue containers#5138). This commit allows a user to specify if they want to revert to the previous approach of leveraging slirp4netns add_hostfwd() api which, although not as stellar performance, restores usefulness of seeing incoming traffic origin IP addresses. The change should be transparent; when not specified, rootlessport will continue to be used, however if specifying --net slirp4netns:slirplisten the old approach will be used. Note: the above may imply the restored port-forwarding via slirp4netns is not as performant as the new rootlessport approach, however the figures shared in the original commit that introduced rootlessport are as follows: slirp4netns: 8.3 Gbps, RootlessKit: 27.3 Gbps, which are more than sufficient for many use cases where the origin of traffic is more important than limits that cannot be reached due to bottlenecks elsewhere. Signed-off-by: Aleks Mariusz <[email protected]> Signed-off-by: Giuseppe Scrivano <[email protected]>

aleks-mariusz · 2020-07-17T11:56:17Z

update: this will be fixed thanks to #6965 being merged, either by building from master or waiting until the next release where this PR is picked up - you'll want to adding parameter --network slirp4netns:port_handler=slirp4netns to your command line.. config-file handling to be handled in a future PR

barosl · 2021-01-01T12:30:43Z

I want to note that this also has a security implication. PostgreSQL (via the library/postgres image) by default regards all local connections as trusted. This means that anyone can connect to the database when Podman is operating under the rootless mode with default options.

AkihiroSuda · 2021-01-01T12:34:27Z

That sounds like an issue on Postgres side. Connections from localhost should never be trusted. https://cathyjf.com/articles/local-servers-can-get-you-compromised

giuseppe · 2021-01-21T14:05:50Z

opened a PR to address this issue: #9052

set the source IP to the slirp4netns address instead of 127.0.0.1 when using rootlesskit. Closes: containers#5138 Signed-off-by: Giuseppe Scrivano <[email protected]>

set the source IP to the slirp4netns address instead of 127.0.0.1 when using rootlesskit. Closes: containers#5138 Fixes RHBZ#1922866 Signed-off-by: Giuseppe Scrivano <[email protected]>

set the source IP to the slirp4netns address instead of 127.0.0.1 when using rootlesskit. Closes: containers#5138 Signed-off-by: Giuseppe Scrivano <[email protected]>

@nhlpl

This requires rebuilding pod. Thanks to @nhlpl for reporting problem with all logins seen from the same IP. Also see: containers/podman#5138 (comment)

- podman configures the source IP of external connections to 10.0.2.100 since podman 3.0.0, hence binding to 127.0.0.1 will stop working. See: CVE-2021-20199 mentioned in https://github.com/containers/podman/releases/tag/v3.0.0 mheon/libpod@5172cfe containers/podman#5138

openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Feb 9, 2020

AkihiroSuda removed the kind/bug Categorizes issue or PR as related to a bug. label Feb 10, 2020

rhatdan closed this as completed Feb 18, 2020

aleks-mariusz mentioned this issue Apr 28, 2020

allow switching of port-forward approaches when rootless using slirp4netns #6025

Closed

AkihiroSuda reopened this Apr 28, 2020

giuseppe mentioned this issue Jul 14, 2020

allow switching of port-forward approaches in rootless/using slirp4netns #6965

Merged

AkihiroSuda closed this as completed Jul 17, 2020

andrewgdunn mentioned this issue Jul 25, 2020

port-forwarded service always appears to originate from 127.0.0.1 rootless-containers/rootlesskit#155

Closed

giuseppe mentioned this issue Jan 20, 2021

port: add ChildIP rootless-containers/rootlesskit#206

Merged

giuseppe reopened this Jan 21, 2021

giuseppe mentioned this issue Jan 21, 2021

rootlessport: set source IP to slirp4netns device #9052

Merged

openshift-merge-robot closed this as completed in #9052 Jan 22, 2021

E314c mentioned this issue Mar 7, 2023

Alternate port_handler that keeps the source ip for user-defined rootless networks #8193

Open

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source IP always 127.0.0.1 in rootless Podman 1.8.0 #5138

Source IP always 127.0.0.1 in rootless Podman 1.8.0 #5138

basvdlei commented Feb 9, 2020

mheon commented Feb 9, 2020

AkihiroSuda commented Feb 10, 2020

basvdlei commented Feb 10, 2020

AkihiroSuda commented Feb 10, 2020

giuseppe commented Feb 10, 2020

AkihiroSuda commented Feb 10, 2020

AkihiroSuda commented Feb 10, 2020

giuseppe commented Feb 10, 2020

rhatdan commented Feb 10, 2020

aleks-mariusz commented Apr 23, 2020 •

edited

Loading

giuseppe commented Apr 23, 2020

aleks-mariusz commented Apr 23, 2020 •

edited

Loading

giuseppe commented Apr 23, 2020

aleks-mariusz commented Apr 23, 2020

giuseppe commented Apr 23, 2020

aleks-mariusz commented Apr 28, 2020

disaster123 commented May 19, 2020

disaster123 commented May 19, 2020

mheon commented May 19, 2020

disaster123 commented May 19, 2020

aleks-mariusz commented Jul 17, 2020

barosl commented Jan 1, 2021 •

edited

Loading

AkihiroSuda commented Jan 1, 2021

giuseppe commented Jan 21, 2021

Source IP always 127.0.0.1 in rootless Podman 1.8.0 #5138

Source IP always 127.0.0.1 in rootless Podman 1.8.0 #5138

Comments

basvdlei commented Feb 9, 2020

mheon commented Feb 9, 2020

AkihiroSuda commented Feb 10, 2020

basvdlei commented Feb 10, 2020

AkihiroSuda commented Feb 10, 2020

giuseppe commented Feb 10, 2020

AkihiroSuda commented Feb 10, 2020

AkihiroSuda commented Feb 10, 2020

giuseppe commented Feb 10, 2020

rhatdan commented Feb 10, 2020

aleks-mariusz commented Apr 23, 2020 • edited Loading

giuseppe commented Apr 23, 2020

aleks-mariusz commented Apr 23, 2020 • edited Loading

giuseppe commented Apr 23, 2020

aleks-mariusz commented Apr 23, 2020

giuseppe commented Apr 23, 2020

aleks-mariusz commented Apr 28, 2020

disaster123 commented May 19, 2020

disaster123 commented May 19, 2020

mheon commented May 19, 2020

disaster123 commented May 19, 2020

aleks-mariusz commented Jul 17, 2020

barosl commented Jan 1, 2021 • edited Loading

AkihiroSuda commented Jan 1, 2021

giuseppe commented Jan 21, 2021

aleks-mariusz commented Apr 23, 2020 •

edited

Loading

aleks-mariusz commented Apr 23, 2020 •

edited

Loading

barosl commented Jan 1, 2021 •

edited

Loading