userspace network between containers

[tmds]

Creating a network requires root.

To avoid that, you can:

• put containers in a pod and communicate via "localhost"
• use the "--network host" network

Both options use ports from the same "host", so it is not possible to run multiple containers that have the same containerport, like when the same image is used to create multiple containers.

Can podman support creating a 'network' between containers that: doesn't require root, allows containers to bind to the same port, and map ports from the containers to the host?

Related slirp4netns issue: rootless-containers/slirp4netns#172

This would be useful also for https://github.com/containers/podman-compose.

[vrothberg]

Thanks for reaching out, @tmds!

I think that what you need is mentioned on podman.io "Container <-> Host Communication".

[tmds]

@vrothberg I'm looking for Container <-> Container communication.

This is what I'd like (similar to rootfull network):

• limit which containers can communicate (no communication with containers which are not part of the 'network')
• don't expose ports publically on the machine (bound to 0.0.0.0)
• port map container port, so the same image can run multiple times
• disallow access to any service running on the host (no --network host).

[vrothberg]

That doesn't work at the moment. Pulling in the slirp4netns specialists: @AkihiroSuda @giuseppe, what do you think?

[mheon]

I think this is something we need to look into in the future; the limitation on what we can do with networking in rootless is a major drawback.

[AkihiroSuda]

Duplicate: #1733

[AkihiroSuda]

Documented how to set up inter-namespace communication: https://github.com/rootless-containers/slirp4netns/blob/master/slirp4netns.1.md#inter-namespace-communication

So, podman could support multi-container networks in two possible ways:

• Nest multiple network namespaces and vEths inside a singleton slirp4netns instance (rootless containers should have dedicated netns&veth under the singleton slirp4netns instance #1733)
• Use vde_plug (or its equivalent) for communication across multiple slirp4netns instances

Rootless Docker supports docker network create using the former one.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[AkihiroSuda]

POC scripts for allowing an equivalent of podman network create with Rootless Podman, by running CNI plugins inside a "sandbox" container that is similar to Kubernetes Pod sandboxes: https://github.com/AkihiroSuda/podman-network-create-for-rootless-podman

Haven't got DNS to work yet, but probably it shouldn't be difficult. (EDIT: now DNS works)

@giuseppe @rhatdan Do you think executing CNI inside a "sandbox" container is the right approach?

[giuseppe]

@giuseppe @rhatdan Do you think executing CNI inside a "sandbox" container is the right approach?

definitely. I've started working on the side on supporting it in rootless podman: https://github.com/giuseppe/libpod/tree/allow-rootless-cni

But before that, we need a new release of iptables with this patch: https://git.netfilter.org/iptables/commit/?id=5105e844c2ff6a3e320bf8cdadc5094d4eaef2b7

[AkihiroSuda]

. I've started working on the side on supporting it in rootless podman: https://github.com/giuseppe/libpod/tree/allow-rootless-cni

The branch doesn't seem to contain sandboxing code at the moment, is there anything I can help?

But before that, we need a new release of iptables with this patch:

Not necessarily if you execute sandbox as a container, or at least unshare mount namespace and mount a writable tmpfs on /run.

[giuseppe]

I like your idea of sandboxing the network and I think it makes sense for something like KIND.

. I've started working on the side on supporting it in rootless podman: https://github.com/giuseppe/libpod/tree/allow-rootless-cni

The branch doesn't seem to contain sandboxing code at the moment, is there anything I can help?

I am still not sure how it should work, so I was just playing around with it. The painful part with having a new network namespace and then create bridges there, is that for each exposed port we need to make sure slirp4netns allocates them and also frees them when the container exits while now we don't risk such leaks if the slirp4netns process is per-container. I am all hears to suggestions on how it should look like :-) Perhaps something similar to what you've done but per pod and embedded in Podman itself? We already have some limitations with what ports are exposed, so maybe per pod it would not be too bad.

But before that, we need a new release of iptables with this patch:

Not necessarily if you execute sandbox as a container, or at least unshare mount namespace and mount a writable tmpfs on /run.

but that would cover the host /run that we would still like to access if a container uses a bind mount from there.

[AkihiroSuda]

Should be per user rather than per pod, so as to support attaching multiple networks.

(For single-network usecase, we can optionally support spawning slirp4netns per network. That would provide slightly better isolation.)

[giuseppe]

Should be per user rather than per pod, so as to support attaching multiple networks.

it would be better, I am just worried it becomes more difficult to keep track of open ports and cleaning them up.

[AkihiroSuda]

As long as the port_handler is set to rootlesskit, pkg/rootlessport is spawned per Pod, so we don't need to keep track of ports.

[AkihiroSuda]

Opened draft PR #7460

[tmds]

@mheon @giuseppe @AkihiroSuda I'd like to try this out.

I've installed podman from copr repo:

podman.x86_64                                          99:2.1.0-1594142770.git54d16f3b.fc32   @copr:copr.fedorainfracloud.org:baude:Upstream_CRIO_Family

I'm not able to create a network:

$ podman network create podman1
Error: cannot run command "podman network create" in rootless mode

What is the recommended way to install a version of podman that supports this feature on Fedora 32?

[gasinvein]

@tmds Podman 2.1.0 is available in updates-testing repo for Fedora 32.

[mheon]

@baude What's going on with the COPR? It's claiming to provide 2.1.0 but it seems like it's a very old version if it hasn't picked this up yet?

[tmds]

Podman 2.1.0 is available in updates-testing repo for Fedora 32.

I have it installed from updates-testing now:

$ dnf list installed podman
Installed Packages
podman.x86_64                                     2:2.1.0-2.fc32                                      @updates-testing

Creating a network doesn't work for me yet, I get a permission denied error:

$ podman network create podman1
WARN[0000] Found deprecated file /home/tmds/.config/containers/libpod.conf, please remove. Use /home/tmds/.config/containers/containers.conf to override defaults. 
WARN[0000] Ignoring libpod.conf EventsLogger setting "journald". Use "/home/tmds/.config/containers/containers.conf" if you want to change this setting and remove libpod.conf files. 
WARN[0000] Found deprecated file /home/tmds/.config/containers/libpod.conf, please remove. Use /home/tmds/.config/containers/containers.conf to override defaults. 
WARN[0000] Ignoring libpod.conf EventsLogger setting "journald". Use "/home/tmds/.config/containers/containers.conf" if you want to change this setting and remove libpod.conf files. 
Error: open /etc/cni/net.d/podman1.conflist: permission denied

@giuseppe @AkihiroSuda is this broken, or am I still missing some bits?

[AkihiroSuda]

If you have /etc/cni strings in /home/tmds/.config/containers/libpod.conf, please try removing them

[tmds]

@AkihiroSuda I commented out cni_config_dir = "/etc/cni/net.d/" in libpod.conf and now it works, thanks!!