podman and IPv6

[fansari]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

/kind feature

Description
This is not really a bug. I am wondering who to configure IPv6 in podman. I did not find any documentation to this topic so far.

Steps to reproduce the issue:

Describe the results you received:

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

(paste your output here)

Output of podman info --debug:

(paste your output here)

Additional environment details (AWS, VirtualBox, physical, etc.):

[rhatdan]

I believe we are hung up on CNI support?

[Gigadoc2]

I am not sure about that, CNI claims they do support ipv6 and dual-stack, at least for some of their plugins.
For example, the host-local plugin: https://github.com/containernetworking/plugins/tree/master/plugins/ipam/host-local

For that plugin in particular (which is probably a good test case and a real-life usage scenario for single dual-stack hosts), you can pass an arbitrary number of IPs which the plugin will attempt to allocate via the args field.

If I am not mistaken, podman uses CNI_ARGS, which only allows a single ipv4 address to be set for most ipam plugins, and is deprecated. Maybe all that is needed is a switch to using the args field and allowing multiple IPs (of any family) to be passed on the command line?

But even if that is the case, one should keep in mind (and maybe it should be mentioned in the documentation of podman somewhere) that the behaviour of that feature heavily depends on the CNI plugins used, and might not work as expected (or at all).
Still, this is an important feature, and dual-stack with fixed IPs should at least work for the combination of bridge plugin with host-local ipam. FWIW, this is the reason I can't switch to podman yet (my services are not running behind a load-balancer, so I need static IPs of both address families to have them reachable from the outside).

[fansari]

For me the missing of IPv6 seems to make podman unusable for certain scenarios. I have tried to get bind9 running in a podman container in a way I can do address resolution with this- but it behaves strange. Sometimes it works and then it is hanging again. I have no idea what could be the reason so I guessed this might be because of this IPv4.

I built the same scenario with docker and had the same problem. Then I added IPv6 to my docker interface (yes I had to think of neighbor proxy - the usual issue with IPv6) and since I also wanted to use a different ULA network than my standard one I had to add a route to my router (FritzBox). But after this address resolution worked without this constant "hanging".

I think as long as there is no IPv6 support in podman I will have to use both in parallel.

My other containers (I did one for ldap and one for postgresql) work good with podman - I think the difference is that these containers don't need to get data from the "outside" world so they work with IPv4.

Please add IPv6 support to podman. Since docker can do it - why can't podman?

[mheon]

Per CNI, there is partial support for v6 (partial in the sense that port-mapping doesn't work, as it requires NAT, and v6 NAT is horrifying and best avoided at all costs). I believe static v6 IPs are now supported as of the 0.8.x plugin releases, though. We haven't actually looked into this, but in theory, everything except static v6 addresses can be accomplished by modifying config files on the CNI side. It's on our to-do list to look into making those changes default (or perhaps providing a separate dual-stack CNI network that can be made default through config editing?)

[Gigadoc2]

We haven't actually looked into this, but in theory, everything except static v6 addresses can be accomplished by modifying config files on the CNI side.

That is true. However, precisely because you don't use NAT with IPv6, the ability to specify static addresses is paramount.

If you want to run a service accessible over IPv6 on a one-server setup, the easiest and best way is to just define a static IPv6 for the service and point your DNS towards it.
Even if you were to use a dynamic load-balancing service and thus could deal with dynamic addresses, maybe you would like to run that load-balancer with podman?

Per CNI, there is partial support for v6 (partial in the sense that port-mapping doesn't work, as it requires NAT, and v6 NAT is horrifying and best avoided at all costs).

I would go with how docker also does it, and just don't support port-mapping for IPv6. In large-scale environments, you'd use some kind of external loadbalancers with dynamic addresses, and in small-scale environments you just have one container per service anyway. In neither case port-mapping is required.

[fansari]

So far I could not figure out how to do it.

This is my original cni file. What is the cni version? How do I find it?

{
    "cniVersion": "0.3.0",
    "name": "podman",
    "plugins": [
      {
        "type": "bridge",
        "bridge": "cni0",
        "isGateway": true,
        "ipMasq": true,
        "ipam": {
            "type": "host-local",
            "subnet": "10.88.0.0/16",
            "routes": [
                { "dst": "0.0.0.0/0" }
            ]
        }
      },
      {
        "type": "portmap",
        "capabilities": {
          "portMappings": true
        }
      }
    ]
}

I have this installed:

podman-1.4.0-2.fc30.x86_64
containernetworking-plugins-0.7.5-1.fc30.x86_64

Now I tried this:

{
    "cniVersion": "0.3.0",
    "name": "podman",
    "plugins": [
      {
        "type": "bridge",
        "bridge": "cni0",
        "isGateway": true,
        "ipMasq": true,
        "ipam": {
            "type": "host-local",
            "ranges": [
                 [
                     {
                        "subnet": "10.88.0.0/16"
                     }
                 ],
                 [
                     {
                         "subnet": "fdc2:4ba9:85d4:f3c1::/64"
                     }
                 ]
             ],
            "dataDir": "/run/my-orchestrator/container-ipam-state"
        }
      }
    ]
}

Trying to create a container gives this error message:

ERRO[0002] Error adding network: failed to set bridge addr: could not set bridge's mac: invalid argument
ERRO[0002] Error while adding pod to CNI network "podman": failed to set bridge addr: could not set bridge's mac: invalid argument
Error: error configuring network namespace for container 2cd6bc458d4114e8758786fcfb77bbfb2dac5d5defa72b7eeead609c95763bc0: failed to set bridge addr: could not set bridge's mac: invalid argument

Perhaps something is wrong with my syntax. I wanted to make a very easy example to start with.

[fansari]

Strange. Today it suddenly worked. I have now changed this a little bit so I also have default routes.

{
    "cniVersion": "0.3.0",
    "name": "podman",
    "plugins": [
      {
        "type": "bridge",
        "bridge": "cni0",
        "isGateway": true,
        "ipMasq": true,
        "ipam": {
            "type": "host-local",
            "ranges": [
                 [
                     {
                        "subnet": "10.88.0.0/16"
                     }
                 ],
                 [
                     {
                         "subnet": "fdc2:4ba9:85d4:f3c1::/64"
                     }
                 ]
             ],
             "routes": [
                   { "dst": "0.0.0.0/0", "gw": "10.88.0.1" },
                   { "dst": "::/0", "gw": "fdc2:4ba9:85d4:f3c1::1" }
            ],
            "dataDir": "/run/my-orchestrator/container-ipam-state"
        }
      }
    ]
}

[fansari]

This solution seems ok - except for one thing: "podman run --ip6" as not implmented according to the man page.

I would need to set a static IPV6 address and add it to neigbour proxy.

Can you please implement this?

What I can say: no more "hanging" in address resolution. Forwarding uses the ULA IP asking my FritzBox. Since this is IPv6 it works without NAT.

Still a little bit tricky: if I restart my firewalld I have no connection anymore. I have to restart the container after the firewalld restart and then it works again.

After stopping and starting my container I see this in the logs. I have put cni0 into the trusted zone. It behaves in the same way regardless whether I use iptables or nftables as firewalld backend.

Jun 17 21:27:31 bat.localdomain firewalld[28682]: ERROR: UNKNOWN_SOURCE: '10.88.0.2/32' is not in any zone
Jun 17 21:27:31 bat.localdomain firewalld[28682]: ERROR: UNKNOWN_SOURCE: 'fdc2:4ba9:85d4:f3c1::12/128' is not in any zone

[fansari]

Regarding the firewalld issue I have also opened this bug. Maybe I am doing something wrong or it is really a bug.

https://bugzilla.redhat.com/show_bug.cgi?id=1721275

[fansari]

This "UNKNOWN SOURCE" stuff seems clear now. firewalld adds allow rules dynamically when you start a container. This means when you stop it it says "oops" I don't see this IP anymore. This seems to be the background of this error. I only get it when firewalld and the container are both running and I then stop the container.

Unfortunately it also seems that I still have some "hanging" when I use IPv4 for my bind scenario. I am not 100% sure right now - I have to do more tests.

It would really be big help to be able to assign a static IPv6 address to a container.

[mheon]

Current status there:

We need to get CNI v0.8.1 pushed out to Fedora, so we can use it in our CI.

Then we can migrate Podman itself over to using v0.8.1 and the new static IP reservation mechanism. That will allow us to add static IPv6 address support.

[rhatdan]

@mheon what is the latest state on this one?

[mheon]

CNI 0.8.1 is now landed in all relevant distros.

I am anticipating v6 support will be ready for Podman 1.5.1

[rhatdan]

Awesome, great work.

[rhatdan]

Can we close this issue?

[NotAFile]

It does not look like ipv6 is available as of 1.6.2, fedora 31:

podman run -it --rm debian ping 2600::1
connect: Network is unreachable

Is there a newer tracking issue for ipv6 support?

[Gigadoc2]

@NotAFile: IPv6 is available in the sense that you can have containers with one or more ipv6 addresses, and they can talk to stuff (provided that you set up some routing or something, the CNI plugins do not seem to offer ipv6 NAT or anything similar).
Have you changed the default network configuration? Because the default podman CNI config does unfortunately not include any ipv6 configuration, probably because there is no "automatic internet access via NAT" available for ipv6 as it is done for ipv4. Have a look at https://github.com/containernetworking/plugins/tree/master/plugins/ipam/host-local for how to add ipv6 ranges to the default network.

@rhatdan: I believe this issue should not yet be closed, as it is still not possible to specify static ipv6 addresses on the command line:

$ sudo podman run --rm -ti --ip fd76:b436:28bc:100::42 busybox
Error: fd76:b436:28bc:100::42 is not an IPv4 address: invalid argument
$ sudo podman run --rm -ti --ip6 fd76:b436:28bc:100::42 busybox
Error: unknown flag: --ip6

I believe that all that is missing now is to switch from the deprecated CNI_ARGS environment variable (which only allows 1 static ipv4) to the new CNI mechanism to pass static ips (as mentioned in #3245 (comment) and #3245 (comment)) and then (re-)implement --ip and --ip6 to use this new mechanism.

For anyone currently looking to solve this problem of static allocation, I have resorted to generating a network configuration with static IPAM for each container I start.

[mheon]

The two main things we need on the Podman side are:

• Support for static IPv6 addresses
• Support for IPv6 port forwarding

Both have support in CNI and OCICNI, so it's just a matter of plumbing them through from the frontend.

Unfortunately, the matter of enabling IPv6 by default in our CNI configs is substantially more difficult. We may require users to manually add a routed prefix to their CNI networks if they want to enable v6.

[NotAFile]

In the CNI plugin linked, I can also find the dhcp method, which uses an external dhcp server. Given that, shouldn't it also be possible to support external slaac or dhcpv6 for ipv6?

Some googling around reveals lots of arguments, it seems ipv6 does not really cleanly apply to the docker model. However, I don't think requiring static configuration for ipv6 to work is really an acceptable solution. It seems perfectly reasonable for servers, but I assume many more people use podman for local development, which means it must work from dynamically changing and unpredictable network environments. This is my problem in this case. Some of our infrastructure is ipv6-only, and that can not be accessed from podman builds or scratch containers. I think that whatever solution is chosen, outbound ipv6 connectivity should be there by default without configuration. This might not be possible without v6 NAT.

[mheon]

With IPv4, there's a generally-accepted best solution for internal networks for containers and VMs in the form of NAT. With IPv6, we don't like NAT because it breaks end-to-end reachability, so we don't have such a convenient solution. I recall a discussion in IRC a few weeks ago on IPv6 where user-supplied routed prefixes, prefix delegation, and 6to6 NAT were all suggested, all with pros and cons, and none of them seeming to be universally applicable. I don't think there's an easy solution here, and manual configuration may well be required.

[Gigadoc2]

Though, with IPv4 NAT, you also don't have any end-to-end reachability, so technically NAT66 is not worse than NAT44, right?
I mean the main reason to avoid NAT with IPv6 probably is that it was designed to obsolete the need for it, so one is naturally reluctant to use it again. But for VMs and containers on traveling notebooks I also don't really see how NAT can be avoided...

I'd like to talk (and read!) more about this topic, but it is derailing the issue a bit. @mheon, are there public logs of this IRC channel? I'd really like to read what other people have to say on this topic.

[NotAFile]

I agree that there is a not a universal solution. Static, globally routable addresses and dynamic network configuration are pretty fundamentally at odds. I think defaulting to NAT66 for local development and encouraging static IPv6 or DHCPv6 PD for servers covers both of the common cases very well though.

[Schrottfresse]

The two main things we need on the Podman side are:

* Support for static IPv6 addresses

@mheon Is there any indication when static IPv6 addresses (the --ip6 flag) get implemented?

[mheon]

It's not an immediate priority, but ought to be easy enough to add.

[aleks-mariusz]

@mheon / @rhatdan can i ask why this case is closed? Seems there are still things missing, so can we either re-open this issue (or create a new one with a more specific title about what needs work) please?

My experience is as such: While I've gotten ipv6 generally working only on my host (running CentoOS7), i cannot seem to get this working within a podman-created container still..

I updated my /etc/cni/net.d/87-podman-bridge.conflist with my globally-routable ipv6 netblock and gateway (being an address on my host's outbound interface on the same network). When I start a container (as root), i cannot reach the general internet (dns works at least).. Is there something missing setting up routing? I am able to ping the gateway address inside the container..

Additionally, a really strange side-effect of starting a container is that my ipv6 breaks outside the container (on my host system).. for some reason i start getting ping: sendmsg: Network is unreachable from a continuous ping to ipv6.google.com. I have to systemctl restart network to get networking working again. What might have i broken or what might be broken to cause that?

Lastly, how does all this change when running rootless (what i really want to be using with IPv6).

[mheon]

Rootless IPv6 sounds like a separate issue - that uses an entirely different network backend (slirp4netns vs CNI).

The rest of that sounds like a potential bug with CNI - I'd open an issue about it with more details.

[rhatdan]

@aleks-mariusz Yes I think newer issues are better and easier to consume. Also breaking apart issues from rootless and rootfull networking since as @mheon points out they are very different beasts.

[aleks-mariusz]

The rest of that sounds like a potential bug with CNI - I'd open an issue about it with more details.

Opened #6114 with full gory technical details..