podman: IPv6 support

[fansari]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

I have three questions:

I have updated to Fedora 35 with podman 3.4.0 but am still not able to assign static IPv6 addresses to containers.

Did I miss something or is this feature still missing? Maybe yes - I found this:

#7511

So according to this we have to work for podman 4.0.

Also I was not able to reach the container with proxy neighboring. In KVM normally I do something like this (if 2a02.:...:5 is the IPv6 inside the VM and enp6s0 the ethernet NIC on the host):

ip -6 neigh add proxy 2a02.:...:5 dev enp6s0

If the IPv6 address inside the container is given as proxy to the ethernet interface then the container should be reachable.

I can ping this IP from the host where podman is running. But from inside the container I cannot reach external IPv6. Also another host inside the network cannot ping the container.

In a whireshark trace I see several neighbor solicitations (on enp6s0) from the other host in my network from where I run the ping but there is no answer.

Since it does not work for some reason I added a masquerade rule to nftables and this works (I still cannot ping the IPv6 of the container but this is not so important for my scenario. I need to reach external IPv6 from inside the container).

I then tried to proxy the ULA. But the behavior is same. I can ping it from the host but not from another host in the same network.

ip -6 neigh add proxy fdc2:4ba9:85d4:f3c1::2 dev enp6s0 

Routing also looks ok:

fdc2:4ba9:85d4:f3c1::/64 dev cni-podman0 

Stopping nftables also did not help.

Should the neighbor proxy method work for podman?

I used to use "ipMasq": true in my /etc/cni/net.d/87-podman-bridge.conflist.

Today after the update to Fedora 35 my root containers did not start because podman tries to do something with iptables. I don't use iptables but nftables instead. I have removed this option and added an ip6 nat table and put a masquerade rule in the POSTROUTING chain. This works - but I am not sure whether it is the idea of IPv6 that you use NAT on public addresses.

--> creating bind...                                                                                                                                                                                                                          
time="2021-10-16T16:38:07+02:00" level=error msg="error loading cached network config: network \"podman\" not found in CNI cache"                                                                                                             
time="2021-10-16T16:38:07+02:00" level=warning msg="falling back to loading from existing plugins on disk"                                                                                                                                    
time="2021-10-16T16:38:07+02:00" level=error msg="Error tearing down partially created network namespace for container 26a9bd26b000d38f1a3813432052d05e553f062cc6ad2d2633a8412fca965109: error removing pod dns_dns from CNI network \"podman\
": running [/usr/sbin/iptables -t nat -D POSTROUTING -s 10.88.0.2 -j CNI-f17fc921f69382c08deeffb1 -m comment --comment name: \"podman\" id: \"26a9bd26b000d38f1a3813432052d05e553f062cc6ad2d2633a8412fca965109\" --wait]: exit status 2: iptab
les v1.8.7 (nf_tables): Chain 'CNI-f17fc921f69382c08deeffb1' does not exist\nTry `iptables -h' or 'iptables --help' for more information.\n"                                                                                                  
time="2021-10-16T16:38:07+02:00" level=error msg="error starting some container dependencies"                                                                                                                                                 
time="2021-10-16T16:38:07+02:00" level=error msg="\"error configuring network namespace for container 26a9bd26b000d38f1a3813432052d05e553f062cc6ad2d2633a8412fca965109: error adding pod dns_dns to CNI network \\\"podman\\\": failed to list
 chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 1: iptables v1.8.7 (nf_tables): table `nat' is incompatible, use 'nft' tool.\\n\\n\""                                                                                     
Error: error starting some containers: internal libpod error                                                          

Steps to reproduce the issue:

1. see above

Describe the results you received:
see above

Describe the results you expected:
see above

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.4.0                                                                                                                                                                                                                           
API Version:  3.4.0                                                                                                                                                                                                                           
Go Version:   go1.16.8                                                                                                                                                                                                                        
Built:        Thu Sep 30 21:32:16 2021                                                                                                                                                                                                        
OS/Arch:      linux/amd64          

Output of podman info --debug:

host:                                                                                                                                                                                                                                         
  arch: amd64                                                                                                                                                                                                                                 
  buildahVersion: 1.23.1                                                                                                                                                                                                                      
  cgroupControllers:                                                                                                                                                                                                                          
  - cpuset                                                                                                                                                                                                                                    
  - cpu                                                                                                                                                                                                                                       
  - io                                                                                                                                                                                                                                        
  - memory                                                                                                                                                                                                                                    
  - hugetlb                                                                                                                                                                                                                                   
  - pids                                                                                                                                                                                                                                      
  cgroupManager: systemd                                                                                                                                                                                                                      
  cgroupVersion: v2                                                                                                                                                                                                                           
  conmon:                                                                                                                                                                                                                                     
    package: conmon-2.0.30-2.fc35.x86_64                                                                                                                                                                                                      
    path: /usr/bin/conmon                                                                                                                                                                                                                     
    version: 'conmon version 2.0.30, commit: '                                                                                                                                                                                                
  cpus: 8                                                                                                                                                                                                                                     
  distribution:                                                                                                                                                                                                                               
    distribution: fedora                                                                                                                                                                                                                      
    variant: silverblue                                                                                                                                                                                                                       
    version: "35"                                                                                                                                                                                                                             
  eventLogger: journald                                                                                                                                                                                                                       
  hostname: bat.localdomain                                                                                                                                                                                                                   
  idMappings:                                                                                                                                                                                                                                 
    gidmap: null                                                                                                                                                                                                                              
    uidmap: null                                                                                                                                                                                                                              
  kernel: 5.14.10-300.fc35.x86_64                                                                                                                                                                                                             
  linkmode: dynamic                                                                                                                                                                                                                           
  logDriver: journald                                                                                                                                                                                                                         
  memFree: 3165151232                                                                                                                                                                                                                         
  memTotal: 8206110720                                                                                                                                                                                                                        
  ociRuntime:                                                                                                                                                                                                                                 
    name: crun                                                                                                                                                                                                                                
    package: crun-1.1-1.fc35.x86_64                                                                                                                                                                                                           
    path: /usr/bin/crun                                                                                                                                                                                                                       
    version: |-                                                                                                                                                                                                                               
      crun version 1.1                                                                                                                                                                                                                        
      commit: 5b341a145c4f515f96f55e3e7760d1c79ec3cf1f                                                                                                                                                                                        
      spec: 1.0.0                                                                                                                                                                                                                             
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL                                                                                                                                                                             
  os: linux                                                                                                                                                                                                                                   
  remoteSocket:                                                                                                                                                                                                                               
    path: /run/podman/podman.sock                                                                                                                                                                                                             
  security:                                                                                                                                                                                                                                   
    apparmorEnabled: false                                                                                                                                                                                                                    
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT                                                                                 
    rootless: false                                                                                                                                                                                                                           
    seccompEnabled: true                                                                                                                                                                                                                      
    seccompProfilePath: /usr/share/containers/seccomp.json                                                                                                                                                                                    
    selinuxEnabled: true                                                                                                                                                                                                                      
  serviceIsRemote: false 
  slirp4netns:                                                                                                                                                                                                                                
    executable: /usr/bin/slirp4netns                                                                                                                                                                                                          
    package: slirp4netns-1.1.12-2.fc35.x86_64                                                                                                                                                                                                 
    version: |-                                                                                                                                                                                                                               
      slirp4netns version 1.1.12                                                                                                                                                                                                              
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3                                                                                                                                                                                        
      libslirp: 4.6.1                                                                                                                                                                                                                         
      SLIRP_CONFIG_VERSION_MAX: 3                                                                                                                                                                                                             
      libseccomp: 2.5.0                                                                                                                                                                                                                       
  swapFree: 16795033600                                                                                                                                                                                                                       
  swapTotal: 16795033600                                                                                                                                                                                                                      
  uptime: 3h 25m 27.04s (Approximately 0.12 days)                                                                                                                                                                                             
plugins:                                                                                                                                                                                                                                      
  log:                                                                                                                                                                                                                                        
  - k8s-file                                                                                                                                                                                                                                  
  - none                                                                                                                                                                                                                                      
  - journald                                                                                                                                                                                                                                  
  network:                                                                                                                                                                                                                                    
  - bridge                                                                                                                                                                                                                                    
  - macvlan                                                                                                                                                                                                                                   
  volume:                                                                                                                                                                                                                                     
  - local                                                                                                                                                                                                                                     
registries:                                                                                                                                                                                                                                   
  search:                                                                                                                                                                                                                                     
  - registry.fedoraproject.org                                                                                                                                                                                                                
  - registry.access.redhat.com                                                                                                                                                                                                                
  - docker.io                                                                                                                                                                                                                                 
  - quay.io                                                                                                                                                                                                                                   
store:                                                                                                                                                                                                                                        
  configFile: /etc/containers/storage.conf                                                                                                                                                                                                    
  containerStore:                                                                                                                                                                                                                             
    number: 4                                                                                                                                                                                                                                 
    paused: 0                                                                                                                                                                                                                                 
    running: 4                                                                                                                                                                                                                                
    stopped: 0                                                                                                                                                                                                                                
  graphDriverName: overlay                                                                                                                                                                                                                    
  graphOptions:                                                                                                                                                                                                                               
    overlay.mountopt: nodev,metacopy=on                                                                                                                                                                                                       
  graphRoot: /var/mnt/data/containers/storage                                                                                                                                                                                                 
  graphStatus:                                                                                                                                                                                                                                
    Backing Filesystem: xfs                                                                                                                                                                                                                   
    Native Overlay Diff: "false"                                                                                                                                                                                                              
    Supports d_type: "true"                                                                                                                                                                                                                   
    Using metacopy: "true"                                                                                                                                                                                                                    
  imageStore:                                                                                                                                                                                                                                 
    number: 28                                                                                                                                                                                                                                
  runRoot: /var/run/containers/storage                                                                                                                                                                                                        
  volumePath: /var/mnt/data/containers/storage/volumes                                                                                                                                                                                        
version:                                                                                                                                                                                                                                      
  APIVersion: 3.4.0                                                                                                                                                                                                                           
  Built: 1633030336   
BuiltTime: Thu Sep 30 21:32:16 2021                                                                                                                                                                                                         
  GitCommit: ""                                                                                                                                                                                                                               
  GoVersion: go1.16.8                                                                                                                                                                                                                         
  OsArch: linux/amd64                                                                                                                                                                                                                         
  Version: 3.4.0              

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.4.0-1.fc35.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes/No

Additional environment details (AWS, VirtualBox, physical, etc.):

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@Luap99 @mheon @baude PTAL

[mheon]

For the first question, yes, --ipv6 is still not implemented, target remains v4.0 once the network rewrite is complete.

For the second question - we require assignment of a routable subnet for v6 use, with your network infrastructure configured appropriately to route the subnet to the system using Podman. The address you're using is going to make it out to the internet as such, and since you're using a ULA that won't work, your traffic will be dropped as a bogon the moment it hits the outside internet. I am very curious as to how this works with KVM - are they doing address translation to mask the ULA addresses?

For the third question, updates to iptables and CNI should have fixed this.

[fansari]

I have a public IPv6 subnet inside of my container. Otherwise my question would make no sense.

Example:

2: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default                                                                                                                                                  
    link/ether 62:93:a4:07:70:e5 brd ff:ff:ff:ff:ff:ff link-netnsid 0                                                                                                                                                                         
    inet 10.88.0.2/16 brd 10.88.255.255 scope global eth0                                                                                                                                                                                     
       valid_lft forever preferred_lft forever                                                                                                                                                                                                
    inet6 2a02:...:/96 scope global                                                                                                                                                                                     
       valid_lft forever preferred_lft forever                                                                                                                                                                                                
    inet6 fdc2:4ba9:85d4:f3c1::3/64 scope global                                                                                                                                                                                              
       valid_lft forever preferred_lft forever                                                                                                                                                                                                
    inet6 fe80::6093:a4ff:fe07:70e5/64 scope link                                                                                                                                                                                             
       valid_lft forever preferred_lft forever          

For this reason I wrote that I tried to proxy it to my outer network interface:

ip -6 neigh add proxy 2a02.:...:5 dev enp6s0

This is exactly what works with kvm but not with podman so I was wondering why this is the case.

I have tested this again today and it does not work. Even with the proxy without a NAT rule I cannot reach public IPv6.

As I wrote I work with nftables. nftables has replaced iptables and it is not clear to me why docker/podman still prefer iptables. Or maybe I should say: they do not support nftables for whatever reason.. Of course you can write your rules manually and this is what I am doing in general. But regardless of the firewall proxy neighboring should work.

[mheon]

We are in the process of a complete rewrite of our networking backend, so the nftables problem should be resolved in Podman 4.0 or 4.1. I don't think the v6 experience we improve much in 4.0 as we're still trying to get the basics working, but I expect it will be a priority for Podman 4.1.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[baude]

IPv6 is looking good in netavark which should come out in 4.0. Closing

[egberts]

To get rid of that libvirt error, my permanent workaround in Debian 11 (as a host) with libvirtd daemon is to block the loading of iptables-related modules:

Create a file in /etc/modprobe.d/nft-only.conf:

#  Source: https://www.gaelanlloyd.com/blog/migrating-debian-buster-from-iptables-to-nftables/
#
blacklist x_tables
blacklist iptable_nat
blacklist iptable_raw
blacklist iptable_mangle
blacklist iptable_filter
blacklist ip_tables
blacklist ipt_MASQUERADE
blacklist ip6table_nat
blacklist ip6table_raw
blacklist ip6table_mangle
blacklist ip6table_filter
blacklist ip6_tables

libvirtd daemon now starts without any error.

Post-analysis: Apparently, I had iptables module loaded alongside with many nft-related modules; once iptables was gone, the pesky error message went away.