failed to create bridge "cni-podman0" when trying to start a container

[rhatdan]

Copied from containers/podman#4114
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

start a container eg. podman run docker.io/busybox

Describe the results you received:

container failed to start

podman run docker.io/busybox

Trying to pull docker.io/busybox...
Getting image source signatures
Copying blob 7c9d20b9b6cd done
Copying config 19485c79a9 done
Writing manifest to image destination
Storing signatures
ERRO[0014] Error adding network: failed to create bridge "cni-podman0": could not add "cni-podman0": operation not supported
ERRO[0014] Error while adding pod to CNI network "podman": failed to create bridge "cni-podman0": could not add "cni-podman0": operation not supported
Error: error configuring network namespace for container 6aaba61c7b57874b46c1b5358edf6c88f921f504a57b3c9ea09b7496eceb535b: failed to create bridge "cni-podman0": could not add "cni-podman0": operation not supported

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

I used the exact configure file as https://github.com/containers/libpod/blob/master/cni/87-podman-bridge.conflist

the log in everything could be helpful ?

Sep 26 19:25:20 some_host [606172.428539] systemd-udevd[14153]: Process 'net.sh cni-podman0 start' failed with exit code 1.
Sep 26 19:25:20 some_host [606172.430647] systemd-udevd[14153]: Process 'net.sh cni-podman0 stop' failed with exit code 1.

I've checked the network as the document in https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#network-plugin-requirements

the net/bridge/bridge-nf-call-iptables is already set to 1.

How can I troubleshoot this problem now?

Output of podman version:

Version: 1.6.0-rc1
RemoteAPI Version: 1
Go Version: go1.12.9
OS/Arch: linux/amd64

Output of podman info --debug:

debug:
compiler: gc
git commit: ""
go version: go1.12.9
podman version: 1.6.0-rc1
host:
BuildahVersion: 1.11.2
Conmon:
package: Unknown
path: /usr/libexec/podman/conmon
version: 'conmon version 2.0.1, commit: 4dc8bcfec41e10ca760c8e2089474c2843dfd066'
Distribution:
distribution: gentoo
version: unknown
MemFree: 216048619520
MemTotal: 270106537984
OCIRuntime:
package: Unknown
path: /usr/bin/runc
version: |-
runc version 1.0.0-rc8
commit: 425e105d5a03fabd737a126ad93d62a9eeede87f
spec: 1.0.1-dev
SwapFree: 0
SwapTotal: 0
arch: amd64
cpus: 80
eventlogger: file
hostname: some_host
kernel: 4.19.44-gentoo
os: linux
rootless: false
uptime: 168h 19m 38.59s (Approximately 7.00 days)
registries:
blocked: null
insecure: null
search:

• docker.douban
  store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
  number: 33
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
  Backing Filesystem: extfs
  Native Overlay Diff: "true"
  Supports d_type: "true"
  Using metacopy: "false"
  ImageStore:
  number: 3
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

eix libpod
[I] app-emulation/libpod
Available versions: 1.5.1-r1^t{tbz2} ()1.6.0_rc1^t{tbz2}[1] {apparmor btrfs ostree +rootless selinux}
Installed versions: 1.6.0_rc1^t{tbz2}[1](04:20:28 PM 09/23/2019)(rootless -apparmor -btrfs -ostree -selinux)
Homepage: https://github.com/containers/libpod/
Description: Library and podman tool for running OCI-based containers in Pods

Additional environment details (AWS, VirtualBox, physical, etc.):
physical

The cni-plugin version is 0.8.1

[mccv1r0]

Something must not be setup properly. There is nothing strange about what they are trying to do.

$ sudo podman run -it docker.io/busybox sh 
/ # pwd
/
/ # ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
3: eth0@if45: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether 26:44:dc:01:b9:e4 brd ff:ff:ff:ff:ff:ff
    inet 10.89.2.51/24 brd 10.89.2.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 3ffe:ffff:88:1ff::41/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::2444:dcff:fe01:b9e4/64 scope link 
       valid_lft forever preferred_lft forever
/ # exit
$ 

These operation not supported errors would be a good place to start debugging.

ERRO[0014] Error adding network: failed to create bridge "cni-podman0": could not add "cni-podman0": operation not supported
ERRO[0014] Error while adding pod to CNI network "podman": failed to create bridge "cni-podman0": could not add "cni-podman0": operation not supported
Error: error configuring network namespace for container 6aaba61c7b57874b46c1b5358edf6c88f921f504a57b3c9ea09b7496eceb535b: failed to create bridge "cni-podman0": could not add "cni-podman0": operation not supported

[zmedico]

I'm able to reproduce the issue, and this is how strace shows the EOPNOTSUPP error:

sendto(3, {{len=76, type=RTM_NEWLINK, flags=NLM_F_REQUEST|NLM_F_ACK|NLM_F_EXCL|NLM_F_CREATE, seq=1, pid=0}, {ifi_family=AF_UNSPEC, ifi_type=ARPHRD_NETROM, ifi_index=0, ifi_flags=0, ifi_change=0}, [{{nla_len=16, nla_type=IFLA_IFNAME}, "cni-podman0"}, {{nla_len=28, nla_type=IFLA_LINKINFO}, [{{nla_len=10, nla_type=IFLA_INFO_KIND}, "bridge"...}, {{nla_len=12, nla_type=IFLA_INFO_DATA}, {{nla_len=5, nla_type=IFLA_BR_VLAN_FILTERING}, 0}}]}]}, 76, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12 <unfinished ...>
recvfrom(3, {{len=96, type=NLMSG_ERROR, flags=0, seq=1, pid=19683}, {error=-EOPNOTSUPP, msg={{len=76, type=RTM_NEWLINK, flags=NLM_F_REQUEST|NLM_F_ACK|NLM_F_EXCL|NLM_F_CREATE, seq=1, pid=0}, {ifi_family=AF_UNSPEC, ifi_type=ARPHRD_NETROM, ifi_index=0, ifi_flags=0, ifi_change=0}, [{{nla_len=16, nla_type=IFLA_IFNAME}, "cni-podman0"}, {{nla_len=28, nla_type=IFLA_LINKINFO}, [{{nla_len=10, nla_type=IFLA_INFO_KIND}, "bridge"...}, {{nla_len=12, nla_type=IFLA_INFO_DATA}, {{nla_len=5, nla_type=IFLA_BR_VLAN_FILTERING}, 0}}]}]}}}, 65536, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, [112->12]) = 96

Meanwhile, I can avoid the problem by running ip link add cni-podman0 type bridge which succeeds with this strace log:

sendmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=64, type=RTM_NEWLINK, flags=NLM_F_REQUEST|NLM_F_ACK|NLM_F_EXCL|NLM_F_CREATE, seq=1591733768, pid=0}, {ifi_family=AF_UNSPEC, ifi_type=ARPHRD_NETROM, ifi_index=0, ifi_flags=0, ifi_change=0}, [{{nla_len=16, nla_type=IFLA_IFNAME}, "cni-podman0"}, {{nla_len=16, nla_type=IFLA_LINKINFO}, {{nla_len=10, nla_type=IFLA_INFO_KIND}, "bridge"...}}]}, iov_len=64}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 64
recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=NULL, iov_len=0}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_TRUNC}, MSG_PEEK|MSG_TRUNC) = 36
recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=36, type=NLMSG_ERROR, flags=NLM_F_CAPPED, seq=1591733768, pid=20395}, {error=0, msg={len=64, type=RTM_NEWLINK, flags=NLM_F_REQUEST|NLM_F_ACK|NLM_F_EXCL|NLM_F_CREATE, seq=1591733768, pid=0}}}, iov_len=32768}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 36

The successful command has no mention of IFLA_BR_VLAN_FILTERING, which is not enabled in my kernel:

$ zgrep VLAN_FILTERING /proc/config.gz
# CONFIG_BRIDGE_VLAN_FILTERING is not set

[mccv1r0]

That helps.

My reading of https://github.com/containernetworking/plugins/blob/master/plugins/main/bridge/bridge.go#L345 is that unless vlan is in the config, vlan filtering isn't asked for. Maybe bridge plugin assumes it's always in the kernel. I'll look into it.

[zmedico]

Upgrading to cni-plugins-0.8.6 solved it to me, probably thanks to containernetworking/plugins#434.

[rhatdan]

Is this a podman packaging issue? Should podman require a newer version of cni?

[mccv1r0]

Yes, at least cni-plugins-0.8.6 according to above comment