failed to create bridge "cni-podman0" when trying to start a container

[LeoQuote]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

1. start a container eg. podman run docker.io/busybox

Describe the results you received:

container failed to start

# podman run docker.io/busybox
Trying to pull docker.io/busybox...
Getting image source signatures
Copying blob 7c9d20b9b6cd done
Copying config 19485c79a9 done
Writing manifest to image destination
Storing signatures
ERRO[0014] Error adding network: failed to create bridge "cni-podman0": could not add "cni-podman0": operation not supported
ERRO[0014] Error while adding pod to CNI network "podman": failed to create bridge "cni-podman0": could not add "cni-podman0": operation not supported
Error: error configuring network namespace for container 6aaba61c7b57874b46c1b5358edf6c88f921f504a57b3c9ea09b7496eceb535b: failed to create bridge "cni-podman0": could not add "cni-podman0": operation not supported

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

I used the exact configure file as https://github.com/containers/libpod/blob/master/cni/87-podman-bridge.conflist

the log in everything could be helpful ?

Sep 26 19:25:20 some_host [606172.428539] systemd-udevd[14153]: Process 'net.sh cni-podman0 start' failed with exit code 1.
Sep 26 19:25:20 some_host [606172.430647] systemd-udevd[14153]: Process 'net.sh cni-podman0 stop' failed with exit code 1.

I've checked the network as the document in https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#network-plugin-requirements

the net/bridge/bridge-nf-call-iptables is already set to 1.

How can I troubleshoot this problem now?

Output of podman version:

Version:            1.6.0-rc1
RemoteAPI Version:  1
Go Version:         go1.12.9
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.9
  podman version: 1.6.0-rc1
host:
  BuildahVersion: 1.11.2
  Conmon:
    package: Unknown
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.1, commit: 4dc8bcfec41e10ca760c8e2089474c2843dfd066'
  Distribution:
    distribution: gentoo
    version: unknown
  MemFree: 216048619520
  MemTotal: 270106537984
  OCIRuntime:
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8
      commit: 425e105d5a03fabd737a126ad93d62a9eeede87f
      spec: 1.0.1-dev
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 80
  eventlogger: file
  hostname: some_host
  kernel: 4.19.44-gentoo
  os: linux
  rootless: false
  uptime: 168h 19m 38.59s (Approximately 7.00 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.douban
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 33
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 3
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

eix libpod
[I] app-emulation/libpod
     Available versions:  ~1.5.1-r1^t{tbz2} (~)1.6.0_rc1^t{tbz2}[1] {apparmor btrfs ostree +rootless selinux}
     Installed versions:  1.6.0_rc1^t{tbz2}[1](04:20:28 PM 09/23/2019)(rootless -apparmor -btrfs -ostree -selinux)
     Homepage:            https://github.com/containers/libpod/
     Description:         Library and podman tool for running OCI-based containers in Pods

Additional environment details (AWS, VirtualBox, physical, etc.):
physical

The cni-plugin version is 0.8.1

[baude]

just to be sure, what version of the containernetworking-plugins do you have installed?

[LeoQuote]

Thanks for your reply!

I'm not so sure about cni plugin version. All I know is the k8s version is 1.13.1, bootstraped with hyperkube

the cniVersion in 10-calico.conflist is 0.3.0, while the version for 87-podman-bridge.conflist is 0.4.0, could that be a problem?

[LeoQuote]

The cni-plugin version is 0.8.1.

[baude]

@LeoQuote can you paste in the contents of the 87-podman ... file ?

[LeoQuote]

Sure , it’s copied from project example https://github.com/containers/libpod/blob/d76b21e27a7cef26865a05be02632ca76f9cf2ea/cni/87-podman-bridge.conflist

[baude]

that file recently changed, which is why I asked. I wanted to make double sure yours has the updated file.

[LeoQuote]

Thanks for your patience 😂

Here’s the full conflist file:

{
    "cniVersion": "0.4.0",
    "name": "podman",
    "plugins": [
        {
            "type": "bridge",
            "bridge": "cni-podman0",
            "isGateway": true,
            "ipMasq": true,
            "ipam": {
                "type": "host-local",
                "routes": [
                    {
                        "dst": "0.0.0.0/0"
                    }
                ],
                "ranges": [
                    [
                        {
                            "subnet": "10.88.0.0/16",
                            "gateway": "10.88.0.1"
                        }
                    ]
                ]
            }
        },
        {
            "type": "portmap",
            "capabilities": {
                "portMappings": true
            }
        },
        {
            "type": "firewall",
            "backend": "iptables"
        }
    ]
}

———
Seems it’s exactly the same as the file on current master branch.

[LeoQuote]

I found the problem, we're using this node as k8s node at the same time, the calico-node overwrited cni bin files during setup, I fix it by reinstalling cni-plugin. closing the issue

[LeoQuote]

Unfortunately I found reinstalling cni-plugin just fixed some of nodes' problem, some nodes still have the problem above, what can I do to figure out the problem source?

[LeoQuote]

I added cni-podman0 bridge manually and finally fixed it, perhaps we can add this to document?

ip link add cni-podman0 type bridge

[rhatdan]

@LeoQuote Care to open a PR to fix the documentation, or at least document the steps you went through?

[mheon]

I view this as a CNI bug; we should never require manual intervention in bridge creation.

[tuan-hoang1]

I'm hitting this today when running podman as root
Rootless works fine.
Fedora 30, podman 1.6.1

[github-actions]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[vrothberg]

Friendly ping.

[rhatdan]

Has an issue been opened on CNI?

[rhatdan]

Opened this as a CNI issue containernetworking/cni#770

[zmedico]

The cni-plugin version is 0.8.1.

Try cni-plugins-0.8.6 (gentoo ebuilds now require it).

[LeoQuote]

@zmedico this issue didnt show up since then, we're now using libpod 1.9.0 and cni-plugins 0.8.2, will try cni-plugins-0.8.6

Really appreciate your help!

[baude]

closinig as this sounds fixed or distribution specific