current kind does not seem to support netavark backed rootless podman(4.x+)

[wherka-ama]

What happened:
Failure during the cluster creation.

$ /usr/bin/kind create cluster --name test3
enabling experimental podman provider
Creating cluster "test3" ...
✓ Ensuring node image (kindest/node:v1.24.0) 🖼
✓ Preparing nodes 📦
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
sigs.k8s.io/kind/pkg/cluster/internal/providers/podman.getSubnets({0x77d9fb?, 0x7a6280?})
sigs.k8s.io/kind/pkg/cluster/internal/providers/podman/provision.go:275 +0x199
sigs.k8s.io/kind/pkg/cluster/internal/providers/podman.getProxyEnv(0x77db16?, {0x77d9fb, 0x4})

What you expected to happen:
Cluster created successfully

How to reproduce it (as minimally and precisely as possible):
The problem can be recreated on any rootless installation with podman backed with netavark. The key here is the output structure produced by podman network inspect kind. In older version of podman, where the CNI plugin was used the subnets where displayed as [n]plugins[n].ipam.ranges[n][m].subnet while in a netavark it's more like [n].subnets[n].subnet

Examples:
*** cni:

[
    {
        "cniVersion": "0.4.0",
        "name": "kind",
        "plugins": [
            {
                "bridge": "cni-podman2",
                "hairpinMode": true,
                "ipMasq": true,
                "ipam": {
                    "ranges": [
                        [
                            {
                                "gateway": "fc00:8866:27d0:bd7e::1",
                                "subnet": "fc00:8866:27d0:bd7e::/64"
                            }
                        ],
                        [
                            {
                                "gateway": "10.89.1.1",
                                "subnet": "10.89.1.0/24"
                            }
                        ]
                    ],
                    "routes": [
                        {
                            "dst": "::/0"
                        },
                        {
                            "dst": "0.0.0.0/0"
                        }
                    ],
                    "type": "host-local"
                },
                "isGateway": true,
                "type": "bridge"
            },
            {
                "capabilities": {
                    "portMappings": true
                },
                "type": "portmap"
            },
            {
                "backend": "",
                "type": "firewall"
            },
            {
                "type": "tuning"
            },
            {
                "capabilities": {
                    "aliases": true
                },
                "domainName": "dns.podman",
                "type": "dnsname"
            }
        ]
    }
]

*** netavark:

[
     {
          "name": "kind",
          "id": "03cd18f9e02e8b4eaff1da01c520534f57c54d366589a3b67a5ad5e9ff4c0a54",
          "driver": "bridge",
          "network_interface": "podman1",
          "created": "2022-08-15T22:15:49.913031878Z",
          "subnets": [
               {
                    "subnet": "fc00:f853:ccd:e793::/64",
                    "gateway": "fc00:f853:ccd:e793::1"
               },
               {
                    "subnet": "10.89.0.0/24",
                    "gateway": "10.89.0.1"
               }
          ],
          "ipv6_enabled": true,
          "internal": false,
          "dns_enabled": true,
          "ipam_options": {
               "driver": "host-local"
          }
     }
]

Anything else we need to know?:
I've already modified the podman.getSubnets to be more flexible and to support both layouts. Everything works as expected. I'm happy to propose such implementation as a PR.

Environment:

• kind version: (use kind version):
  0.14.0

• Kubernetes version: (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"clean", BuildDate:"2022-07-13T14:30:46Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}
  Kustomize Version: v4.5.4
  Unable to connect to the server: dial tcp 10.xxx:443: i/o timeout

• Docker version: (use docker info):
  host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:

  • cpuset
  • cpu
  • io
  • memory
  • pids
    cgroupManager: systemd
    cgroupVersion: v2
    conmon:
    package: conmon-2.1.0-1.module+el8.6.0+14877+f643d2d6.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: 87b7a9037cbd1d81288bdf2d6705dfda889f7cf9'
    cpus: 8
    distribution:
    distribution: '"rhel"'
    version: "8.6"
    eventLogger: file
    hostname: xxx
    idMappings:
    gidmap:
    • container_id: 0
      host_id: 895
      size: 1
    • container_id: 1
      host_id: 165536
      size: 65536
      uidmap:
    • container_id: 0
      host_id: 1000
      size: 1
    • container_id: 1
      host_id: 165536
      size: 65536
      kernel: 4.18.0-372.16.1.el8_6.x86_64
      linkmode: dynamic
      logDriver: k8s-file
      memFree: 23087063040
      memTotal: 67530907648
      networkBackend: netavark
      ociRuntime:
      name: runc
      package: runc-1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_64
      path: /usr/bin/runc
      version: |-
      runc version 1.0.3
      spec: 1.0.2-dev
      go: go1.17.7
      libseccomp: 2.5.2
      os: linux
      remoteSocket:
      path: /run/user/1000/podman/podman.sock
      security:
      apparmorEnabled: false
      capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
      rootless: true
      seccompEnabled: true
      seccompProfilePath: /usr/share/containers/seccomp.json
      selinuxEnabled: false
      serviceIsRemote: false
      slirp4netns:
      executable: /usr/bin/slirp4netns
      package: slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6.x86_64
      version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
      swapFree: 1073737728
      swapTotal: 1073737728
      uptime: 41h 21m 53.77s (Approximately 1.71 days)
      plugins:
      log:
  • k8s-file
  • none
  • passthrough
  • journald
    network:
  • bridge
  • macvlan
    volume:
  • local
    registries:
    docker.io:
    Blocked: false
    Insecure: false
    Location: dockerhub.xxx/registry-1-docker-io-remote
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io
    gcr.io:
    Blocked: false
    Insecure: false
    Location: dockerhub.xxx/gcr-io-docker-remote
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: gcr.io
    mcr.microsoft.com:
    Blocked: false
    Insecure: false
    Location: dockerhub.xxx/mcr-microsoft-com-docker-remote
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: mcr.microsoft.com
    quay.io:
    Blocked: false
    Insecure: false
    Location: dockerhub.xxx/docker-quay-io-remote
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: quay.io
    registry.connect.redhat.com:
    Blocked: false
    Insecure: false
    Location: dockerhub.xxx/docker-registry-connect-redhat-com-remote
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: registry.connect.redhat.com
    registry.redhat.io:
    Blocked: false
    Insecure: false
    Location: dockerhub.xxx/docker-registry.access.redhat.com-remote
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: registry.redhat.io
    search:
  • registry.fedoraproject.org
  • registry.access.redhat.com
  • registry.centos.org
  • docker.io
    store:
    configFile: /home/xxx/.config/containers/storage.conf
    containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
    graphDriverName: overlay
    graphOptions: {}
    graphRoot: /home/xxx/.local/share/containers/storage
    graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
    imageCopyTmpDir: /var/tmp
    imageStore:
    number: 2
    runRoot: /run/user/1000/containers
    volumePath: /home/xxx/.local/share/containers/storage/volumes
    version:
    APIVersion: 4.0.2
    Built: 1650363392
    BuiltTime: Tue Apr 19 10:16:32 2022
    GitCommit: ""
    GoVersion: go1.17.7
    OsArch: linux/amd64
    Version: 4.0.2

• OS (e.g. from /etc/os-release):
  NAME="Red Hat Enterprise Linux"
  VERSION="8.6 (Ootpa)"
  ID="rhel"
  ID_LIKE="fedora"
  VERSION_ID="8.6"
  PLATFORM_ID="platform:el8"

[aojea]

@rhatdan @mheon what is the plan here with the network or who can point me to it?

I'd like to have a better understanding of the whole picture before start adding more if to the podman provider ;)

[rhatdan]

@mheon @baud @Luap99 @flouthoc PTAL

[mheon]

Are you sure this is Netavark-specific? My understanding is that Podman 4.0 uses the new inspect format for every network.

Basically, everything pre-4.0 did not bother to format the output of inspect at all, so we just print the literal CNI config file. 4.0 and up parse the network's information and output Docker-compatible inspect JSON. This is a breaking change, hence the major release bump (among other things).

We do not foresee the format changing any more, we're satisfied with where 4.0 left us.

[wherka-ama]

Are you sure this is Netavark-specific? My understanding is that Podman 4.0 uses the new inspect format for every network.

Basically, everything pre-4.0 did not bother to format the output of inspect at all, so we just print the literal CNI config file. 4.0 and up parse the network's information and output Docker-compatible inspect JSON. This is a breaking change, hence the major release bump (among other things).

We do not foresee the format changing any more, we're satisfied with where 4.0 left us.

Thanks a lot for clarification @mheon ! It makes sense indeed. So that's a generic change related to a v4.x bump and the network inspect layout.

@aojea : in this light I'd say my fix with covering both layouts(pre v4.x and the current) makes a lot of sense. It is backwards compatible, so we should not upset anyone here.

[aojea]

4.0 and up parse the network's information and output Docker-compatible inspect JSON. This is a breaking change, hence the major release bump (among other things).

awesome

@aojea : in this light I'd say my fix with covering both layouts(pre v4.x and the current) makes a lot of sense. It is backwards compatible, so we should not upset anyone here

we have already some pre and post 4.0 checks and behavior

kind/pkg/cluster/internal/providers/podman/provider.go

Line 435 in 1e0b57b

if info.Rootless && !v.AtLeast(version.MustParseSemantic("4.0.0")) {

[wherka-ama]

we have already some pre and post 4.0 checks and behavior

kind/pkg/cluster/internal/providers/podman/provider.go

Line 435 in 1e0b57b

if info.Rootless && !v.AtLeast(version.MustParseSemantic("4.0.0")) {

@aojea : would you prefer to be explicit in this context or accommodate both layouts transparently and without mentioning the version? Both options are feasible.

[aojea]

@aojea : would you prefer to be explicit in this context or accommodate both layouts transparently and without mentioning the version? Both options are feasible.

I would prefer to be explicit so we can delete the old code once 3.x version is EOL

[BenTheElder]

Thanks everyone, I don't think it should be a problem to support these and the direction to use docker compatible structs makes sense. The patch supported by @wherka-ama looks a great starting point.

I hope someone's already brought this up, but the CNI binaries being removed (?) on upgrade in fedora with existing networks still trying to use CNI seems problematic #2821, is there something we can point users to for this?

I've been slammed switching roles / job and I know Antonio has been busy too, we're clearly behind on these changes on our end.

[BenTheElder]

thanks! #2883

I think after #2874, #2885 we will try to drop v0.15 with k8s v1.25 (which should release next week) including this fix.