Kind cluster not coming up when the podman is stop and start

[tppalani]

What happened:

What you expected to happen:

When I'm stopping the podman and strating again kind cluster getting error message

$ wsl -l -v
  NAME                      STATE           VERSION
* podman-machine-default    Running         2
  podman-net-usermode       Running         2

Unable to connect to the server: dial tcp 127.0.0.1:60449: connectex: No connection could be made because the target machine actively refused it.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• kind version: (use kind version): kind v0.18.0 go1.20.2 windows/amd64
• Runtime info: (use docker info or podman info):

$ podman info
host:
  arch: amd64
  buildahVersion: 1.33.2
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.8-2.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 98.61
    systemPercent: 0.58
    userPercent: 0.81
  cpus: 12
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "39"
  eventLogger: journald
  freeLocks: 2029
  hostname: LDD4C6G3
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.133.1-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: journald
  memFree: 15601303552
  memTotal: 16566603776
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.9.0-1.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.9.0
    package: netavark-1.9.0-1.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.9.0
  ociRuntime:
    name: crun
    package: crun-1.12-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.12
      commit: ce429cb2e277d001c2179df1ac66a470f00802ae
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231204.gb86afe3-1.fc39.x86_64
    version: |
      pasta 0^20231204.gb86afe3-1.fc39.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 4294967296
  swapTotal: 4294967296
  uptime: 1h 9m 24.00s (Approximately 0.04 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 0
    stopped: 4
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 1081101176832
  graphRootUsed: 10075152384
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 7
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.8.1
  Built: 1701777650
  BuiltTime: Tue Dec  5 17:30:50 2023
  GitCommit: ""
  GoVersion: go1.21.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.8.1

• OS (e.g. from /etc/os-release): git bash
• Kubernetes version: (use kubectl version): Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
• Any proxies or other special environment settings?: windows system connected under the VPN

[tppalani]

net mode enabled, even though we are getting same error

$ wsl -l -v
  NAME                      STATE           VERSION
* podman-machine-default    Running         2
  podman-net-usermode       Running         2

[aojea]

There are a lot of these issues with podman like this #2272 , most probably this is a duplicate of one of those

/area podman
/kind support
/remove-kind bug

[k8s-ci-robot]

@aojea: The label(s) area/podman cannot be applied, because the repository doesn't have them.

In response to this:

There are a lot of these issues with podman like this #2272 , most probably this is a duplicate of one of those

/area podman
/kind support
/remove-kind bug

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tppalani]

There are a lot of these issues with podman like this #2272 , most probably this is a duplicate of one of those

/area podman /kind support /remove-kind bug

No item number # 2272 not helping for me.

$ podman restart kind-control-plane
Error: crun: writing file `devices.allow`: Invalid argument: OCI runtime error

kubectl get nodes
Unable to connect to the server: dial tcp 127.0.0.1:49287: connectex: No connection could be made because the target machine actively refused it.

[BenTheElder]

$ podman restart kind-control-plane
Error: crun: writing file devices.allow: Invalid argument: OCI runtime error

That's a bug in podman/crun, not kind.

Unfortunately we don't have time to debug crun/podman.

I recommend using the docker node backend if you need reboot suport. Podman support is still somewhat limited.

No item number # 2272 not helping for me.

#2272 is about reboot even being supported for podman, which it is not currently.

[kunalgir]

Facing the same issue with Macbook Pro as well. When trying to stop and start the podman and try back connecting to Kind Cluster getting below error message. The connection to the server 127.0.0.1:56218 was refused - did you specify the right host or port? . Please have this issue addressed.

[aojea]

please. see last comment from Ben @kunalgir

[tppalani]

HI @BenTheElder

I can able to resolve the above issue using below command

$ kind get nodes
kind-control-plane

$ podman start kind-control-plane
kind-control-plane