WSL2 ERROR: kind create cluster fails on Starting control-plane

[zdenko-kovac]

What happened:

On WSL2 (Ubuntu-22.04), creation of kind cluster using:

kind create cluster

failed on:

ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged kind-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1

PLease note: It used to work fine last week!

Logs attached: 3165244420.zip

What you expected to happen:

Creating cluster "kind" ...
✓ Ensuring node image (kindest/node:v1.26.3) 🖼
✓ Preparing nodes 📦 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
✓ Joining worker nodes 🚜
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Thanks for using kind! 😊

How to reproduce it (as minimally and precisely as possible):

kind create cluster

Anything else we need to know?:

Environment:

• kind version: (use kind version): kind v0.18.0 go1.20.2 linux/amd64
• Runtime info: (use docker info or podman info):

Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.10.4
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.17.2
Path: /usr/libexec/docker/cli-plugins/docker-compose

Server:
Containers: 65
Running: 3
Paused: 0
Stopped: 62
Images: 101
Server Version: 23.0.3
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 2806fc1057397dbaeefbea0e4e17bddfbd388f38
runc version: v1.1.5-0-gf19387a
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.90.1-microsoft-standard-WSL2
Operating System: Ubuntu 22.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 14
Total Memory: 54.92GiB
Name: w-PF42KZH6
ID: ITXK:IZJO:LRJS:ZZ5M:FYBZ:3ZJN:HRD7:AOLG:O7SA:SFZT:AZHX:7UDC
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Default Address Pools:
Base: 10.180.128.0/17, Size: 22

• OS (e.g. from /etc/os-release):

PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

• Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"clean", BuildDate:"2022-07-13T14:30:46Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6", GitCommit:"ff2c119726cc1f8926fb0585c74b25921e866a28", GitTreeState:"clean", BuildDate:"2023-01-18T19:15:26Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

• Any proxies or other special environment settings?:

[BenTheElder]

PLease note: It used to work fine last week!

And what changed since last week?

will look at the logs in the morning

[zdenko-kovac]

Please note: It used to work fine last week!

And what changed since last week?

Actually, nothing I could be aware of, that's strange ...

[LHozzan]

@BenTheElder Thank you for assistance! We really appreciate it!

And what changed since last week?

Our environment is based on M$ products, so, there is a chance, that some updates was apply to our machines without our assistance and this may have consequence on WSL. But only kind stoped working correctly, so, I fully agree, it is strange ...

[BenTheElder]

Our environment is based on M$ products, so, there is a chance, that some updates was apply to our machines without our assistance and this may have consequence on WSL. But only kind stoped working correctly, so, I fully agree, it is strange ...

Was docker or WSL2 updated?

From the node serial logs:

INFO: ensuring we can execute mount/umount even with userns-remap
chown: cannot access '': No such file or directory

Uh, hmmm. That's an odd but familiar issue

[BenTheElder]

See:
#3160 (comment)
#3160 (comment)

[zdenko-kovac]

From the node serial logs:

INFO: ensuring we can execute mount/umount even with userns-remap
chown: cannot access '': No such file or directory

Uh, hmmm. That's an odd but familiar issue

Thanks,

"default-cgroupns-mode": "host"

in /etc/docker/daemon.json solved the problem, it seems.

After setting that value, restarting docker service, pruning docker system (just to make sure) and restarting WSL, kind cluster can successfully be created :)

$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.26.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/

[BenTheElder]

I was going to link #3165 (comment)

"default-cgroupns-mode": "host"

Oh, we don't have ci for cgroupns-mode:private but I thought we had this working actually ...

#2709 opencontainers/runc#3784 (comment) cc'ing someone with much more expertise @kolyshkin

We could probably detect new-enough docker and add --cgroupns=host to avoid depending on the default, but previously we actually discussed moving towards --cgroupns=private.

[zdenko-kovac]

I was going to link #3165 (comment)

That's, of course, much better solution and it works like a charm even with default --cgroupns=private!

Thanks a lot!

[BenTheElder]

That's, of course, much better solution and it works like a charm even with default --cgroupns=private!

Phew, ok, so it is just a bug in WSL 😅 Thank you for confirming!

So to clarify: WSL2 1.2.0.0 has this and a fix is in 1.2.3/1.2.4 preview?
I think we need to pin an issue outlining the situation and fix.

[zdenko-kovac]

That's, of course, much better solution and it works like a charm even with default --cgroupns=private!

Phew, ok, so it is just a bug in WSL 😅 Thank you for confirming!

So to clarify: WSL2 1.2.0.0 has this and a fix is in 1.2.3/1.2.4 preview? I think we need to pin an issue outlining the situation and fix.

Yes, after I ran wsl --update --pre-release, my WSL was updated from 1.2.0.0 to 1.2.4.0. Then I removed the workaround from /etc/docker/daemon.json, restarted docker and it works.

[BenTheElder]

Thank you! For now I'm pinning #3180 to the top of the issue tracker outlining the situation and linking back to this and #3165 (comment)

[BenTheElder]

Sorry for the noise @kolyshkin

[phbrgnomo]

Here is a solution i found to a similar error while trying to run "kind create cluster"

I am running WSL2, but got the same error the op mentioned.

After some frustration trying to find a solution (and some help from ChatGPT), i was able to solve with the following process:

---

1. Manually pull the Kindest Node image:

docker pull kindest/node:v1.27.3
Manually create the kind-control-plane container:

docker run -it --privileged --name kind-control-plane kindest/node:v1.27.3

Check the container logs:

docker logs kind-control-plane

If the container starts successfully, stop and remove it:

docker stop kind-control-plane
docker rm kind-control-plane

Retry creating the Kind cluster:

kind create cluster

If successful, check the logs:

docker logs kind-control-plane

---

The key point is was to manually create the kind-control-plane container.

Hope this helps.

[BenTheElder]

You should NOT manually create the container and that container is missing a lot of essential options, I'm shocked it is functioning at all. This is not supported.

EDIT: I misread this, the container is being deleted and not used. This workaround is bizarre.

Note: The title here is a symptom, not a root cause. Please file a bug report with all of the requested information in the template and we may be able to diagnose the issue. Windows support is somewhat best effort due to lack of CI (#1529) and lack of local usage by maintainers (we're on mac and Linux personally).

[feynmanliang]

Here is a solution i found to a similar error while trying to run "kind create cluster"

I am running WSL2, but got the same error the op mentioned.

After some frustration trying to find a solution (and some help from ChatGPT), i was able to solve with the following process:

1. Manually pull the Kindest Node image:

docker pull kindest/node:v1.27.3
Manually create the kind-control-plane container:

docker run -it --privileged --name kind-control-plane kindest/node:v1.27.3

Check the container logs:

docker logs kind-control-plane

If the container starts successfully, stop and remove it:

docker stop kind-control-plane
docker rm kind-control-plane

Retry creating the Kind cluster:

kind create cluster

If successful, check the logs:

docker logs kind-control-plane

The key point is was to manually create the kind-control-plane container.

Hope this helps.

This helped resolve my issue on WSL 2.0.14.0 - I have no idea why creating and deleting the kind-control-plane container manually prior to running kind create cluster fixes things...

[afa4]

Here is a solution i found to a similar error while trying to run "kind create cluster"

I am running WSL2, but got the same error the op mentioned.

After some frustration trying to find a solution (and some help from ChatGPT), i was able to solve with the following process:

1. Manually pull the Kindest Node image:

docker pull kindest/node:v1.27.3
Manually create the kind-control-plane container:

docker run -it --privileged --name kind-control-plane kindest/node:v1.27.3

Check the container logs:

docker logs kind-control-plane

If the container starts successfully, stop and remove it:

docker stop kind-control-plane
docker rm kind-control-plane

Retry creating the Kind cluster:

kind create cluster

If successful, check the logs:

docker logs kind-control-plane

The key point is was to manually create the kind-control-plane container.

Hope this helps.

Worked for me - WSL 2.0.14.0

[Nivani]

Here is a solution i found to a similar error while trying to run "kind create cluster"

I am running WSL2, but got the same error the op mentioned.

After some frustration trying to find a solution (and some help from ChatGPT), i was able to solve with the following process:

1. Manually pull the Kindest Node image:

docker pull kindest/node:v1.27.3
Manually create the kind-control-plane container:

docker run -it --privileged --name kind-control-plane kindest/node:v1.27.3

Check the container logs:

docker logs kind-control-plane

If the container starts successfully, stop and remove it:

docker stop kind-control-plane
docker rm kind-control-plane

Retry creating the Kind cluster:

kind create cluster

If successful, check the logs:

docker logs kind-control-plane

The key point is was to manually create the kind-control-plane container.

Hope this helps.

Same here, had to manually pull and run kindest/node:v1.30.0. After that kind cluster create worked just fine.

WSL version: 2.2.4.0