kind doesn't work on btrfs

[sudoforge]

What happened:

After installing kind, I ran a basic create cluster, which failed to start the control plane:

➜ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.17.0) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✗ Starting control-plane 🕹️
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged kind-control-plane kubeadm init --ignore-preflight-errors=all --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1

What you expected to happen:

I expected a cluster to be created.

How to reproduce it (as minimally and precisely as possible):

Run kind create cluster.

Anything else we need to know?:

Using -v 1, I'm able to see that the error is related to a failed health check:

➜ kind -v 1 create cluster
Creating cluster "kind" ...
DEBUG: docker/images.go:61] Image: kindest/node:v1.17.0@sha256:9512edae126da271b66b990b6fff768fbb7cd786c7d39e86bdf55906352fdf62 present locally
 ✓ Ensuring node image (kindest/node:v1.17.0) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
⠈⠑ Starting control-plane 🕹️
...
I0318 02:34:54.181668     119 waitcontrolplane.go:80] [wait-control-plane] Waiting for the API server to be healthy
I0318 02:34:54.183466     119 loader.go:375] Config loaded from file:  /etc/kubernetes/admin.conf
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0318 02:34:54.187200     119 round_trippers.go:443] GET https://172.17.0.3:6443/healthz?timeout=10s  in 0 milliseconds
...
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.

Output is truncated for brevity

docker logs kind-control-plane doesn't seem to provide any useful information:

➜ docker logs kind-control-plane
INFO: ensuring we can execute /bin/mount even with userns-remap
INFO: remounting /sys read-only
INFO: making mounts shared
INFO: fix cgroup mounts for all subsystems
INFO: clearing and regenerating /etc/machine-id
Initializing machine ID from random generator.
INFO: faking /sys/class/dmi/id/product_name to be "kind"
INFO: faking /sys/class/dmi/id/product_uuid to be random
INFO: faking /sys/devices/virtual/dmi/id/product_uuid as well
systemd 242 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to Ubuntu 19.10!

Set hostname to <kind-control-plane>.
Failed to bump fs.file-max, ignoring: Invalid argument
Configuration file /kind/systemd/kubelet.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Configuration file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
[  OK  ] Reached target Slices.
[  OK  ] Listening on Journal Socket.
         Starting Apply Kernel Variables...
         Starting Remount Root and Kernel File Systems...
[  OK  ] Listening on Journal Audit Socket.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Reached target Sockets.
         Mounting Kernel Debug File System...
         Starting Create list of re…odes for the current kernel...
         Starting Journal Service...
[  OK  ] Set up automount Arbitrary…s File System Automount Point.
[  OK  ] Reached target Swap.
         Mounting Huge Pages File System...
[  OK  ] Started Dispatch Password …ts to Console Directory Watch.
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Reached target Paths.
[  OK  ] Started Apply Kernel Variables.
[  OK  ] Started Remount Root and Kernel File Systems.
[  OK  ] Started Create list of req… nodes for the current kernel.
         Starting Create System Users...
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Mounted Kernel Debug File System.
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Started Create System Users.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Started Create Static Device Nodes in /dev.
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
[  OK  ] Started Journal Service.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Timers.
[  OK  ] Reached target Basic System.
[  OK  ] Started kubelet: The Kubernetes Node Agent.
         Starting containerd container runtime...
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started containerd container runtime.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Update UTMP about System Runlevel Changes.
[  OK  ] Started Flush Journal to Persistent Storage.

Full debug logs from kind export logs: kind-export-logs.tar.gz

Environment:

• kind version: (use kind version): 0.7.0
• Kubernetes version: (use kubectl version): v1.17.4
• Docker version: (use docker info): 9.03.7-ce
• OS (e.g. from /etc/os-release): Arch Linux
• Kernel version: 5.5.9-arch1-2

[BenTheElder]

thanks for sharing the logs. from the docker info in the logs I see that you're on btfs.
https://kind.sigs.k8s.io/docs/user/known-issues/#docker-on-btrfs-or-zfs

If you can, please attempt with the following config. if it works we'll try to figure out how to include this automatically when appropriate.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraMounts:
  - hostPath: /dev/mapper
    containerPath: /dev/mapper

(place this in a file and pass the file path with kind create cluster --config=path)

[sudoforge]

Hey @BenTheElder, sorry for the delay, I had one of those weeks. Adding /dev/mapper as an extra mount point works! I'll do some additional reading and more extensive testing to ensure that we're not doing anything funky on the filesystem.

[james-callahan]

I had this same issue on my laptop (archlinux; btrfs; using devicemapper). Your workaround worked @BenTheElder.

[smoke]

@BenTheElder your workaround from #1416 (comment) worked for me as well - Gentoo + btrfs
Thank you, big time!!!

[BenTheElder]

going forward this workaround should be automatically employed when running from HEAD

[jdonszelmann]

There seems to be a bug with this (introduced in #1464). In this commit you added

	if mountDevMapper() {
		args = append(args, "--volume", "/dev/mapper", "/dev/mapper")
	}

But I think this has to be

	if mountDevMapper() {
		args = append(args, "--volume", "/dev/mapper:/dev/mapper")
	}

Without this I get

ERROR: failed to create cluster: docker run error: command "docker run --hostname kind-control-plane --name kind-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/modules:ro --detach --tty --label io.x-k8s.kind.cluster=kind --volume /dev/mapper /dev/mapper --publish=127.0.0.1:46023:6443/TCP kindest/node:v1.17.2@sha256:59df31fc61d1da5f46e8a61ef612fa53d3f9140f82419d1ef1a6b9656c6b737c" failed with error: exit status 125

Should I make a seperate bug report for this?

running:

• Docker version 19.03.8-ce, build afacb8b7f0
• Kind version: master (f34ddc3) (kind v0.8.0-alpha go1.14.2 linux/amd64)
• Kernel version: Linux 5.6.5-arch3-1
• OS: Arch linux

To recreate:
run

kind create cluster

or

go run main.go create cluster

[BenTheElder]

thanks for reporting!
yes, a new open bug is helpful.