From bcba1113d1c5b72c26842f1d699f7a7398cb18cb Mon Sep 17 00:00:00 2001
From: Simon Geard <92153729+simon-geard@users.noreply.github.com>
Date: Tue, 11 Jan 2022 19:56:22 +1300
Subject: [PATCH] Document known issue around BTRFS

Following discussions under issue #2411, documenting problem with finding rootfs
device with BTRFS (and maybe other unrecognised filesystems), along with the
workaround of adding devices as extra mounts.

Also threw in a quick reminder at the top of the page about how to obtain logs
if cluster creation fails.
---
 site/content/docs/user/known-issues.md | 36 ++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/site/content/docs/user/known-issues.md b/site/content/docs/user/known-issues.md
index 578ee8c49a..0754b5fa9e 100644
--- a/site/content/docs/user/known-issues.md
+++ b/site/content/docs/user/known-issues.md
@@ -23,6 +23,7 @@ description: |-
 
 
 ## Contents
+* [Troubleshooting Kind](#troubleshooting-kind)
 * [Kubectl Version Skew](#kubectl-version-skew) (Kubernetes limits supported version skew)
 * [Older Docker Installations](#older-docker-installations) (untested, known to have bugs)
 * [Docker Installed With Snap](#docker-installed-with-snap) (snap filesystem restrictions problematic)
@@ -39,6 +40,12 @@ description: |-
 * [IPv6 Port Forwarding](#ipv6-port-forwarding) (docker doesn't seem to implement this correctly)
 * [Couldn't find an alternative telinit implementation to spawn](#docker-init-daemon-config)
 * [Fedora](#fedora) (various)
+* [Failed to get rootfs info](#failed-to-get-rootfs-info--stat-failed-on-dev)
+
+## Troubleshooting Kind
+
+If the cluster fails to create, try again with the `--retain` option (preserving the failed container),
+then run `kind export logs` to export the logs from the container to a temporary directory on the host.
 
 ## Kubectl Version Skew
 
@@ -306,6 +313,34 @@ your workloads inside the cluster via the nodes IPv6 addresses.
 
 See Previous Discussion: [kind#1326]
 
+## Failed to get rootfs info / "stat failed on /dev/..."
+
+On some systems, creating a cluster times out with these errors in kubelet.log (device varies):
+
+```
+stat failed on /dev/nvme0n1p3 with error: no such file or directory
+"Failed to start ContainerManager" err="failed to get rootfs info: failed to get device for dir \"/var/lib/kubelet\": could not find device with major: 0, minor: 40 in cached partitions map"
+```
+
+Kubernetes needs access to storage device nodes in order to do some stuff, e.g. tracking free disk space. Therefore, Kind needs to mount the necessary device nodes from the host into the control-plane container — however, it cannot always determine which device Kubernetes requires, since this varies with the host filesystem. For example, Kind doesn't handle BTRFS, which is the default for modern Fedora.
+
+This can be worked around by including the necessary device as an extra mount in the cluster configuration file.
+
+```yaml
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+nodes:
+- role: control-plane
+  extraMounts:
+    - hostPath: /dev/nvme0n1p3
+      containerPath: /dev/nvme0n1p3
+      propagation: HostToContainer
+```
+
+The expected device is named in the error message, but will typically be the location where container volumes are stored — for rootless Docker or Podman, this will usually be $HOME.
+
+See Previous Discussion: [kind#2411]
+
 ## Fedora
 
 ### Firewalld
@@ -348,6 +383,7 @@ Although the policy has been fixed in Fedora 34, the fix has not been backported
 [kind#1179]: https://github.com/kubernetes-sigs/kind/issues/1179
 [kind#1326]: https://github.com/kubernetes-sigs/kind/issues/1326
 [kind#2296]: https://github.com/kubernetes-sigs/kind/issues/2296
+[kind#2411]: https://github.com/kubernetes-sigs/kind/issues/2411
 [moby#9939]: https://github.com/moby/moby/issues/9939
 [moby#17666]: https://github.com/moby/moby/issues/17666
 [Docker resource lims]: https://docs.docker.com/docker-for-mac/#advanced