Document known issue around BTRFS #2584

simon-geard · 2022-01-11T06:58:07Z

Following discussions under issue #2411, documenting problem with finding rootfs device with BTRFS (and maybe other unrecognised filesystems), along with the workaround of adding devices as extra mounts.

Also threw in a quick reminder at the top of the page about how to obtain logs if cluster creation fails.

Following discussions under issue kubernetes-sigs#2411, documenting problem with finding rootfs device with BTRFS (and maybe other unrecognised filesystems), along with the workaround of adding devices as extra mounts. Also threw in a quick reminder at the top of the page about how to obtain logs if cluster creation fails.

k8s-ci-robot · 2022-01-11T06:58:09Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: [email protected]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

linux-foundation-easycla · 2022-01-11T06:58:10Z

The committers are authorized under a signed CLA.

✅ Simon Geard (bcba111)

k8s-ci-robot · 2022-01-11T06:58:15Z

Welcome @simon-geard!

It looks like this is your first PR to kubernetes-sigs/kind 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kind has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2022-01-11T06:58:15Z

Hi @simon-geard. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

site/content/docs/user/known-issues.md

aojea · 2022-01-11T07:44:29Z

site/content/docs/user/known-issues.md

+    - hostPath: /dev/nvme0n1p3
+      containerPath: /dev/nvme0n1p3


should we add some detailed instructions for users on how to obtain the device path?

That's in the following paragraph, where I state "the expected device is named in the error message".

Just in case, in my Fedora 35 it complained about /dev/mapper/luks-903aad3d-... and using hostPath/containerPath: /dev/mapper/luks-903aad3d-.. didn't worked because it is a symlink to /dev/dm-0. Using /dev/dm-0 worked.

ls -l /dev/mapper/ total 0 crw-------. 1 root root 10, 236 ene 11 08:06 control lrwxrwxrwx. 1 root root 7 ene 11 08:06 luks-903aad3d-... -> ../dm-0

cat << EOF | kind create cluster --config=- kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane extraMounts: - hostPath: /dev/dm-0 containerPath: /dev/dm-0 propagation: HostToContainer EOF

So, two variations - if the device cited in the error message is under /dev/mapper, the extra mount should be the /dev/dm-* device it points to; otherwise it should be the device from the message?

I guess it needs to be the real device, not a symlink. So if the error complains about /dev/foo/bar, the config needs to have the real device such as readlink -f /dev/foo/bar

MYDEVICE=$(readlink -f /dev/mapper/luks-903aad3d-...) cat << EOF | kind create cluster --config=- kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane extraMounts: - hostPath: ${MYDEVICE} containerPath: ${MYDEVICE} propagation: HostToContainer EOF

aojea · 2022-01-11T07:46:17Z

/ok-to-test
/assign @BenTheElder @AkihiroSuda
They are the experts of this area

AkihiroSuda · 2022-01-11T07:49:53Z

site/content/docs/user/known-issues.md

+"Failed to start ContainerManager" err="failed to get rootfs info: failed to get device for dir \"/var/lib/kubelet\": could not find device with major: 0, minor: 40 in cached partitions map"
+```
+
+Kubernetes needs access to storage device nodes in order to do some stuff, e.g. tracking free disk space. Therefore, Kind needs to mount the necessary device nodes from the host into the control-plane container — however, it cannot always determine which device Kubernetes requires, since this varies with the host filesystem. For example, Kind doesn't handle BTRFS, which is the default for modern Fedora.


modern Fedora -> modern Fedora on Desktop, IIUC?

Desktop, certainly... I don't know about other variants. But I don't think Fedora itself is relevant to the issue... I mention it only as an example of a common distro where the problem will occur on a stock configuration. I can adjust the wording if you have any suggestions?

AkihiroSuda · 2022-01-11T07:51:22Z

site/content/docs/user/known-issues.md

+      propagation: HostToContainer
+```
+
+The expected device is named in the error message, but will typically be the location where container volumes are stored — for rootless Docker or Podman, this will usually be $HOME.


Does this workaround really work on rootless?
I also guess rootless may not require this workaround, as it uses fuse-overlayfs?

It certainly works for Podman on Fedora - and yes, the workaround is definitely required; otherwise I wouldn't have known this issue existed.

AkihiroSuda · 2022-01-11T07:52:48Z

site/content/docs/user/known-issues.md

+
+Kubernetes needs access to storage device nodes in order to do some stuff, e.g. tracking free disk space. Therefore, Kind needs to mount the necessary device nodes from the host into the control-plane container — however, it cannot always determine which device Kubernetes requires, since this varies with the host filesystem. For example, Kind doesn't handle BTRFS, which is the default for modern Fedora.
+
+This can be worked around by including the necessary device as an extra mount in the cluster configuration file.


I guess, setting $KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER to native or fuse-overlafys may work too?

That I don't know, and can't test right now (it's late at night).

Is that setting documented somewhere? I've been using Kind for about 2 days now, so not an expert by any means.

Ok, did some quick experimenting today. Assuming you're expecting something like:

KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=native kind create cluster --retain

...then neither native nor fuse-overlayfs has any apparent effect. Both fail with the original problem, "stat failed on /dev/nvme0n1p3 with error".

Added a little more explanation about identifying the device to be mounted, since some variations need to deal with symlinked device names.

simon-geard · 2022-02-06T09:01:54Z

A question for whomever it might concern — what's still needed in order to get this documentation change merged?

I believe I've satisfied the feedback around the change itself, but as someone entirely unfamiliar with the processes surrounding this project, it's not clear if there's anything else I need to do? There's an automated comment advising of a build failure, but this does not seem to have an obvious connection to my change...

aojea · 2022-02-06T09:16:40Z

/easycla
/retest

aojea · 2022-02-06T09:20:08Z

A question for whomever it might concern — what's still needed in order to get this documentation change merged?

sorry, we have been busy and some reviews slip

/lgtm
/approve

k8s-ci-robot · 2022-02-06T09:20:23Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AkihiroSuda, aojea, simon-geard

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [aojea]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

aojea · 2022-02-06T10:03:31Z

easycla pass but cla/linuxfoundation doesn't

@mrbobbytables is it ok to override?

BenTheElder · 2022-02-06T10:40:31Z

Both CLA are passing now

simon-geard · 2022-02-06T11:13:26Z

Yeah, the CLA thing was my mistake, I think... I thought I'd done it, but there are actually two of them?

aojea · 2022-02-06T15:39:04Z

/retest

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 11, 2022

k8s-ci-robot requested review from aojea and munnerz January 11, 2022 06:58

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jan 11, 2022

aojea reviewed Jan 11, 2022

View reviewed changes

site/content/docs/user/known-issues.md Show resolved Hide resolved

aojea reviewed Jan 11, 2022

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 11, 2022

AkihiroSuda reviewed Jan 11, 2022

View reviewed changes

AkihiroSuda approved these changes Jan 12, 2022

View reviewed changes

k8s-ci-robot assigned AkihiroSuda Jan 12, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 12, 2022

Improved wording for "Failed to get rootfs info"

749e30e

Added a little more explanation about identifying the device to be mounted, since some variations need to deal with symlinked device names.

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 12, 2022

k8s-ci-robot assigned aojea Feb 6, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 6, 2022

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 6, 2022

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Feb 6, 2022

k8s-ci-robot merged commit 676f31d into kubernetes-sigs:main Feb 6, 2022

This was referenced Apr 18, 2023

Can not create a cluster when running on BTRFS + LUKS encryption #2411

Closed

Control Plane Fails to Start on Fedora 34 Silverblue #2521

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document known issue around BTRFS #2584

Document known issue around BTRFS #2584

simon-geard commented Jan 11, 2022

k8s-ci-robot commented Jan 11, 2022

linux-foundation-easycla bot commented Jan 11, 2022 •

edited

Loading

k8s-ci-robot commented Jan 11, 2022

k8s-ci-robot commented Jan 11, 2022

aojea Jan 11, 2022 •

edited

Loading

simon-geard Jan 11, 2022

e-minguez Jan 11, 2022 •

edited

Loading

simon-geard Jan 12, 2022

e-minguez Jan 12, 2022

aojea commented Jan 11, 2022

AkihiroSuda Jan 11, 2022

simon-geard Jan 11, 2022

AkihiroSuda Jan 11, 2022

simon-geard Jan 11, 2022

AkihiroSuda Jan 11, 2022

simon-geard Jan 11, 2022 •

edited

Loading

simon-geard Jan 12, 2022

simon-geard commented Feb 6, 2022

aojea commented Feb 6, 2022

aojea commented Feb 6, 2022

k8s-ci-robot commented Feb 6, 2022

aojea commented Feb 6, 2022

BenTheElder commented Feb 6, 2022

simon-geard commented Feb 6, 2022

aojea commented Feb 6, 2022


		Kubernetes needs access to storage device nodes in order to do some stuff, e.g. tracking free disk space. Therefore, Kind needs to mount the necessary device nodes from the host into the control-plane container — however, it cannot always determine which device Kubernetes requires, since this varies with the host filesystem. For example, Kind doesn't handle BTRFS, which is the default for modern Fedora.

		This can be worked around by including the necessary device as an extra mount in the cluster configuration file.

Document known issue around BTRFS #2584

Document known issue around BTRFS #2584

Conversation

simon-geard commented Jan 11, 2022

k8s-ci-robot commented Jan 11, 2022

linux-foundation-easycla bot commented Jan 11, 2022 • edited Loading

k8s-ci-robot commented Jan 11, 2022

k8s-ci-robot commented Jan 11, 2022

aojea Jan 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

e-minguez Jan 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aojea commented Jan 11, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simon-geard Jan 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simon-geard commented Feb 6, 2022

aojea commented Feb 6, 2022

aojea commented Feb 6, 2022

k8s-ci-robot commented Feb 6, 2022

aojea commented Feb 6, 2022

BenTheElder commented Feb 6, 2022

simon-geard commented Feb 6, 2022

aojea commented Feb 6, 2022

linux-foundation-easycla bot commented Jan 11, 2022 •

edited

Loading

aojea Jan 11, 2022 •

edited

Loading

e-minguez Jan 11, 2022 •

edited

Loading

simon-geard Jan 11, 2022 •

edited

Loading