[zfs] kind create cluster using podman provider fails to bootstrap the cluster - failed to mount rootfs component #2868

maciekmm · 2022-08-08T19:57:16Z

What happened:

Creating a cluster with kind create cluster under rootful and rootless podman hangs on ⢎⡠ Starting control-plane 🕹️ and later fails with

[kubelet-check] Initial timeout of 40s passed.
114 round_trippers.go:553] GET https://kind-control-plane:6443/healthz?timeout=10s in 2 milliseconds

~ » sudo kind create cluster                                                                                                                                                                        
enabling experimental podman provider
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.24.0) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
⢎⡠ Starting control-plane 🕹️ ^C

What you expected to happen:

I'd expect the kind create cluster to successfully create a cluster.

How to reproduce it (as minimally and precisely as possible):

On a zfs backed Arch system installation running podman (rootful or rootless) with fuse-overlayfs storage driver execute kind create cluster.

Anything else we need to know?:

Running kind with --retain flag and peeking inside the container we see:

# journalctl -u kubelet.service
Aug 08 19:47:12 kind-control-plane kubelet[223]: E0808 19:47:12.457606     223 kubelet.go:2419] "Error getting node" err="node \"kind-control-plane\" not found"
Aug 08 19:47:12 kind-control-plane kubelet[223]: E0808 19:47:12.524655     223 dns.go:157] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: fc00:f853:ccd:e793::1 10.89.0.1 84.208.20.110"
Aug 08 19:47:12 kind-control-plane kubelet[223]: E0808 19:47:12.558482     223 kubelet.go:2419] "Error getting node" err="node \"kind-control-plane\" not found"
Aug 08 19:47:12 kind-control-plane kubelet[223]: E0808 19:47:12.637581     223 remote_runtime.go:201] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: failed to mount rootfs component &{overlay overlay [index=off workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/295/work upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/295/fs lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/7/fs]}: invalid argument: unknown"
Aug 08 19:47:12 kind-control-plane kubelet[223]: E0808 19:47:12.637641     223 kuberuntime_sandbox.go:70] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: failed to mount rootfs component &{overlay overlay [index=off workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/295/work upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/295/fs lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/7/fs]}: invalid argument: unknown" pod="kube-system/etcd-kind-control-plane"
Aug 08 19:47:12 kind-control-plane kubelet[223]: E0808 19:47:12.637675     223 kuberuntime_manager.go:815] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: failed to mount rootfs component &{overlay overlay [index=off workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/295/work upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/295/fs lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/7/fs]}: invalid argument: unknown" pod="kube-system/etcd-kind-control-plane"
Aug 08 19:47:12 kind-control-plane kubelet[223]: E0808 19:47:12.637744     223 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"etcd-kind-control-plane_kube-system(f3dbec4949f2648b73c0b4e85ed47e2c)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"etcd-kind-control-plane_kube-system(f3dbec4949f2648b73c0b4e85ed47e2c)\\\": rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: failed to mount rootfs component &{overlay overlay [index=off workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/295/work upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/295/fs lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/7/fs]}: invalid argument: unknown\"" pod="kube-system/etcd-kind-control-plane" podUID=f3dbec4949f2648b73c0b4e85ed47e2c
Aug 08 19:47:12 kind-control-plane kubelet[223]: E0808 19:47:12.658726     223 kubelet.go:2419] "Error getting node" err="node \"kind-control-plane\" not found"
Aug 08 19:47:12 kind-control-plane kubelet[223]: E0808 19:47:12.758958     223 kubelet.go:2419] "Error getting node" err="node \"kind-control-plane\" not found"

All logs: logs.tar.gz

The same happens under rootless podman.

Environment:

kind version: (use kind version): kind v0.14.0 go1.18.3 linux/amd64
Kubernetes version: (use kubectl version): node image is 1.24.0
Docker version: (use docker info):

~ » sudo podman info                                                                                                                                                                           1 ↵ 
host:
  arch: amd64
  buildahVersion: 1.26.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.3-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.3, commit: ab52a597278b20173440140cd810dc9fa8785c93'
  cpuUtilization:
    idlePercent: 92.57
    systemPercent: 3.14
    userPercent: 4.28
  cpus: 8
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: <redacted>
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.18.16-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 16413462528
  memTotal: 24849227776
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.5-1
    path: /usr/bin/crun
    version: |-
      crun version 1.5
      commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 24m 20.01s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 381226975232
  graphRootUsed: 34348859392
  graphStatus:
    Backing Filesystem: zfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.1.1
  Built: 1659559968
  BuiltTime: Wed Aug  3 22:52:48 2022
  GitCommit: f73d8f8875c2be7cd2049094c29aff90b1150241-dirty
  GoVersion: go1.19
  Os: linux
  OsArch: linux/amd64
  Version: 4.1.1

OS (e.g. from /etc/os-release):

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
LOGO=archlinux-logo

The text was updated successfully, but these errors were encountered:

BenTheElder · 2022-08-08T20:23:12Z

Are those logs from the rootful or rootless logs?

Is rootful podman running with fuse-overlayfs?

I don't have an environment with zfs x podman, but when we detect zfs inside the kind node we have to switch to the "native" driver, with lower precedence than on rootless (userns remapping) we will need to use fuse-overlayfs / overlayfs depending on the host kernel.

We seem to be runing with overlayfs snapshotter inside the container, it probably needs to be fuse-overlayfs or native.

you can override this with KIND_EXPERIMENTAL_SNAPSHOTTER=native kind create cluster (note: native may have poor performance)

kind/images/base/files/usr/local/bin/entrypoint

Lines 102 to 128 in 3ad5fe9

    
           configure_containerd() { 
        
             local snapshotter=${KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER:-} 
        
             if [[ -n "$userns" ]]; then 
        
               # userns (rootless) configs 
        
               # Adjust oomScoreAdj 
        
               sed -i 's/restrict_oom_score_adj = false/restrict_oom_score_adj = true/' /etc/containerd/config.toml 
        
               # Use fuse-overlayfs if overlayfs is not preferrable: https://github.com/kubernetes-sigs/kind/issues/2275 
        
               if [[ -z "$snapshotter" ]] && ! overlayfs_preferrable; then 
        
                 snapshotter="fuse-overlayfs" 
        
               fi 
        
             else 
        
               # we need to switch to the 'native' snapshotter on zfs 
        
               if [[ -z "$snapshotter" ]] && [[ "$(stat -f -c %T /kind)" == 'zfs' ]]; then 
        
                 snapshotter="native" 
        
               fi 
        
             fi 
        
             if [[ -n "$snapshotter" ]]; then 
        
               echo "INFO: changing snapshotter from \"overlayfs\" to \"$snapshotter\"" 
        
               sed -i "s/snapshotter = \"overlayfs\"/snapshotter = \"$snapshotter\"/" /etc/containerd/config.toml 
        
               if [[ "$snapshotter" = "fuse-overlayfs" ]]; then 
        
                 echo 'INFO: enabling containerd-fuse-overlayfs service' 
        
                 systemctl enable containerd-fuse-overlayfs 
        
               fi 
        
             fi 
        
           }

maciekmm · 2022-08-08T21:01:50Z

The attached logs were from rootful podman.
With or without mount_program set to fuse-overlayfs (I removed it while debugging) the behavior is the same.

Rootful

Running sudo kind create cluster fails 🟥
Running sudo KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=fuse-overlayfs kind create cluster works ✔️
Running sudo KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=native kind create cluster works ✔️

Running stat -f -c %T /kind inside the container returns fuseblk

Rootless

Running kind create cluster fails 🟥
Running KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=fuse-overlayfs kind create cluster works ✔️
Running KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=native kind create cluster works ✔️

So it seems like we are picking the wrong snapshotter.
I tried running some of the commands from overlayfs_prefferable and they seem to succeed under the overlayfs snapshotter:

root@kind-control-plane:/#   tmp=$(mktemp -d)
root@kind-control-plane:/#   mkdir -p "${tmp}/l" "${tmp}/u" "${tmp}/w" "${tmp}/m"
root@kind-control-plane:/# $?
bash: 0: command not found
root@kind-control-plane:/# mount -t overlay -o lowerdir="${tmp}/l,upperdir=${tmp}/u,workdir=${tmp}/w" overlay "${tmp}/m"
root@kind-control-plane:/# $?
bash: 0: command not found

It seems like it tries to use the overlayfs but then the real invocation of it fails with invalid argument.
Not sure how useful that is as I lack some knowledge here.

Thanks for the pointers @BenTheElder . Happy to help with this further.

BenTheElder · 2022-08-09T10:49:31Z

Thanks!

I think we should probably rework the snapshotter selection to be independent of the rest of userns config and always select fuse-overlays when we’re in fuseblk. It seems like a safe bet that we’re in fuse-overlayfs if /kind is on fuseblk

caniko · 2022-08-10T11:24:58Z

I also encountered fails this morning when running both with podman and docker; I am also on Arch

caniko · 2022-08-10T11:52:16Z

@maciekmm could you make a PR for the docs where you provide a guide on how to run kind with rootless podman on Arch? At least explain it here.

Command:

KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=fuse-overlayfs kind create cluster

Error:

ERROR: failed to create cluster: running kind with rootless provider requires setting systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/

BenTheElder · 2022-08-10T17:44:30Z

working on a fix here #2874

BenTheElder · 2022-08-10T18:08:32Z

test image docker.io/bentheelder/kind-node:v1.24.3@sha256:9fdef952a166325f89cae70b7e69767b50b7a7b0ace1f03df362228b82ee34e4 contains the proposed image fix from #2874

use with kind create cluster --image=... (should be compatible with current stable KIND)

maciekmm · 2022-08-10T19:38:33Z

@BenTheElder thanks for working on this.

It seems like the image you have linked does not contain the fixes.

sudo podman ps
CONTAINER ID  IMAGE                                                                                                    COMMAND     CREATED        STATUS            PORTS                      NAMES
6d883a2f508d  docker.io/bentheelder/kind-node@sha256:9fdef952a166325f89cae70b7e69767b50b7a7b0ace1f03df362228b82ee34e4              5 minutes ago  Up 4 minutes ago  127.0.0.1:41301->6443/tcp  kind-control-plane

~ » sudo podman exec -it 6d883a2f508d grep -A30 SNAPSHOTTER /usr/local/bin/entrypoint                                                                                                              
    echo "WARN: UserNS: SELinux might be Enforcing. If you see an error related to overlayfs, try setting \`KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=fuse-overlayfs\` ." >&2
  fi
  return 0
}

configure_containerd() {
  local snapshotter=${KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER:-}
  if [[ -n "$userns" ]]; then
    # userns (rootless) configs

    # Adjust oomScoreAdj
    sed -i 's/restrict_oom_score_adj = false/restrict_oom_score_adj = true/' /etc/containerd/config.toml

    # Use fuse-overlayfs if overlayfs is not preferrable: https://github.com/kubernetes-sigs/kind/issues/2275
    if [[ -z "$snapshotter" ]] && ! overlayfs_preferrable; then
      snapshotter="fuse-overlayfs"
    fi
  else
    # we need to switch to the 'native' snapshotter on zfs
    if [[ -z "$snapshotter" ]] && [[ "$(stat -f -c %T /kind)" == 'zfs' ]]; then
      snapshotter="native"
    fi
  fi
  if [[ -n "$snapshotter" ]]; then
    echo "INFO: changing snapshotter from \"overlayfs\" to \"$snapshotter\""
    sed -i "s/snapshotter = \"overlayfs\"/snapshotter = \"$snapshotter\"/" /etc/containerd/config.toml
    if [[ "$snapshotter" = "fuse-overlayfs" ]]; then
      echo 'INFO: enabling containerd-fuse-overlayfs service'
      systemctl enable containerd-fuse-overlayfs
    fi
  fi
}

configure_proxy() {
  # ensure all processes receive the proxy settings by default
  # https://www.freedesktop.org/software/systemd/man/systemd-system.conf.html
  mkdir -p /etc/systemd/system.conf.d/

BenTheElder · 2022-08-10T20:15:56Z

Thanks, let me see if I grabbed the wrong image or built wrong and update.

BenTheElder · 2022-08-10T20:35:07Z

fixed, should be docker.io/bentheelder/kind-node:v1.24.3@sha256:4b649f5099d392a97f8031d4ff0730c2a13392fc18879b08a982768d00686ec1

maciekmm · 2022-08-10T20:47:23Z

Thanks! Can confirm this works under both rootful and rootless podman on zfs running fuse overlayfs. 🎉

caniko · 2022-08-11T07:21:40Z

Any snapshots for v1.22?

BenTheElder · 2022-08-11T17:18:46Z

Thanks for confirming @maciekmm 🙏

@caniko No, but you can build your own if you checkout the PR https://kind.sigs.k8s.io/docs/user/quick-start/#building-images

BenTheElder · 2022-08-18T21:19:34Z

should be fixed by #2874

current plan is to try to cut a release next week alongside kubernetes 1.25

maciekmm added the kind/bug Categorizes issue or PR as related to a bug. label Aug 8, 2022

BenTheElder added the area/provider/podman Issues or PRs related to podman label Aug 8, 2022

BenTheElder self-assigned this Aug 9, 2022

caniko mentioned this issue Aug 10, 2022

Fail to run rootful podman on Arch #2873

Closed

BenTheElder mentioned this issue Aug 10, 2022

rework snapshotter selection, detect fuseblk and use overlayfs-fuse #2874

Merged

BenTheElder changed the title ~~kind create cluster using podman provider fails to bootstrap the cluster - failed to mount rootfs component~~ [zfs] kind create cluster using podman provider fails to bootstrap the cluster - failed to mount rootfs component Aug 10, 2022

BenTheElder closed this as completed Aug 18, 2022

BenTheElder added this to the v0.15.0 milestone Aug 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[zfs] kind create cluster using podman provider fails to bootstrap the cluster - failed to mount rootfs component #2868

[zfs] kind create cluster using podman provider fails to bootstrap the cluster - failed to mount rootfs component #2868

maciekmm commented Aug 8, 2022

BenTheElder commented Aug 8, 2022

maciekmm commented Aug 8, 2022 •

edited

Loading

BenTheElder commented Aug 9, 2022

caniko commented Aug 10, 2022 •

edited

Loading

caniko commented Aug 10, 2022 •

edited

Loading

BenTheElder commented Aug 10, 2022

BenTheElder commented Aug 10, 2022 •

edited

Loading

maciekmm commented Aug 10, 2022

BenTheElder commented Aug 10, 2022

BenTheElder commented Aug 10, 2022

maciekmm commented Aug 10, 2022 •

edited

Loading

caniko commented Aug 11, 2022

BenTheElder commented Aug 11, 2022

BenTheElder commented Aug 18, 2022

[zfs] kind create cluster using podman provider fails to bootstrap the cluster - failed to mount rootfs component #2868

[zfs] kind create cluster using podman provider fails to bootstrap the cluster - failed to mount rootfs component #2868

Comments

maciekmm commented Aug 8, 2022

BenTheElder commented Aug 8, 2022

maciekmm commented Aug 8, 2022 • edited Loading

Rootful

Rootless

BenTheElder commented Aug 9, 2022

caniko commented Aug 10, 2022 • edited Loading

caniko commented Aug 10, 2022 • edited Loading

BenTheElder commented Aug 10, 2022

BenTheElder commented Aug 10, 2022 • edited Loading

maciekmm commented Aug 10, 2022

BenTheElder commented Aug 10, 2022

BenTheElder commented Aug 10, 2022

maciekmm commented Aug 10, 2022 • edited Loading

caniko commented Aug 11, 2022

BenTheElder commented Aug 11, 2022

BenTheElder commented Aug 18, 2022

maciekmm commented Aug 8, 2022 •

edited

Loading

caniko commented Aug 10, 2022 •

edited

Loading

caniko commented Aug 10, 2022 •

edited

Loading

BenTheElder commented Aug 10, 2022 •

edited

Loading

maciekmm commented Aug 10, 2022 •

edited

Loading