add support for rootless Docker

[medyagh]

continuation of #9495

[azhao155]

/assign @azhao155

[afbjorklund]

Related to #9495 and #8719 (for podman)

[azhao155]

Try out dockerless for minikube, docker driver doesn't work due to some connection issue. will investigate.

zyanshu@zyanshukvm:~/minikube$ ./out/minikube start
😄  minikube v1.18.1 on Debian rodete (kvm/amd64)
✨  Automatically selected the docker driver. Other choices: kvm2, ssh
❗  docker is currently using the vfs storage driver, consider switching to overlay2 for better performance
👍  Starting control plane node minikube in cluster minikube
🔥  Creating docker container (CPUs=2, Memory=26100MB) ...
🤦  StartHost failed, but will try again: creating host: create: creating: setting up container node: preparing volume for minikube container: docker run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.18@sha256:ddd0c02d289e3a6fb4bba9a94435840666f4eb81484ff3e707b69c1c484aa45e -d /var/lib: exit status 125
stdout:

stderr:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:340: applying cgroup configuration for process caused: write unix @: sendmsg: broken pipe: unknown.

🤷  docker "minikube" container is missing, will recreate.
🔥  Creating docker container (CPUs=2, Memory=26100MB) ...
😿  Failed to start docker container. Running "minikube delete" may fix it: recreate: creating host: create: creating: setting up container node: preparing volume for minikube container: docker run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.18@sha256:ddd0c02d289e3a6fb4bba9a94435840666f4eb81484ff3e707b69c1c484aa45e -d /var/lib: exit status 125
stdout:

stderr:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:340: applying cgroup configuration for process caused: read unix @->/run/systemd/private: read: connection reset by peer: unknown.

❗  Startup with docker driver failed, trying with alternate driver kvm2: Failed to start host: recreate: creating host: create: creating: setting up container node: preparing volume for minikube container: docker run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.18@sha256:ddd0c02d289e3a6fb4bba9a94435840666f4eb81484ff3e707b69c1c484aa45e -d /var/lib: exit status 125
stdout:

stderr:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:340: applying cgroup configuration for process caused: read unix @->/run/systemd/private: read: connection reset by peer: unknown.

🔥  Deleting "minikube" in docker ...
🔥  Removing /usr/local/google/home/zyanshu/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.
👍  Starting control plane node minikube in cluster minikube
🔥  Creating kvm2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.3 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v4
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

[medyagh]

speaking to @afbjorklund he suggested using a newer kernel

[azhao155]

@afbjorklund my kernel version is Debian 5.7.17-1rodete5 what version do you suggest to upgrade to?

[afbjorklund]

When looking at the new podman, I saw there were some improvements with overlayfs without fuse.

However, they did require Linux 5.11 (so not LTS): https://rootlesscontaine.rs/how-it-works/overlayfs/

Before that, you needed to install fuse-overlayfs

Anyway, it shouldn't fail ẁith vfs either (just slow)

[afbjorklund]

@azhao155 : Will this feature really be ready for v1.20.0 (in two weeks) ?

I think it hasn't even landed in kind yet, even though it is in progress (0.11)

https://kind.sigs.k8s.io/docs/user/rootless/

@medyagh : we might want to consider lowering the priority on this feature

[azhao155]

@afbjorklund after update the kernel, it solves the above problem, but kubeadm init fail with "getting the final child's pid from pipe caused "EOF"": unknown". Still debugging it now. Not certain if it could be ready in two weeks.

[AkihiroSuda]

The first step toward this is to bring proper support for cgroup2: #11310

[azhao155]

The first step toward this is to bring proper support for cgroup2: #11310

Yes, this is what i see to use containerd runtime, is that due to /sys/fs/cgroups/systemd not mount correctly?

[AkihiroSuda]

/sys/fs/cgroups/systemd

No, this does not exist in the v2 ecosystem.

I was thinking that https://github.com/kubernetes-sigs/kind/pull/2014/files might be related, but seems a different issue.

[AkihiroSuda]

Opened PR: #12359