Use lima as a driver #12508
Comments
|
Ultimately we might want to have a QEMU driver for Mac, or reuse the current libvirt driver called "kvm2" The main problem is with allocating the network, currrently lima is doing the ssh hacks we rejected earlier: Note that the default container engine in lima is containerd/buildkitd, which is not yet supported by minikube (or kind) So you will need to change that for either docker or podman, in order to run. But there are example yaml files of both... |
|
I think Docker's HyperKit is already deprecated, being replaced with Apple's Virtualization.framework Unfortunately the needed QEMU patches for arm64 (M1) are not merged yet, but it works on amd64... There is an See upstream lima for how to build a patched qemu version, there is a third-party |
|
@afbjorklund Ok, I get it.
Can you link the issue or pr here? I am using the M1 mac now. |
This comment has been minimized.
This comment has been minimized.
|
This app is also looking promising, as a GUI instead of VirtualBox: https://mac.getutm.app/ There is also a cross-platform version called AQEMU: https://github.com/tobimensch/aqemu Note that with these solutions, you end up with a virtual machine (VM) running on your Mac. So you still need to configure
|
|
Using lima is similar to using WSL2 on Windows, it will handle the VM and network and host mounts for you.
Even though intended for macOS, you can run lima also on Linux. This is good for the developers, such as myself. The mounted filesystems have the same performance problems as the other solutions, and the networking is limited. |
|
If you want to run multi-node or complex networking, or want full control over storage, I would recommend using a VM driver. But for casual users, these drivers are handy. I call it the "Wizard of Oz mode", pay no attention to the man behind the curtain! |
|
I am not familiar with lima, but if it is just like docker or podman, and can use standard OCI image, I would accept a contribution that adds it as another minikube driver |
It is similar to WSL. It uses containerd/buildkitd by default, though. See kubernetes-sigs/kind#2317 for a discussion about But Lima can use any distribution with any container runtime... |
|
The default docker installation in lima is rootless:
$ limactl start ./docker.yaml
$ export DOCKER_HOST=unix://$HOME/docker.sock
$ minikube start
$ minikube start --driver docker --container-runtime=containerd
So minikube works in lima, when using the "docker" (or "podman") driver. It is also possible to run minikube with the "none" driver. (see #12926) But currently there is no need to support lima, as a minikube driver ? Instead it is used to provide a VM - similar to the VM of Docker Desktop. |
|
I've tried to follow the instructions above, but for me it cannot create the control plane:
|
|
Maybe it needs some special configuration, to run with rootless docker ? I haven't tried that myself. There is some talk about cgroup delegation: https://rootlesscontaine.rs/getting-started/common/cgroup2/ https://minikube.sigs.k8s.io/docs/drivers/docker/#rootless-docker The alternative would be to use the regular
And modify in the lima yaml file: |
|
This issue happens with the latest minikube, but with 1.24.0, it works fine. However, I tried @afbjorklund's suggestion for the newest minikube didn't work; I might be missing something. Docker socket is not usable from the host machine, but inside VM, it is usable for both lima user and root. minikube v1.24.0 installation for MacOS: |
|
I think you want to open an issue on https://github.com/lima-vm/lima for that, the ssh tunneling of the unix socket "should" work |
|
Theoretically, one could write a howto on how to deploy rootful docker on Lima and then use that to run kubernetes-in-docker. But for the casual user, I think it would be much more straight-forward to just start a virtual machine with Kubernetes on it ? https://github.com/lima-vm/lima/blob/master/examples/k8s.yaml
It is basically an executable version of the upstream documentation: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/ Just that it chooses a distribution (ubuntu) and a runtime (containerd) and a network (flannel) for you automatically. Of course, you could also run |
|
Giving access to all users fixes the problem with root-docker.yaml# Example to use Docker instead of containerd & nerdctl
# $ limactl start ./root-docker.yaml
# $ limactl shell docker docker run -it -v $HOME:$HOME --rm alpine
# To run `docker` on the host (assumes docker-cli is installed):
# $ export DOCKER_HOST=$(limactl list docker --format 'unix://{{.Dir}}/sock/docker.sock')
# $ docker ...
# This example requires Lima v0.8.0 or later
images:
# Hint: run `limactl prune` to invalidate the "current" cache
- location: "https://cloud-images.ubuntu.com/impish/current/impish-server-cloudimg-amd64.img"
arch: "x86_64"
- location: "https://cloud-images.ubuntu.com/impish/current/impish-server-cloudimg-arm64.img"
arch: "aarch64"
mounts:
- location: "~"
- location: "/tmp/lima"
writable: true
# containerd is managed by Docker, not by Lima, so the values are set to false here.
containerd:
system: false
user: false
provision:
- mode: system
script: |
#!/bin/sh
sed -i 's/host.lima.internal.*/host.lima.internal host.docker.internal/' /etc/hosts
- mode: system
script: |
#!/bin/bash
set -eux -o pipefail
command -v docker >/dev/null 2>&1 && exit 0
export DEBIAN_FRONTEND=noninteractive
curl -fsSL https://get.docker.com | sh
# You can active it here but doesn't change the behavior.
# usermod -aG docker lima # lima is default user for my system.
- mode: user
script: |
#!/bin/bash
set -eux -o pipefail
sudo usermod -aG docker $USER
portForwards:
- guestSocket: "/var/run/docker.sock"
hostSocket: "{{.Dir}}/sock/docker.sock"
message: |
To run `docker` on the host (assumes docker-cli is installed):
$ export DOCKER_HOST=unix://{{.Dir}}/sock/docker.sock
$ docker ...If you try the YAML example above, execute |
You need a new login session, or to use "Rebooting", as in stopping and starting also works (as usual). Eventually it should be possible to run the KIC installation also in rootless docker, but it might require some tweaks. |
Went back to minikube 1.24 and indeed the issue disappeared. Great thanks @eminaktas. Is mounting from lima supported atm? I can mount my home directory into lima vm ( The path |
|
You can add volume mounts from the docker host (VM) to the minikube node (container) Forgot the syntax right now, but should be on docker driver page (or volumes) EDIT: The volumes go in --mount-string |
|
If it works with EDIT: Just learned that the former runs via the driver and the latter via 9P. |
|
Just tried it The folder is generated, but empty: I have plenty files under this folder on my host and the path is the same under lima vm. |
|
Note that https://minikube.sigs.k8s.io/docs/handbook/persistent_volumes/ Check the location with |
|
Awesome! Changing the dir to /mydata works. Everything works (minikube 1.24). 👍 Also, for anyone else using an ARM architecture, keep in mind that the default docker configuration doesn't support cross compiling. So, you may not get amd64 images to work. I finally found a hint on how to deal with this here: lima-vm/lima#42 (comment) |
|
Even if this is closed - I had to downgrade minikube from 1.25.x -> 1.24 to get this working. I am using the example/docker.yaml for starting the lima-vm. The issue encountered with the latest minikube version is exactly the same as @metaswirl explained in #12508 (comment). Or would you recommend to use colima with k8s, as it integrates k8s as a maintained feature? |
The long term plan (well, "current") is to provide a similar driver for minikube (to lima's), and then run the minikube os with it. Sadly minikube has some issues running under lima (#12926), so therefore it is recommended to run kubeadm (k8s.yaml)
I'm not sure why you would use colima, it seems like a limited version of Rancher Desktop ? But it is theoretically possible to install docker using lima, and then use minikube's docker driver. I think the first step would be to have some better documentation in lima, on how these work |
|
Thanks for the quick reply. Maybe some background ;-) ... My use-case is just having some kind of k8s playground on my local machine. So what I am looking for is an easy approach to be able to run kubectl on my machine. So after migrating from docker for desktop to lima-vm (with config: ./docker.yaml), I am looking for an easy way to install a k8s cluster. What I did understand from your answer and #12926 is, that minikube deploys the k8s cluster in a docker container, which creates multiple additional docker containers (so Docker in Docker). So maybe I give this a shot - thanks a lot. And one last thing: minikube > v1.24.0 has problems using the redirected docker.socks file? |
Correct. It is more similar to the "ssh" (generic) driver in minikube, with the VM already provided by lima.
It is a little surprised by it. In the old days, you either had a local unix socket or a remote tcp socket. This remote unix socket confuses some old assumptions, so needs some workaround to get the IP... |
Steps to reproduce the issue:
I want to use lima instead of docker on mac. Can I use lima as a minikube driver?
Run
minikube logs --file=logs.txtand drag and drop the log file into this issueFull output of failed command if not
minikube start:The text was updated successfully, but these errors were encountered: