Adding worker node to K8s cluster created using Lima

[shivaveeru]

I created K8s cluster using k8s.yaml. I have created worker node instance. But when I tried to join the worker node to cluster using kubeadm,I am receiving error" couldn't validate the identity of API server "https://127.0.0.1:6443" Has anyone tried the multinode cluster using Lima.Please let me know.

[afbjorklund]

This would need some better configuration of the URL in the config, obviously "localhost" won't work for a multi-node cluster...

sed -e "s/${LIMA_CIDATA_SLIRP_IP_ADDRESS:-192.168.5.15}/127.0.0.1/" -i $KUBECONFIG

But I think that it does create certificates for both the IP and the name of the node, so maybe it will work if you change it ?

The kubeconfig was mostly intended to run from the host, with the 6443 being tunneled...

---

To create a multi-node cluster, it probably needs some "parameters" to the yaml : like if it is a control plane or a worker node

In order for the join command to work, it would also need to know 1) the address of the API server 2) the token from the CP

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#initializing-your-control-plane-node

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#join-nodes

The command to run should be in the ~/.lima/k8s/serial.log file:

[  121.211642] cloud-init[2044]: Then you can join any number of worker nodes by running the following on each as root:
[  121.211734] cloud-init[2044]: kubeadm join 192.168.5.15:6443 --token z29xai.a676mmrpm5glpgsc \
[  121.213731] cloud-init[2044]: 	--discovery-token-ca-cert-hash sha256:73b89e14897e5e92c7082a1f7485ac865e68d3041da870d3b0ce201285773624

You need the token, the url, and the hash (discovery-token-ca-cert-hash)

This should be done, instead of the whole the control plane system script...

---

EDIT: For k3s, the kubeconfig is in /etc/rancher/k3s/k3s.yaml

And the token is stored in /var/lib/rancher/k3s/server/node-token

https://docs.k3s.io/quick-start

If using k3s.yaml instead, that is.

[afbjorklund]

As long as your nodes and your pods get different IPs, there should be no issue. You can check with -o wide.

[louhisuo]

They got, checked this earlier earlier. But I just noticed that it is eth0 IP assigned as internal IP for Kubernetes nodes so will need to check this why and if it matters.

$ kubectl get nodes -o wide
NAME          STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
lima-test     Ready    control-plane   94m   v1.28.1   192.168.5.5   <none>        Ubuntu 22.04.3 LTS   5.15.0-82-generic   containerd://1.7.3
lima-test-2   Ready    <none>          91m   v1.28.1   192.168.5.6   <none>        Ubuntu 22.04.3 LTS   5.15.0-82-generic   containerd://1.7.3

$ kubectl get pods -A -o wide
NAMESPACE     NAME                                READY   STATUS    RESTARTS   AGE   IP            NODE          NOMINATED NODE   READINESS GATES
kube-system   cilium-kv8ln                        1/1     Running   0          84m   192.168.5.6   lima-test-2   <none>           <none>
kube-system   cilium-operator-788c4f69bc-6nstc    1/1     Running   0          84m   192.168.5.6   lima-test-2   <none>           <none>
kube-system   cilium-zfrzt                        1/1     Running   0          84m   192.168.5.5   lima-test     <none>           <none>
kube-system   coredns-5dd5756b68-cv8gl            1/1     Running   0          89m   10.0.0.198    lima-test     <none>           <none>
kube-system   coredns-5dd5756b68-khkqb            1/1     Running   0          89m   10.0.0.163    lima-test     <none>           <none>
kube-system   etcd-lima-test                      1/1     Running   0          89m   192.168.5.5   lima-test     <none>           <none>
kube-system   kube-apiserver-lima-test            1/1     Running   0          89m   192.168.5.5   lima-test     <none>           <none>
kube-system   kube-controller-manager-lima-test   1/1     Running   0          89m   192.168.5.5   lima-test     <none>           <none>
kube-system   kube-proxy-2tqlf                    1/1     Running   0          89m   192.168.5.5   lima-test     <none>           <none>
kube-system   kube-proxy-wfdm9                    1/1     Running   0          86m   192.168.5.6   lima-test-2   <none>           <none>
kube-system   kube-scheduler-lima-test            1/1     Running   0          89m   192.168.5.5   lima-test     <none>           <none>

[louhisuo]

Looks like kubelet uses default IP for a node which is IP for the first interface if --node-ip is not set. This is the reason why I see user-v2 network addresses as node INTERNAL-IPs. If I remember correctly kubeadm config has a separate section to configure kubelet so I can probably manipulate --node-ip flag value in that way.

[louhisuo]

Did some further testing and succeeded to setup 1x Control Plane and 3x Workers Kubernetes cluster with kubeadm with fairly little efforts (and no hacks).

I used Shared network, instead of user-v2 network in Lima because for me Host-VM communication is also important. I set advertiseAddress, node-ip and controlPlaneEndpoint fields in kubeadm init config (InitConfiguration and ClusterConfiguration) file to IP assigned to lima0interface in Control Plane node. I had also kubeadm join config file where I set apiServerEndpoint to point controlPlaneEndpoint in init config and node-ip to IP assigned to lima0 interface for a Worker node. Then instead of passing kubeadm joincommand around, I have pre-configured discovery.bootstrapToken to simplify the process for workers to join the cluster.

I used Kubernetes 1.28.1, Cilium 1.14.1 (CNI), MetalLB (L2 loadbalancer), Ingress-NGINX (L7 Ingress Controller) and an Echo-Server app to confirm quickly that things work. I did not try Control Plane HA setup with kube-vip yet, neither tried Cilium without kube-proxy but those will hopefully work as well.

So in summary, it is possible to setup multi-node K8s cluster using current Lima 0.17.2 release (kudos for Lima team). No hacks needed. Lima is used to provision VMs and kubeadm boot up a cluster after VMs are up and running. This process could be further streamlined if Lima starts supporting static or pre-configurable preditable IPs for its Shared and user-v2 networks somewhere in future. My personal preference here would be Sharednetwork as it gives Host-VM connectivity in addition to VM-VM connectivity enabling e.g. Kubernetes focused portable demos as an example use case. For development, people probably will be happy with VM-VMconnectivity provided by user-v2network.

[louhisuo]

Per advice in #1785, it was possible to make IP addresses deterministic when using shared network.

To keep lima.yaml as common template for all VMs created, I used below command syntax to (also limactl create works equally well) to create VMs.

limactl start --set='.networks[].macAddress = "52:55:55:12:34:56"' --name test test.yaml

Works like sharm. It is now possible to prepare kubeadm initand kubeadm joinconfigs for a cluster in advance, as IP addresses can be planned in advance, making it possible to create/tear down multi-node Kubernetes clusters quickly and with little manual efforts.