kubeadm master ip change without reseting the cluster (by changing all certificate including ca )

[rajibul007]

I have followed below ,but problem is it is using same certificate,kindly help me to to the same task by replacing all certificate(including ca)

systemctl stop kubelet docker
cd /etc/
mv kubernetes kubernetes-backup
mv /var/lib/kubelet /var/lib/kubelet-backup
mkdir -p kubernetes
cp -r kubernetes-backup/pki kubernetes
rm kubernetes/pki/{apiserver.,etcd/peer.}
rm -rf /root/.kube

change ip

systemctl start docker
kubeadm init --apiserver-advertise-address new_ip --ignore-preflight-errors=DirAvailable--var-lib-etcd
--kubernetes-version=1.15.4
rm -rf /root/.kube
mkdir -p "$HOME"/.kube
sudo cp -rf /etc/kubernetes/admin.conf "$HOME"/.kube/config
sudo chown $(id -u):$(id -g) "$HOME"/.kube/config
kubectl taint node --all node-role.kubernetes.io/master-

[rajibul007]

after that If I tried to change all cert by doing this #2015
..its failing

[neolit123]

kubeadm master ip change

this is not supported by kubeadm and the kubeadm team. it's a core kubernetes problem!

there are some proposals here:
#338
but some of them can fail depending on your wokloads.

a cluster should be created with a DNS name (or hostname in /etc/hosts). using an IP is a security contract and changing it is very complicated!!

[neolit123]

/triage support

[rajibul007]

@neolit123 could you help me how to create kubeadm init with only fqdn/DNS name so if we dont have to worry to change ip later

kubeadm init --apiserver-advertise-address ubuntu --apiserver-cert-extra-sans ubuntu --node-name ubuntu
couldn't use "ubuntu" as "apiserver-advertise-address", must be ipv4 or ipv6 address

So I did
kubeadm init --apiserver-advertise-address 127.0.0.1 --apiserver-cert-extra-sans ubuntu --node-name ubuntu
W0206 03:20:09.391491 4908 common.go:148] WARNING: overriding requested API server bind address: requested "127.0.0.1", actual "192.168.120.132"

after added hostname by using --apiserver-cert-extra-sans .I could see my hostname is pushed in ihe certificate ,but still after changing ip I am not able to run kubectl command as it is still checking with ip only not hostname

root@ubuntu:~/.kube# kubectl get po
Unable to connect to the server: x509: certificate is valid for 10.96.0.1, 192.168.120.132, not 192.168.120.133

Note (I have changed the ip in kubeconfig file ,all manifest file ,all config file
still no luck)

Please help

[neolit123]

please see the notes about --control-plane-endpoint in our docs:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#initializing-your-control-plane-node

it supports DNS names and can be used instead of --apiserver-advertise-address

[rajibul007]

@neolit123 thanks for you suggestion its works like a charm
after kubeadm init with --control-plane-endpoint ubuntu , I was able to run kubectl command after changing IP ,but after restarting machine or if we just restart docker to restart all conatiner
kubectl is not able to authenticate with api server

ping -c 2 ubuntu
PING ubuntu (192.168.120.133) 56(84) bytes of data.
64 bytes from ubuntu (192.168.120.133): icmp_seq=1 ttl=64 time=0.051 ms

root@ubuntu:/kubernetes/yamls# kubectl get po -A
The connection to the server ubuntu:6443 was refused - did you specify the right host or port?

I have tried changing ip in all manifest but still same
could see from kubelet log
Feb 06 05:35:47 ubuntu kubelet[43711]: E0206 05:35:47.805594 43711 kubelet_node_status.go:388] Error updating node status, will retry: error getting node "ubuntu": Get https://ubuntu:6443/api/v1/nodes/u
Feb 06 05:35:47 ubuntu kubelet[43711]: E0206 05:35:47.806170 43711 kubelet_node_status.go:388] Error updating node status, will retry: error getting node "ubuntu": Get https://ubuntu:6443/api/v1/nodes/u
Feb 06 05:35:47 ubuntu kubelet[43711]: E0206 05:35:47.806557 43711 kubelet_node_status.go:388] Error updating node status, will retry: error getting node "ubuntu": Get https://ubuntu:6443/api/v1/nodes/u
Feb 06 05:35:47 ubuntu kubelet[43711]: E0206 05:35:47.806981 43711 kubelet_node_status.go:388] Error updating node status, will retry: error getting node "ubuntu": Get https://ubuntu:6443/api/v1/nodes/u
Feb 06 05:35:47 ubuntu kubelet[43711]: E0206 05:35:47.807288 43711 kubelet_node_status.go:388] Error updating node status, will retry: error getting node "ubuntu": Get https://ubuntu:6443/api/v1/nodes/u
Feb 06 05:35:47 ubuntu kubelet[43711]: E0206 05:35:47.807331 43711 kubelet_node_status.go:375] Unable to update node status: update node status exceeds retry count
Feb 06 05:35:47 ubuntu kubelet[43711]: E0206 05:35:47.955982 43711 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSIDriver: Get https://ubuntu:6443/apis/storage.k8
Feb 06 05:35:48 ubuntu kubelet[43711]: W0206 05:35:48.156692 43711 status_manager.go:529] Failed to get status for pod "etcd-ubuntu_kube-system(8492253d260d04631559eacb7ce2f2a5)": Get https://ubuntu:644
Feb 06 05:35:48 ubuntu kubelet[43711]: E0206 05:35:48.357946 43711 reflector.go:123] object-"kube-system"/"coredns": Failed to list *v1.ConfigMap: Get https://ubuntu:6443/api/v1/namespaces/kube-system/c
Feb 06 05:35:48 ubuntu kubelet[43711]: E0206 05:35:48.556478 43711 reflector.go:123] object-"kube-system"/"coredns-token-n4cw8": Failed to list *v1.Secret: Get https://ubuntu:6443/api/v1/namespaces/kube
Feb 06 05:35:48 ubuntu kubelet[43711]: E0206 05:35:48.756380 43711 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Get https://ubuntu:6443/api/v1/nodes?fieldSele
Feb 06 05:35:48 ubuntu kubelet[43711]: E0206 05:35:48.956924 43711 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.RuntimeClass: Get https://ubuntu:6443/apis/node.k8
Feb 06 05:35:49 ubuntu kubelet[43711]: E0206 05:35:49.120480 43711 controller.go:135] failed to ensure node lease exists, will retry in 7s, error: Get https://ubuntu:6443/apis/coordination.k8s.io/v1/nam
Feb 06 05:35:49 ubuntu kubelet[43711]: E0206 05:35:49.156278 43711 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Get https://ubuntu:6443/api/v1/services?lim
Feb 06 05:35:49 ubuntu kubelet[43711]: E0206 05:35:49.357471 43711 reflector.go:123] object-"kube-system"/"weave-net-token-pd8tb": Failed to list *v1.Secret: Get https://ubuntu:6443/api/v1/namespaces/ku
Feb 06 05:35:49 ubuntu kubelet[43711]: W0206 05:35:49.556172 43711 status_manager.go:529] Failed to get status for pod "kube-apiserver-ubuntu_kube-system(6d2a022a9e7aca9a1fa4decac8d49365)": Get https://
Feb 06 05:35:49 ubuntu kubelet[43711]: E0206 05:35:49.756779 43711 reflector.go:123] object-"kube-system"/"kube-proxy-token-6lt9r": Failed to list *v1.Secret: Get https://ubuntu:6443/api/v1/namespaces/k
Feb 06 05:35:49 ubuntu kubelet[43711]: E0206 05:35:49.957741 43711 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://ubuntu:6443/api/v1/pods?fie
Feb 06 05:35:50 ubuntu kubelet[43711]: E0206 05:35:50.157074 43711 reflector.go:123] object-"kube-system"/"kube-proxy": Failed to list *v1.ConfigMap: Get https://ubuntu:6443/api/v1/namespaces/kube-syste
Feb 06 05:35:50 ubuntu kubelet[43711]: E0206 05:35:50.356170 43711 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSIDriver: Get https://ubuntu:6443/apis/storage.k8

[rajibul007]

@neolit123 please help

[neolit123]

thanks for you suggestion its works like a charm
after kubeadm init with --control-plane-endpoint ubuntu , I was able to run kubectl command after changing IP ,but after restarting machine or if we just restart docker to restart all conatiner
kubectl is not able to authenticate with api server

The connection to the server ubuntu:6443 was refused - did you specify the right host or port?

i don't have an explanation of why this is happening.