How to rotate karmada certificate if it is expired

[chaosi-zju]

Please provide an in-depth description of the question you have:

In many installation methods, the expiration time of the Karmada certificates is 365 days, when the time is up, most components will crash, just like:

$ kubectl get pod
NAME                                               READY   STATUS             RESTARTS            AGE
etcd-0                                             1/1     Running            22688 (38s ago)     569d
karmada-apiserver-6dd844fdfd-stzzg                 0/1     CrashLoopBackOff   62176 (4m54s ago)   569d
karmada-controller-manager-7dbf7c6578-kmztg        0/1     CrashLoopBackOff   57450 (4m9s ago)    326d
karmada-kube-controller-manager-656cdc675f-cj6vw   0/1     CrashLoopBackOff   57286 (3m42s ago)   569d
karmada-scheduler-764fbdcd6d-6jhrd                 1/1     Running            1                   569d
karmada-webhook-6489787db4-wc9pp                   1/1     Running            0                   569d

$ kubectl logs -f karmada-apiserver-6dd844fdfd-stzzg
W0328 02:20:08.682680       1 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://etcd-client.karmada-system.svc.cluster.local:2379/  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-03-28T02:20:08Z is after 2023-09-05T06:51:02Z". Reconnecting...

$ kubectl logs -f karmada-controller-manager-7dbf7c6578-kmztg
E0328 02:26:10.815013       1 controllermanager.go:78] failed to build controller manager: Get "https://karmada-apiserver.karmada-system.svc.cluster.local:5443/api?timeout=32s": dial tcp 10.254.52.163:5443: connect: connection refused
Get "https://karmada-apiserver.karmada-system.svc.cluster.local:5443/api?timeout=32s": dial tcp 10.254.52.163:5443: connect: connection refused

$ kubectl logs -f karmada-kube-controller-manager-656cdc675f-cj6vw
I0328 02:26:42.065358       1 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "client-ca-bundle::/etc/karmada/pki/server-ca.crt"
unable to load configmap based request-header-client-ca-file: Get "https://karmada-apiserver.karmada-system.svc.cluster.local:5443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication": dial tcp 10.254.52.163:5443: connect: connection refused

so, how to rotate karmada certificate if it is expired?

How to reproduce it?:

Step 1. modify hack/util.sh:

karmada/hack/util.sh

Lines 216 to 217 in c81649a

	${OPENSSL_BIN} req -x509 -sha256 -new -nodes -days 3650 -newkey rsa:2048 -keyout "${dest_dir}/${id}.key" -out "${dest_dir}/${id}.crt" -subj "/CN=${cn}/"
	echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment",${purpose}]}}}' > "${dest_dir}/${id}-config.json"

modify -days 365 to -days 1 in Line 216, so that default CA cert expiration time change to 1 day,
modify "expiry":"43800h" to "expiry":"1h" in Line 217, so that default karmada-apiserver cert expiration time change to 1 hour,
just like:

  ${OPENSSL_BIN} req -x509 -sha256 -new -nodes -days 1 -newkey rsa:2048 -keyout "${dest_dir}/${id}.key" -out "${dest_dir}/${id}.crt" -subj "/CN=${cn}/"
  echo '{"signing":{"default":{"expiry":"1h","usages":["signing","key encipherment",${purpose}]}}}' > "${dest_dir}/${id}-config.json"

Step 2. re-install karmada by hack/local-up-karmada.sh

Step 3. 1 hour later, the most component would crash down.

Environment:

• Karmada version:
• Kubernetes version:
• Others:

[pptfz]

Is there a solution to this problem?

[RainbowMango]

Hope we can get an answer by karmada-io/community#69.

[saowu]

I come from Jiajiayue Group, and my certificate expires in 1 year by default. I installed it through Karmadectl. Manually building certificates through cfSSL is very cumbersome, and I hope to develop a one click renewal tool similar to kubeadm certs renew all.
The current manual renewal method is as follows：

https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/certificates/

----------------证书清单-----------------------------------
rw-r--r-- 1 root root 1822 Aug 15 13:51 apiserver.crt
-rw------- 1 root root 1679 Aug 15 13:51 apiserver.key
-rw-r--r-- 1 root root 1834 Aug 15 13:51 karmada.crt
-rw------- 1 root root 1679 Aug 15 13:51 karmada.key
-rw-r--r-- 1 root root 1127 Aug 15 13:51 etcd-client.crt
-rw------- 1 root root 1675 Aug 15 13:51 etcd-client.key
-rw-r--r-- 1 root root 1359 Aug 15 13:51 etcd-server.crt
-rw------- 1 root root 1679 Aug 15 13:51 etcd-server.key
-rw-r--r-- 1 root root 1135 Aug 15 13:51 front-proxy-client.crt
-rw------- 1 root root 1675 Aug 15 13:51 front-proxy-client.key
-rw------- 1 root root 1675 Aug 15 13:51 tls.crt
-rw------- 1 root root 1675 Aug 15 13:51 tls.key
-rw------- 1 root root 1675 Aug 15 13:51 karmada-apiserver.config
-----------------------------
#创建 ca-config.json

# 查看原证书信息，获取CN、O、host配置，确认签发ca
openssl x509 -in apiserver.crt -noout -text


# 创建各类型-csr.json文件：主要关注CN、O、host配置，名字保持不变

# kubeconfig文件，ca.crt、证书、证书密钥
# 生成证书，karmada-apiserver.config需要base64解码生成crt文件，然后遵循以上步骤
cfssl gencert -ca=../pki/ca.crt -ca-key=../pki/ca.key -config=ca-config.json -profile=kubernetes apiserver-csr.json | cfssljson -bare apiserver
cfssl gencert -ca=../pki/ca.crt -ca-key=../pki/ca.key -config=ca-config.json -profile=kubernetes karmada-csr.json | cfssljson -bare karmada
cfssl gencert -ca=../pki/etcd-ca.crt -ca-key=../pki/etcd-ca.key -config=ca-config.json -profile=kubernetes etcd-client-csr.json | cfssljson -bare etcd-client
cfssl gencert -ca=../pki/etcd-ca.crt -ca-key=../pki/etcd-ca.key -config=ca-config.json -profile=kubernetes etcd-server-csr.json | cfssljson -bare etcd-server
cfssl gencert -ca=../pki/front-proxy-ca.crt -ca-key=../pki/front-proxy-ca.key -config=ca-config.json -profile=kubernetes front-proxy-client-csr.json | cfssljson -bare front-proxy-client
cfssl gencert -ca=../pki/ca.crt -ca-key=../pki/ca.key -config=ca-config.json -profile=kubernetes tls-csr.json | cfssljson -bare tls
cfssl gencert -ca=../pki/ca.crt -ca-key=../pki/ca.key -config=ca-config.json -profile=kubernetes karmada-api-config-csr.json | cfssljson -bare karmada-api-config


# 生成之后，更新到karmada-system Secret，除了ca都更新

#重启相关pod，all
# 停止所有正在运行的容器
crictl stop $(crictl ps -q)
# 删除所有容器
crictl rm $(crictl ps -a -q)

karmada_csr.tar.gz