Auto Renewing Etcd Certs

[vignesh-goutham]

What would you like to be added:
Etcdadm generates a root CA that is valid for 10 years. In addition to the root CA, etcdadm generates the following certs for normal operation of etcd. These are created for external etcd stack, kubeadm handles stacked etcd certificates.

• Peer Client Certs
• Api Server Etcd Client Certs
• Etcdctl Client Certs
• Etcd Server Certs

All these four certs have an expiry of 1 year. If there is no upgrade operation on the cluster that involves rolling out all old machines with new OS/nodes, these certs are not renewed and are at risk of expiry. At expiration, etcd fails to operate and api-server will not serve any requests, followed by workload failures.

EKS-A should

• Build in certs renewal. This could be a new EKS-A cli command, or an automated way that could be set with cluster spec during create/upgrade.
• Provide documentation on how to renew certs manually.
• Provide documentation on how to renew certs that have already expired.

[vignesh-goutham]

This person has a PR into etcdadm to expose a renew certs cli like kubeadm has - https://serverfault.com/questions/1054126/etcd-database-cluster-certificate-renewal-for-kubernets-external-database-setup

These code changes were tested and renews the certs appropriately. One complication here is, etcdadm generates a cert on each etcd node for api-server's etcd client. Each of these certs have the etcd node's name as CN on the cert. Instead of using this, etcdam controller should create a cert, update the secret on the cluster, which kubeadm controllers should pick up and update the api-server, like it happens on a create or upgrade workflow. Unfortunately, this might roll out nodes, we have to figure out a way to update the certs without rolling out nodes.

[vivek-koppuru]

We might need to prioritize some recommendation around this in the meantime: #8778