forked from kubernetes/website
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
manual rotation of CA certificates kubernetes#15444 and kubernetes#19165
- Loading branch information
1 parent
2d194dc
commit e8193e4
Showing
1 changed file
with
137 additions
and
0 deletions.
There are no files selected for viewing
137 changes: 137 additions & 0 deletions
137
content/en/docs/tasks/tls/manual-rotation-of-ca-certificates.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,137 @@ | ||
--- | ||
title: Manual Rotation of CA certificates | ||
content_template: templates/task | ||
reviewers: | ||
- | ||
--- | ||
|
||
{{% capture overview %}} | ||
|
||
This page shows how to enable and configure certificate rotation for the kubelet. | ||
|
||
{{% /capture %}} | ||
|
||
|
||
{{% capture prerequisites %}} | ||
|
||
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} | ||
|
||
Read about authentication in kubernetes [here](/docs/reference/access-authn-authz/authentication) | ||
|
||
{{% /capture %}} | ||
|
||
{{% capture steps %}} | ||
|
||
There are two possible ways of rotating or replacing the CA certificates. | ||
|
||
## Replace The Cluster (Recommended) | ||
|
||
This is by far the safest approach to rotate or replace the CA certificates. | ||
|
||
1. Create a new cluster with new CA certificates. | ||
|
||
2. Migrate the workloads to the new cluster. | ||
|
||
3. Remove or destroy the old cluster. | ||
|
||
## Rotate the CA certificates Manually | ||
|
||
{{< warning >}} | ||
The rotation or replacement of CA Certificates is not supported directly by kubeadm at this time and hence this documentation has to be followed with caution. Please make sure to take necessary backups of your certificate directory along with confs and any other necessary files. | ||
|
||
The whole idea of manually rotating the CA works on the key concpet of when exactly the cluster should know about the new certificates. **Please refrain from restarting any pods or updating any component in the cluster, other than this document asks to do**. | ||
|
||
{{< /warning >}} | ||
|
||
1. Take backup of your ```/etc/kubernetes``` directory, hereby referred to as **kubernetes conf directory** and ```/etc/kubernetes/pki```, hereby referred to as **kubernetes certificates directory**. | ||
|
||
{{< note >}} | ||
|
||
Kindly be advised to make necessary assumptions according to your kubernetes setup, in accordance with the directory locations mentioned above. | ||
|
||
{{< /note >}} | ||
|
||
2. Distribute new CA certificates to all your control plane nodes in kubernetes certificates directory. | ||
|
||
Follow [here](docs/setup/best-practices/certificates/#single-root-ca) to know more about best practices for CA certificates. | ||
|
||
Copy CA certificate file or CA bundle (whichever in use) on all nodes needed by kubelet's and control plane component's --client-ca-file flag. | ||
|
||
{{< note >}} | ||
|
||
Control plane components runs as a static pod manifests, managed by kubelet, they will not get disrupted by inplace updates of CA certificates until someone kills or restarts an already running pod. | ||
|
||
{{< /note >}} | ||
|
||
3. Get new certificates and private keys signed for (apiserver) apiserver.crt, (apiserver kubelet client) apiserver-kubelet-client.crt and (front proxy client) front-proxy-client.crt from new CA and replace in kubernetes certificates directory. | ||
|
||
Follow [here](/docs/concepts/cluster-administration/certificates/#openssl) to generate certificates and private keys for your cluster using openssl. You can also use [cfssl](/docs/concepts/cluster-administration/certificates/#cfssl). | ||
|
||
|
||
4. Update the config map **cluster-info** in kube-public namespace. | ||
|
||
```shell | ||
base64_encoded_ca=$(/usr/bin/base64 /etc/kubernetes/pki/ca.crt) | ||
|
||
kubectl get cm/cluster-info --namespace kube-public -o yaml | \ | ||
/bin/sed "s/\(certificate-authority-data:\).*/\1 ${base64_encoded_ca}" | \ | ||
kubectl apply -f - | ||
``` | ||
|
||
5. Update all service account tokens with the new CA certificates. | ||
```shell | ||
base64_encoded_ca=$(/usr/bin/base64 /etc/kubernetes/pki/ca.crt) | ||
|
||
for namespace in $(kubectl get ns --no-headers); do | ||
for token in $(kubectl get secrets --namespace $namepace --field-selector type=kubernetes.io/service-account-token -o name); do | ||
kubectl get $token --namespace $namepace -o yaml | \ | ||
/bin/sed "s/\(ca.crt:\).*/\1 ${base64_encoded_ca}" | \ | ||
kubectl apply -f - | ||
done | ||
done | ||
|
||
``` | ||
|
||
6. Update certificates for user accounts. | ||
|
||
Follow [here](/docs/setup/best-practices/certificates/#configure-certificates-for-user-accounts) for instructions to create certificates for individual user accounts. | ||
|
||
Also update the respective section of client-certificate-data and client-key-data with the contents of certificates and private keys hence generated in their respective conf files, along with (kubelet client) /var/lib/kubelet/kubelet-config-current.pem file. | ||
|
||
7. Update kubelet.conf, kubelet certificate and private key on all worker nodes as well. | ||
|
||
8. Update **certificate-autority-data** section in all admin.conf, controller-manager.conf, scheduler.conf and kubelet.conf files with base64 encoded CA certificate data. | ||
|
||
Example: | ||
```shell | ||
base64_encoded_ca=$(/usr/bin/base64 /etc/kubernetes/pki/ca.crt) | ||
|
||
/bin/sed -i "s/\(certificate-authority-data:\).*/\1 ${base64_encoded_ca}" /path/to/conf/files | ||
``` | ||
|
||
9. Execute below one by one on all control plane and worker nodes. | ||
|
||
* Stop Kubelet service one by one on all control plane and worker nodes. | ||
|
||
* Kill the static pod's containers manually on control plane nodes. | ||
```shell | ||
/bin/docker ps -q --name=k8s_* | xargs --no-run-if-empty docker rm -f - | ||
``` | ||
|
||
* Start kubelet service | ||
|
||
* Make sure control plane comes up and logs has no TLS errors. | ||
|
||
* Make sure container networking is functional. | ||
|
||
* Make sure coredns and kube-proxy are also updated with new CA if they were handled manually and are functional post CA rotation. | ||
|
||
* Kill or restart all other pods, one by one for all available namespaces, leveraging the benefit of rolling restarts to refresh service account credentials. | ||
|
||
```shell | ||
kubectl delete pods --namespace <namespace> | ||
``` | ||
|
||
10. [Distribute Self Signed CA certificates](/docs/concepts/cluster-administration/certificates/#distributing-self-signed-ca-certificate) | ||
|
||
{{% /capture %}} |