Skip to content

Commit

Permalink
manual rotation of CA certificates kubernetes#15444 and kubernetes#19165
Browse files Browse the repository at this point in the history
  • Loading branch information
abhiTamrakar committed May 24, 2020
1 parent 2d194dc commit e8193e4
Showing 1 changed file with 137 additions and 0 deletions.
137 changes: 137 additions & 0 deletions content/en/docs/tasks/tls/manual-rotation-of-ca-certificates.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
---
title: Manual Rotation of CA certificates
content_template: templates/task
reviewers:
-
---

{{% capture overview %}}

This page shows how to enable and configure certificate rotation for the kubelet.

{{% /capture %}}


{{% capture prerequisites %}}

{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}

Read about authentication in kubernetes [here](/docs/reference/access-authn-authz/authentication)

{{% /capture %}}

{{% capture steps %}}

There are two possible ways of rotating or replacing the CA certificates.

## Replace The Cluster (Recommended)

This is by far the safest approach to rotate or replace the CA certificates.

1. Create a new cluster with new CA certificates.

2. Migrate the workloads to the new cluster.

3. Remove or destroy the old cluster.

## Rotate the CA certificates Manually

{{< warning >}}
The rotation or replacement of CA Certificates is not supported directly by kubeadm at this time and hence this documentation has to be followed with caution. Please make sure to take necessary backups of your certificate directory along with confs and any other necessary files.

The whole idea of manually rotating the CA works on the key concpet of when exactly the cluster should know about the new certificates. **Please refrain from restarting any pods or updating any component in the cluster, other than this document asks to do**.

{{< /warning >}}

1. Take backup of your ```/etc/kubernetes``` directory, hereby referred to as **kubernetes conf directory** and ```/etc/kubernetes/pki```, hereby referred to as **kubernetes certificates directory**.

{{< note >}}

Kindly be advised to make necessary assumptions according to your kubernetes setup, in accordance with the directory locations mentioned above.

{{< /note >}}

2. Distribute new CA certificates to all your control plane nodes in kubernetes certificates directory.

Follow [here](docs/setup/best-practices/certificates/#single-root-ca) to know more about best practices for CA certificates.

Copy CA certificate file or CA bundle (whichever in use) on all nodes needed by kubelet's and control plane component's --client-ca-file flag.

{{< note >}}

Control plane components runs as a static pod manifests, managed by kubelet, they will not get disrupted by inplace updates of CA certificates until someone kills or restarts an already running pod.

{{< /note >}}

3. Get new certificates and private keys signed for (apiserver) apiserver.crt, (apiserver kubelet client) apiserver-kubelet-client.crt and (front proxy client) front-proxy-client.crt from new CA and replace in kubernetes certificates directory.

Follow [here](/docs/concepts/cluster-administration/certificates/#openssl) to generate certificates and private keys for your cluster using openssl. You can also use [cfssl](/docs/concepts/cluster-administration/certificates/#cfssl).


4. Update the config map **cluster-info** in kube-public namespace.

```shell
base64_encoded_ca=$(/usr/bin/base64 /etc/kubernetes/pki/ca.crt)

kubectl get cm/cluster-info --namespace kube-public -o yaml | \
/bin/sed "s/\(certificate-authority-data:\).*/\1 ${base64_encoded_ca}" | \
kubectl apply -f -
```

5. Update all service account tokens with the new CA certificates.
```shell
base64_encoded_ca=$(/usr/bin/base64 /etc/kubernetes/pki/ca.crt)

for namespace in $(kubectl get ns --no-headers); do
for token in $(kubectl get secrets --namespace $namepace --field-selector type=kubernetes.io/service-account-token -o name); do
kubectl get $token --namespace $namepace -o yaml | \
/bin/sed "s/\(ca.crt:\).*/\1 ${base64_encoded_ca}" | \
kubectl apply -f -
done
done

```

6. Update certificates for user accounts.

Follow [here](/docs/setup/best-practices/certificates/#configure-certificates-for-user-accounts) for instructions to create certificates for individual user accounts.

Also update the respective section of client-certificate-data and client-key-data with the contents of certificates and private keys hence generated in their respective conf files, along with (kubelet client) /var/lib/kubelet/kubelet-config-current.pem file.

7. Update kubelet.conf, kubelet certificate and private key on all worker nodes as well.

8. Update **certificate-autority-data** section in all admin.conf, controller-manager.conf, scheduler.conf and kubelet.conf files with base64 encoded CA certificate data.

Example:
```shell
base64_encoded_ca=$(/usr/bin/base64 /etc/kubernetes/pki/ca.crt)

/bin/sed -i "s/\(certificate-authority-data:\).*/\1 ${base64_encoded_ca}" /path/to/conf/files
```

9. Execute below one by one on all control plane and worker nodes.

* Stop Kubelet service one by one on all control plane and worker nodes.

* Kill the static pod's containers manually on control plane nodes.
```shell
/bin/docker ps -q --name=k8s_* | xargs --no-run-if-empty docker rm -f -
```

* Start kubelet service

* Make sure control plane comes up and logs has no TLS errors.

* Make sure container networking is functional.

* Make sure coredns and kube-proxy are also updated with new CA if they were handled manually and are functional post CA rotation.

* Kill or restart all other pods, one by one for all available namespaces, leveraging the benefit of rolling restarts to refresh service account credentials.

```shell
kubectl delete pods --namespace <namespace>
```

10. [Distribute Self Signed CA certificates](/docs/concepts/cluster-administration/certificates/#distributing-self-signed-ca-certificate)

{{% /capture %}}

0 comments on commit e8193e4

Please sign in to comment.