34-understand-kubernetes-cluster-upgrade-process

Cluster Maintenance: Understand Kubernetes Cluster Upgrade Process

Back to Certified Kubernetes Administrator (CKA) Tutorial

Also includes:

• Cluster Maintenance: Implement Backup and Restore Methodologies

• Cluster Maintenance: Facilitate operating system upgrades

Addendum: Instead of using the apiserver-etcd-client.crt (and key) to authorize etcdctl, we can simply use the server.crt (and key) in the etcd folder.

Addendum: Did not cover restoring from a snapshot as the K8s documentation is pretty thin here. But following the documentation provided by CKA CKAD Simulator , the process is essentially as follows:

1. Stop kube-apiserver (move static configuration out of manifests folder)

2. Restore the snapshot into a new etcd database folder, e.g, ETCDCTL_API=3 etcdctl snapshot restore /tmp/etcd-backup.db --data-dir /var/lib/etcd-backup

3. Update the etcd static configuration file to point to the new etcd database folder; this will reload etcd

4. Start kube-apiserver (restore static configuration)

Test Preparation: Relevant Kubernetes Documentation

Search for etcd and find Operating etcd clusters for Kubernetes.

Search for upgrade cluster and find Upgrading kubeadm clusters.

Script

First, we can tell the current versions of both the Cluster and the Nodes using:

kubectl version

kubectl get nodes

EKS

AWS EKS provides a turn-key upgrade process that requires little to no explanation.

When a new Kubernetes version is available in Amazon EKS, you can update your cluster to the latest version.

-AWS-Updating an Amazon EKS cluster Kubernetes version

Backup and Restore Strategies

It is more instructive to explore upgrading the Cluster that we installed with kubeadm and the first thing that we need to do is consider our backup and restore strategy.

Make sure to back up any important components, such as app-level state stored in a database. kubeadm upgrade does not touch your workloads, only components internal to Kubernetes, but backups are always a best practice.

-Kubernetes-Upgrading kubeadm clusters

note: It is interesting to observe that backup and restore are not mentioned in the AWS EKS documentation.

After a bit of searching, one recommended approach is to ensure that the K8s configuration is captured in configuration files stored in source control that get deployed through a continous integration process (allows for storing secret data outside of source control).

Another is to backup the database directly:

All Kubernetes objects are stored on etcd. Periodically backing up the etcd cluster data is important to recover Kubernetes clusters under disaster scenarios, such as losing all master nodes. The snapshot file contains all the Kubernetes states and critical information. In order to keep the sensitive Kubernetes data safe, encrypt the snapshot files.

and

Backing up an etcd cluster can be accomplished in two ways: etcd built-in snapshot and volume snapshop

-Kubernetes-Operating etcd clusters for Kubernetes

note: We will not cover taking a volume snapshot.

note: Could not find any documentation on using etcdctl with EKS; even getting it to work with our kubeadm installed Cluster was tricky.

Download (and then uncompress):

wget https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz

Need to run as root (for certs):

Validate connectivity. Explain certificates.

ETCDCTL_API=3 \
  /home/ubuntu/etcd-v3.4.9-linux-amd64/etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/apiserver-etcd-client.crt \
  --key /etc/kubernetes/pki/apiserver-etcd-client.key \
  --debug \
  --endpoints https://127.0.0.1:2379 \
  endpoint status

Take a snapshot:

ETCDCTL_API=3 \
  /home/ubuntu/etcd-v3.4.9-linux-amd64/etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/apiserver-etcd-client.crt \
  --key /etc/kubernetes/pki/apiserver-etcd-client.key \
  --debug \
  --endpoints https://127.0.0.1:2379 \
  snapshot save snapshotdb

Verify:

ETCDCTL_API=3 \
  /home/ubuntu/etcd-v3.4.9-linux-amd64/etcdctl \
  --write-out=table snapshot status snapshotdb

This file can be used to restore the database using the etcdctl CLI tool.

Upgrading kubeadm Clusters

Instructions are detailed:

This page explains how to upgrade a Kubernetes cluster created with kubeadm from version 1.17.x to version 1.18.x, and from version 1.18.x to 1.18.y (where y > x).

-Kubernetes-Upgrading kubeadm clusters

Will summarize steps:

Determine available versions

Upgrade kubeadm

Drain Control Plane Node (master); show:

kubectl drain ip-172-31-32-241 --ignore-daemonsets

Notice new taint too.

Plan upgrade; show:

kubadmn upgrade plan

Gives options to actually do upgrade.

note: Will have to consider upgrades of other addons; including Pod Networking CNI addon.

Uncordon master:

kubectl uncordon ip-172-31-32-241

Notice taints.

Upgrade additional Control Plane Nodes using same process; but different final command.

Update kubectl and kublet on all control plane nodes; includes restarting kublet.

For each Worker Node pretty much the same as the secondary control plane nodes.

note: Upgrade does create some temp files should the upgrade fail and you have to manually restore files.

Also, remember that upgrade process updates the TLS certificates that expire yearly; so regular updates addreses the need to update them.

Node OS Upgrades

While did not find documents on this, seems like best bet is to drain the Nodes, apply upgrades, and then uncordon the Node.

Looks like K8s marks its packages to prevent accidental upgrade when OS is upgraded.