Data loss during etcd2 -> etcd3 migration

[marek-obuchowicz]

Versions used:

kops 1.11.1
k8s cluster: 1.11.9
infrastructure provider: aws

Our team was upgrading the etcd cluster (from 2.2.1 to 3.1.12) using kops, using the following scenario:

• kops edit cluster - add etcd manager and backups
• kops update cluster --out terraform, terraform apply
• kops rolling-update cluster --yes
• kops edit cluster - add etcd version 3.1.12
• kops update cluster --out terraform, terraform apply
• kops rolling-update cluster --yes

After some minutes I have executed kubectl get nodes and a big surprise - I see there only one node, with status "NotReady" - all other cluster nodes are gone.
Quick check and it seems that etcd-manager performed an upgrade of etcd2 to etcd3, but it lost the data and created new, empty cluster.

As an unexpected side effect, it has also affected kube-dns and flannel, which rendered k8s services (and therefore all ingresses and all services exposed via them) unavailable - so I consider a major outage, as not only masters were affected, but also services running inside k8s cluster were not able to reach each other and were not reachable from the Internet.

etcd-manager logged massive amount of data and the whole migration process, hopefully that's good enought to analyse the problem: https://gist.github.com/marek-obuchowicz/adda812f89644accc508b8d4db5db03c

"Luckily v1": "we have backups". At this moment we realised that there is no documentation provided how to restore those backups using etcd-manager. We considered going back to pure etcd (without etcd-manager) first in order to restore the contents, but this idea was rejected.

"Luckily v2": etcd2 data was still available on the volumes, as etcd3 cluster was created with another name (another directory name was used for data). I was able to workaround the issue and bring up my etcd2 cluster with original data by:

• editing state file on one node and forcing it back to old directory name / version 2.2.1 + changing etcd-cluster-spec back to version 2.2.1. It wasn't easy as the state file is a binary file (encoded with protobuf), so we had to write a little bit of go code to unmarshal the file first, change contents and then marshal it again: https://gist.github.com/marek-obuchowicz/c553effc19a97e40f01bc8e924b516ee
• editing etcd-cluster-spec file on s3 - change version back to 2.2.1
• restarting the node, on which state file was adjusted

By doing that, I was able to get again etcd2 cluster with old data. Manager correctly recognised on the node that "cluster wanted" and "local state" versions are 2.2.1, so it automatically created etcd2 cluster, using existing data. This solution however is pretty hacky and took long time to discover.

Please let me know if there is any more information I could provide to help analysing the problem.

We have executed the same operation, with the same steps, around two weeks earlier on a testing cluster - it was succesfull. There are two minor differences between testing cluster (uses CNI networking and is hosted in us-east-1 region) and live cluster that crashed (uses flannel networking and is hosted in eu-central-1 region). So I suspect the different behaviour might have been caused by latest etcd-manager updates.

I'm not sure if this is the correct place to report this issue or if I should open it in kops project, but looks to me like it's related to etcd-manager directly.

[mmerrill3]

When I performed this procedure (in AWS with kops 1.11.1, etcd manager, and weave), after the first master is restarted, I don't see any of my pods either. But, they do show up after the second master is restarted, and forms a 3.1 etcd cluster version. Of course, all of the pods need to be started from scatch, and the cluster needs to stabilize. There is no solution for the etcd version 2 to version 3 upgrade without downtime, and all of your pods restarting. I engineered for it by sending all API traffic to another AWS region with a different k8s cluster.

[marek-obuchowicz]

This worked well on a test cluster, but on production cluster the following scenario happened:

• first master was shut down and recreated by ASG
• it setup a new etcd3 cluster - which was empty, it didn't have the data from etcd2 cluster
• existing masters had their etcd2 stopped by etcd-manager and they started etcd3 and joint the (empty) etcd3 cluster correctly
• kops reported that the cluster didn't stabilize after waiting
• kubectl get nodes showed only one master node (the one new node which was restarted) with status NotReady

[jjo]

Our upgrade scenario:

• kops: 1.11.0 -> 1.12.1
• kube: 1.11.10 -> 1.12.9

We followed recommended 1.11 -> 1.12 upgrade procedure, sp regarding rolling-update --cloud-only ... the masters.

Impact:

After doing the upgrade we got main etcd cluster unformed with:

• 2 masters running with previous etcd 2.2 data volume, stuck without being able to form the cluster
• 1 master (etcd-b) couldn't mount previous filesystem, completely re-mkfs'd its EBS etcd-data master-vol

We had previously taken etcd-v2 backups, nevertheless ripping the "rogue" master instance (etcd-b in our case) and letting the ASG recreate it made the (empty) created FS to be handled as etcd-v2 (empty) replica, re-sync, and complete etcd-manager upgrading steps up to final 3.3.

I'm attaching the etcd-b problematic instance logfile (some edits to obfuscate actual cluster name, bucket name and logged tokens)

kops-1.12-upgrade-etcd-b.mkfs-ing.log