Data loss during etcd2 -> etcd3 migration

[marek-obuchowicz]

kops 1.11.1, k8s 1.11.8, AWS
(this is re-post of kopeio/etcd-manager#197 )

Versions used:

kops 1.11.1
k8s cluster: 1.11.9
infrastructure provider: aws

Our team was upgrading the etcd cluster (from 2.2.1 to 3.1.12) using kops, using the following scenario:

• kops edit cluster - add etcd manager and backups
• kops update cluster --out terraform, terraform apply
• kops rolling-update cluster --yes
• kops edit cluster - add etcd version 3.1.12
• kops update cluster --out terraform, terraform apply
• kops rolling-update cluster --yes

After some minutes I have executed kubectl get nodes and a big surprise - I see there only one node, with status "NotReady" - all other cluster nodes are gone.
Quick check and it seems that etcd-manager performed an upgrade of etcd2 to etcd3, but it lost the data and created new, empty cluster.

As an unexpected side effect, it has also affected kube-dns and flannel, which rendered k8s services (and therefore all ingresses and all services exposed via them) unavailable - so I consider a major outage, as not only masters were affected, but also services running inside k8s cluster were not able to reach each other and were not reachable from the Internet.

etcd-manager logged massive amount of data and the whole migration process, hopefully that's good enought to analyse the problem: https://gist.github.com/marek-obuchowicz/adda812f89644accc508b8d4db5db03c

"Luckily v1": "we have backups". At this moment we realised that there is no documentation provided how to restore those backups using etcd-manager. We considered going back to pure etcd (without etcd-manager) first in order to restore the contents, but this idea was rejected.

"Luckily v2": etcd2 data was still available on the volumes, as etcd3 cluster was created with another name (another directory name was used for data). I was able to workaround the issue and bring up my etcd2 cluster with original data by:

• editing state file on one node and forcing it back to old directory name / version 2.2.1 + changing etcd-cluster-spec back to version 2.2.1. It wasn't easy as the state file is a binary file (encoded with protobuf), so we had to write a little bit of go code to unmarshal the file first, change contents and then marshal it again: https://gist.github.com/marek-obuchowicz/c553effc19a97e40f01bc8e924b516ee
• editing etcd-cluster-spec file on s3 - change version back to 2.2.1
• restarting the node, on which state file was adjusted

By doing that, I was able to get again etcd2 cluster with old data. Manager correctly recognised on the node that "cluster wanted" and "local state" versions are 2.2.1, so it automatically created etcd2 cluster, using existing data. This solution however is pretty hacky and took long time to discover.

Please let me know if there is any more information I could provide to help analysing the problem.

We have executed the same operation, with the same steps, around two weeks earlier on a testing cluster - it was succesfull. There are two minor differences between testing cluster (uses CNI networking and is hosted in us-east-1 region) and live cluster that crashed (uses flannel networking and is hosted in eu-central-1 region). So I suspect the different behaviour might have been caused by latest etcd-manager updates.

I'm not sure if this is the correct place to report this issue or if it should be reported in etcd-manager directly, so please feel free to delete/close one of the duplicates

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[drewwells]

:( losing storage is such a major problem, wish this had better guard rails