Downgrade between two minor releases is not possible when etcd version change

[sayf-eddine-scality]

Component:
etcd, downgrade

What happened:
Downgrade failed from 2.3 to 2.2 because etcd does not support downgrading the cluster from 3.3 to 3.2 if all the cluster members are on 3.3
Check: etcd-io/etcd#10461

What was expected:
Downgrade to be successful

Steps to reproduce
Install a metalk8s 2.3 bootstrap node and downgrade to 2.2

Resolution proposal (optional):
TBD

[gdemonet]

Given this doc, it's impossible to downgrade, unless we create some snapshot of data while having the etcd cluster in v3.2

How would we handle this? I guess downgrade is necessary only after a breaking upgrade, right? We would need to warn users during the 2.2 -> 2.3 upgrade, wouldn't we?

[sayf-eddine-scality]

Given this doc, it's impossible to downgrade, unless we create some snapshot of data while having the etcd cluster in v3.2

How would we handle this? I guess downgrade is necessary only after a breaking upgrade, right? We would need to warn users during the 2.2 -> 2.3 upgrade, wouldn't we?

the operations and the api are obviously backward incompatible, so so the only way is to backup the cluster before going through this path to be able to restore a 2.2 working cluster if things went south

Or (bit complicated), we can have an intermediate state where we only upgrade the etcd and keep the rest on 2.2 then upgrade to 2.3 thus if the upgrade fail we will have a working 3.3 etcd on a 2.2 cluster, not sure if this is totally supported thought

[sayf-eddine-scality]

keeping this open due the possibility of having future versions of etcd non backward compatible.

[gdemonet]

What should be the course of action? We disabled downgrade tests for now, but I guess we'd still want to support it in the long run, no?

[sayf-eddine-scality]

yes I deactivated the downgrade test on the 2.3 branch but we need to find a definitive fix for this