Mark invalid snapshots as obsolete

[alexander-demicev]

We need to mark invalid as obsolete and prevent using obsolete snapshots when performing a restore. The snapshot is obsolete when:

• cluster version is different from when the snapshot was made
• cluster configuration changed(RKE2ControlPlane.Spec or RKE2ConfigTemplate.Spec)
• cluster the snapshot was made for doesn't exist anymore
• RKE2 control plane replicas didn't change

[Danil-Grigorev]

I think last point is already fixed by implementing #841 which maintains up-to-date list of snapshots.

• cluster the snapshot was made for doesn't exist anymore

[yiannistri]

I have tested the following scenario:

Deploy cluster at version 1.31 -> Take S3 snapshots -> Update cluster to 1.32 -> Restore S3 snapshots from 1.31: In this case, the restore seems to have no effect as the control planes appear to be running in different versions for a short period of time but eventually the nodes running the previous version become NotReady, possibly because the CAPI controllers continue to reconcile the cluster to the desired version (1.32):
Before restore

kubectl --kubeconfig rke2.kubeconfig get nodes
NAME                       STATUS   ROLES                       AGE     VERSION
rke2-control-plane-2kjpf   Ready    control-plane,etcd,master   7m42s   v1.32.0+rke2r1
rke2-md-0-sndmg-x9cdk      Ready    <none>                      3m11s   v1.32.0+rke2r1

After restore

kubectl --kubeconfig rke2.kubeconfig get nodes
NAME                       STATUS   ROLES                       AGE   VERSION
rke2-control-plane-2kjpf   Ready    control-plane,etcd,master   44s   v1.32.0+rke2r1
rke2-control-plane-s9t5j   Ready    control-plane,etcd,master   24m   v1.31.0+rke2r1
rke2-md-0-68s2w-rg427      Ready    <none>                      23m   v1.31.0+rke2r1

kubectl --kubeconfig rke2.kubeconfig get nodes
NAME                       STATUS     ROLES                       AGE     VERSION
rke2-control-plane-2kjpf   Ready      control-plane,etcd,master   2m14s   v1.32.0+rke2r1
rke2-control-plane-s9t5j   NotReady   control-plane,etcd,master   26m     v1.31.0+rke2r1
rke2-md-0-68s2w-rg427      NotReady   <none>                      24m     v1.31.0+rke2r1

It's worth noting that currently we only support creation of local snapshots from the management cluster. So attempting to restore S3 snapshots manually (as I've done above) and not through the management cluster may be out-of-scope for the time being.

[yiannistri]

Another scenario that I've tested:

Deploy cluster at version 1.31 -> Take local snapshot from mgmt cluster via the ETCDMachineSnapshot resource -> Restore snapshot from mgmt cluster via the ETCDSnapshotRestore resource: This time snapshot restore worked as expected and the cluster was fully operational after restore.

[yiannistri]

In general, since the current implementation of ETCD snapshot/restore in Turtles only supports local snapshots, it doesn't make a lot of sense to test the scenarios above:

• Updating the version of Kubernetes results in a rollout update that creates new nodes and deletes old CP nodes (and therefore any local snapshots
• Without S3 support, it's hard to restore a snapshot from a previous cluster into a new cluster

For the reasons above, it probably makes sense to close this issue and create a new one once we have S3 support implemented, wdyt @alexander-demicev ?

[yiannistri]

Moving back to backlog until we make a firm decision on the future of this.