Have a strategy on how to react if Gardener changes the Shoot spec

[Disper]

Description:

It is possible that Gardener might modify or delete existing shoot created previously from Runtime CR spec. This may happen when for example Kubernetes version is updated automatically or manually by the Gardner admins. If we do not update Runtime CR accordingly - we will end with data inconsistency between Runtime CR and Gardener shoot.

There are also situations when shoot may become unavailable - because of deletion or manual hibernation.

In both cases such operation will not be noticed by the Runtime controller and will leave existing Runtime CR in "Ready" state when in reality the shoot will be deleted or hibernated.

Solution proposal:

• Add dedicated controller watching for the shoot resource, and update Runtime Spec with fields like Kubernetes version or availability status.
• To prevent shoot deletion Runtime controller can set its own finaliser on the managed shoot resource. This will not allow shoot to be deleted.

AC:

• Align with SRE how we should deal with field-updates applied in Shoot-Spec
  • Clarify how the general behaviour should be. Optional we see:
    1. sync all fields from Shoot-Spec
    2. Is custom sync-logic per field required? (e.g. what if Shoot-Spec gets upgraded but RuntimeCR requests even a newer K8s-version... )
    3. sync only selected fields (especially consider hybernation + deletions of clusters by users/SREs)
    4. ignore Shoot-Spec changes completely (probably risky in case of cluster-upgrades by Gardener)
  • Define which fields have to be synchronised
  • Clarify whether a field-specific upgrade logic is needed (consider cases where the Shoot-Spec and the RuntimeCR get changed in parallel - which field wins?)

[tobiscr]

Results of alignment meeting with SREs (@ebensom from 2024-06-25):