New grafanafolder#spec.uid property for stable UIDs

[Baarsgaard]

I'm in the middle of defining a disaster-recovery process when recreating the Grafana Operator resources from scratch.

If I lose my cluster, ETCD state, or I mess up in a CD pipeline and any grafanafolder cr is deleted.
Recreating it results in an entirely new metadata.uid.

Currently, the only way to recover from this is through manually/programmatically:

1. Connect to every managed instance with the deleted folder
2. Delete all provisioned Alertrules and Dashboards
3. Delete the existing folder with the old UID
4. Have Grafana-Operator recreate the Folder, Alertrules, and Dashboards

Describe the solution you'd like
UID property on grafanafolder resource to ensure it's always the same for a specific folder name.

Describe alternatives you've considered

• Using folderUID on Alertrules/Dashboards
  This could work, except when onboarding new Grafana instances the folder does not exist and I cannot create the dashboards
• Using folder
  The folder will always have a different UID for each Grafana Instance. This could work, both when provisioning and Deleting.
  It does however mean I cannot use the folderRef and folderUID properties on Alertrules and Dashboards and the grafanafolder crd.
• I have tried "Tricking" the operator by applying a GrafanaFolder with the metadata.uid already specified, but it's overwritten on first apply.

[Baarsgaard]

I have since found out, the same issue is present on GrafanaContactPoint which also does not allow specifying UIDs

[pb82]

Allowing custom UIDs on folders can become complicated (more complicated than other resources), because: if you change the UID on an already existing folder, what do you do with the original one and all the dashboards it contains? You'd have to write special logic to handle those cases (and agree on a policy).

[Baarsgaard]

Could that be handled/enforced through an immutability clause requiring a recreate rather than an edit of the Grafanafolder resource?
You already have that on the instanceSelector.matchLabels in the CRDs I believe.

x-kubernetes-validations:
  - message: Value is immutable
    rule: self == oldSelf

That way, changing the UID is officially not supported, but it's possible to define a UID on creation?
This would also be compatible with #1626 which fixes blocking folder deletes when referenced without extra logic.

[theSuess]

@aboulay-numspot will try to implement this with immutable UIDs