OCPBUGS-28647: tuned: operand: add support for deferred updates

Users are able to make tuning changes through NTO/TuneD by updating
the CRs on a running system.
When the tuning change is done, the NTO sends a SIGHUP to the tuned
process, which results in the tuned process rolling back ALL the
tuning and then reapplying it.
In some key use cases, this disrupts the workload behavior.

We add an annotation that cause the update not to be applied
immediately, but to be deferred till the next tuned restart.
The NTO operator will continue merging the tuned objects into the
special `rendered` object as it does today.
The deferred annotation will be "sticky". If even just one of
the current tuned object (e.g. just the perfprofile one) have
the annotation, the rendered tuned will.

However, if there is a deferred updated pending but the tuned
objects are edited in such a way the final `rendered` update will be
immediate (= no deferred annotation), then the immediate update
will be applied overwriting the pending deferred update, which will
then be lost.

The deferred updates can stack, meaning a deferred update can be
applied on top of a previous deferred update; the latter will
just overwrite the former as it never was received.
An immediate update can overwrite a deferred update, and
clear it.

Like the immediate update, the NTO operand will take care
of applying the deferred updates, as such:

The NTO operand will gain another chance to process tuned objects
at startup, right before the main loops start.

If a tuned object is detected and it has NOT the deferred annotation:
- the regular reconcile code will work as today
- the startup reconcile code will not engage

If a tuned object is detected and it DOES HAVE the deferred annotation:
- the regular reconcile code will set a filesystem flag, will move
  the status DEGRADED and will ignore the object.
- at the first followup operand restart, the startup reconcile code
   will notice the filesystem flag, apply the rendered tuned objects
  (granted it DOES have the deferred flag, otherwise it will log and
  skip), clear the filesystem flag. The DEGRADED state will be cleared.

IOW, the startup reconcile code will engage IFF the rendered tuned
has the deferred annotation and the filesystem flag does exist.
Likewise, the regular reconcile loop will only set the filesystem
flag if detects a rendered tuned with the deferred annotation.

The default is obviously kept to immediate updates,
for backward compatibility.

Signed-off-by: Francesco Romani <[email protected]>

pkg/apis/tuned/v1/tuned_types.go

@@ -25,6 +25,10 @@ const (
 	// TunedBootcmdlineAnnotationKey is a Node-specific annotation denoting kernel command-line parameters
 	// calculated by TuneD for the current profile applied to that Node.
 	TunedBootcmdlineAnnotationKey string = "tuned.openshift.io/bootcmdline"
+
+	// TunedDeferredUpdate request the tuned daemons to defer the update of the rendered profile
+	// until the next restart
+	TunedDeferredUpdate string = "tuned.openshift.io/deferred"
 )
 
 /////////////////////////////////////////////////////////////////////////////////