OCPBUGS-44559: Manage TuneD profiles with the same name and different content

[jmencak]

It is possible to create Tuned CRs with TuneD profiles of the same name.
This is problematic when the duplicate TuneD profiles have different
contents. This can cause periodic switches between the duplicate TuneD
profiles and will lead to TuneD restarts.

This change makes the operator control loop stop to avoid switching
between the conflicting TuneD profiles of the same name and warns about
this misconfiguration via:

• Tuned CR status and prometheus metric (nto_invalid_tuned_exist_info)
• NTOInvalidTunedExist alert
• ERROR message issued in the operator logs

Other changes:

• Switch the ProfileStatusCondition condition type to
  StatusCondition. This is cosmetic only, all fields are preserved.
• Obsolete pod label-based matching. Trigger an alert
  (NTOPodLabelsUsed) for this.

Fixes: OCPBUGS-44559.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jmencak

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jmencak]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jmencak]

An example of creating duplicate profiles with conflicting content:
Profile 1

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: openshift-profile-dup
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
  - data: |
      [main]
      summary=Custom OpenShift profile
      include=openshift-node
      [sysctl]
      kernel.shmmni=8192
    name: openshift-profile
  recommend:
  - match:
    - label: profile
    priority: 20
    profile: openshift-profile

Profile 2

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: openshift-profile-dup2
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
  - data: |
      [main]
      summary=Custom OpenShift profile
      include=openshift-node
      [sysctl]
      kernel.shmmni=8191
    name: openshift-profile
  recommend:
  - match:
    - label: profile
    priority: 20
    profile: openshift-profile

$ oc get tuned
NAME                     VALID   AGE
default                  True    20h
openshift-profile-dup    False   20h
openshift-profile-dup2   False   20h

$ oc get tuned/openshift-profile-dup -o jsonpath='{.status}'        
{"conditions":[{"lastTransitionTime":"2024-12-02T10:58:39Z","message":"Duplicate TuneD profile \"openshift-profile\" with conflicting content detected.","reason":"InvalidConfiguration","status":"False","type":"Valid"}]}

[openshift-ci-robot]

@jmencak: This pull request explicitly references no jira issue.

In response to this:

It is possible to create Tuned CRs with TuneD profiles of the same name. This is problematic when the duplicate TuneD profiles have different content. While the problem is obvious when inspecting the operator logs, this configuration issue is serious enough it should be more visible.

This change makes the node-tuning ClusterOperator object Degraded when conflicting TuneD profiles are created. This in turn creates an alert for cluster administrators to deal with this misconfiguration.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jmencak]

/hold
Unfortunately, as is, this will not fix the issue, not even partly. The Tuned CRs can be retrieved in random order.

[jmencak]

/unhold

[openshift-ci-robot]

@jmencak: This pull request references Jira Issue OCPBUGS-44559, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.19.0) matches configured target version for branch (4.19.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira ([email protected]), skipping review request.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

It is possible to create Tuned CRs with TuneD profiles of the same name.
This is problematic when the duplicate TuneD profiles have different
contents. This can cause periodic switches between the duplicate TuneD
profiles and will lead to TuneD restarts.

This change makes the operator control loop stop to avoid switching
between the conflicting TuneD profiles of the same name and warns about
this misconfiguration via:

• Tuned CR status and prometheus metric (nto_invalid_tuned_exist_info)
• NTOInvalidTunedExist alert
• ERROR message issued in the operator logs

Other changes:

• Switch the ProfileStatusCondition condition type to
  StatusCondition. This is cosmetic only, all fields are preserved.
• Obsolete pod label-based matching. Trigger an alert
  (NTOPodLabelsUsed) for this.

Fixes: OCPBUGS-44559.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jmencak]

/retest

[jmencak]

/retest

[openshift-ci]

@jmencak: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[jmencak]

@MarSik , can you please take a look if this is the direction we want to go? I'd like to ask for a pre-merge testing from the QE.

[jmencak]

/hold
Until I add some e2e testing.