Keep getting CPUThrottlingHigh alert on the gc pod

[budimanjojo]

What happened:
After updating to v0.16.0, I keep getting CPUThrottlingHigh alert on the garbage collection pod like this:

CPUThrottlingHigh (Info)
Description: 35.71% throttling of CPU in namespace kube-system for container gc in pod node-feature-discovery-gc-696b644f9-2rwql.

What you expected to happen: Everything should be running like it used to be. I have fairly default helm values:

master:
  extraLabelNs:
    - gpu.intel.com

How to reproduce it (as minimally and precisely as possible): Use the latest v0.16.0 with the values above.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): v1.30.0
• Cloud provider or hardware configuration: baremetal
• OS (e.g: cat /etc/os-release): Talos Linux
• Kernel (e.g. uname -a): 6.6.29-talos
• Install tools: Helm
• Network plugin and version (if this is a network-related bug):
• Others:

[budimanjojo]

Maybe this line is set too low

node-feature-discovery/deployment/helm/node-feature-discovery/values.yaml

Line 533 in 560905f

cpu: 20m

Or maybe there's a bug in the garbage collection logic making it taking too many resources.

[marquiz]

Thanks @budimanjojo for reporting this. How big is your cluster (ca. how many nodes)?

In retrospect, setting the cpu limits might not have been that good idea. We might want to remove those (and cut a patch release) 🧐

The most immediate fix for you would probably be to remove the cpu limits, i.e. do Helm install with --set gc.resources.limits.cpu=null

[budimanjojo]

Hi @marquiz!
I have a 3 nodes cluster so it's a pretty small one.

Yeah I agree with having no CPU limits set at least in the gc pod by default. Should I open a PR or I'll just wait?

[marquiz]

I have a 3 nodes cluster so it's a pretty small one.

OK, not a huge one, then. 😅 Looks like we need to investigate that a bit further 🤔

Yeah I agree with having no CPU limits set at least in the gc pod by default. Should I open a PR or I'll just wait?

Please do, more contributors -> better 😊 Let's remove cpu limits for all daemons. Also, we need to update the tables of parameters in docs/deployment/helm.md, accordingly (for the defaults)

[budimanjojo]

@marquiz I just created the PR, please take a look. I removed CPU limits for all daemons instead of just the garbage collection pod according to your recommendation.