Set limits and requests for exclusive pool containers

[Levovar]

Similarly to shared pool Pods, we need to care about setting requests, and limits for exclusives.
Request: explicitly setting 0 is required, similarly to how it was done for shareds in #14

Limit: we should either set explicit 0 to avoid artificially limiting the share time of a core which is anyway physically isolated; or we need to set (number_of_exclusive_cores)X1000.
Reasoning for 0: even if the limit is actually higher than what is possible to get, it will still add a CFS quota to the container. There are anecdotal reports out there indicating just the presence of a CFS quota can negatively affect performance.
Reasoning for explicitly setting limit: there are some admission controllers in K8s which mandate a Pod to define requests, and limits. These mandated requests are ofc pointless in case of an exclusive user, but failure to comply results in failing Pod admission.
So, if the presence of CFS quotas do not affect performance, it is actually the safer option to set the limit!

@TimoLindqvist : WDYT Timo?

[TimoLindqvist]

I'm under the impression that CFS quota can affect performance / latency so we should configure it so that CFS quota is disbaled. Is it disabled if we set the CPU limit to zero ?

[Levovar]

K8s community actually proposed kernel patches to correct this issue :)
so I think we shouldn't disable CFS quotas. they should work much better from 4.14, and as intended in the latest versions.

[Levovar]

sorry, first improvement is available from 4.18:
torvalds/linux@512ac99#diff-1c5364196d98130348bddabaad0a701f

And this one should totally fix them:
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=de53fd7aedb100f03e5d2231cfce0e4993282425

BTW based on the description the CFS quota was only misbehaving for workloads which frequently idle, and only consume some slices.
our exclusive workloads are quite the opposite, they always use 100% of the core

[TimoLindqvist]

I'm still a bit against setting the limit to something else than 0 thus enabling the CFS quota. If container allocates exclusive core(s), it must have full access to the core(s). On the other hand it cannot use other cores so why is the quota needed ?

[Levovar]

On the other hand it cannot use other cores so why is the quota needed ?

purely for Kubernetes compatibility reasons I described in the opening post. If the user has LimitRanger admission plugins enabled in their cluster, instantiating Pods within their CPU-Pooler enhanced cluster will fail:
https://kubernetes.io/docs/concepts/policy/limit-range/#overview-of-limit-range

I think the same might be an issue when the operator would set some constraints in the Namespace object

[TimoLindqvist]

We can add the limit and requests to exclusive containers to avoid failures in pod instantiation but would it be ok to set the limit to zero ?