(Re-)Enable resource reservation by default for core components and host container runtime

[Tristan971]

What would you like to be added:

Considering Kubespray's "deploy a production ready Kubernetes cluster" slogan, it seems like we should always restrain the cgroup slice of Kubernetes pods to not consume ALL CPU and memory resources of the host.

Why is this needed:

After #9209, which is a neat change (nothing against it, basically), Kubespray's default is now to not reserve any CPU/memory for the host and core kubernetes components anymore. And that is a bit of a downgrade I feel.

Running the upgrade in one of our dev clusters I noticed it in the allocatable resources being equal to the host resources, which is obviously somewhat concerning, at least with regards to memory, as we'd obviously prefer to never see the Linux OOMKiller see any action ever if possible...

I appreciate that this somehow now depends on the usage of systemd with cgroups support, but it still seems quite useful as a default to me?

If not, we should at least document it more prominently in the 2.21 release notes, as reserving 200m of CPU and 512Mi of Memory was the previous default

[Tristan971]

Ok so after a week of running this in production, I feel like either I missed some critical piece of information, or it works VERY differently from what it feels like it should.

An image is worth a thousand words, so:

Unless I'm mistaken, it doesn't just reserve memory, but also enforces the limits of the cgroups, including the system slice's.

So you can very well end up with OOMkills on your control plane nodes' etcd... despite plenty of memory being available (here in the section of time for the past week; plenty of free memory, at the cost of OOMkill'd etcds...):

While I appreciate this works in a similar fashion to k8s' memory limits the whole part of naming it system_reserved, system_master_memory_reserved, etc is... confusing to me.

My understanding (which was clearly wrong) was that those reservations only took away from the pods slice, rather than applying as hard limits...

Makes me a bit uneasy overall.

[Tristan971]

To note, my guess is that the "culprit" is actually kubelet_enforce_node_allocatable after reading https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#enforcing-node-allocatable closer

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rptaylor]

I agree, this seems to be a serious issue.

Before #9209 the default result in /etc/kubernetes/kubelet-config.yaml included

kubeReserved:
  cpu: 100m
  memory: 256Mi

Referring to official docs (capitalization mine for emphasis)

To OPTIONALLY enforce kubeReserved on kubernetes system daemons, specify the parent control group for kube
daemons as the value for kubeReservedCgroup setting, and add kube-reserved to enforceNodeAllocatable.

My interpretation is that defining resources for KubeReserved simply reduces the node allocatable resources by a small amount, to ensure that pods don't overwhelm a node. This is a simple scheduling matter: allocatable resources = (node resources) - (kube reserved) - (system reserved)
Preserving some node resources for critical daemons should be a default behaviour as it is very important for stability of production clusters.

As the doc notes, an optional additional step can be taken, to define a separate cgroup for k8s daemons to run in. But doing this, and actually creating that cgroup, aside from being optional and unnecessary, can be dangerous if it restricts the resource usage of k8s daemons.

In other words I think we need the following as default behaviour:

1. Some baseline kubeReserved resources should be specified by default (like a k8s resource request for important daemons).
2. Enforcement or limitation of the resource usage of important daemons can cause instability. So it seems to me a kubeReserved cGroup should NOT be specified by default (like a k8s resource limit that can cause important daemons to be killed).

[Tristan971]

As the doc notes, an optional additional step can be taken, to define a separate cgroup for k8s daemons to run in. But doing this, and actually creating that cgroup, aside from being optional and unnecessary, can be dangerous if it restricts the resource usage of k8s daemons.

That would explain the strange behaviors I ended up seeing indeed.

I sort of gave up and massively increased RAM capacity to work around it as I wasn’t able to come up with a sensible solution at the time…

But if having a kube system cgroup causes things like etcd to be OOMkilled from going over it, it is indeed more harmful than anything else.

[rptaylor]

@Tristan971 What value of kubelet_enforce_node_allocatable did you have? Based on the documentation it should only apply to pods by default, but it's a bit ambiguous because kubeReservedCgroup is described as "enforcing kubeReserved on kubernetes system daemons". So I'm not sure if kubeReservedCgroup is sufficient to explain the etcd OOM killing you noticed, or whether kubelet_enforce_node_allocatable would also have to include "kube-reserved".

[Tristan971]

What value of kubelet_enforce_node_allocatable did you have?

It appears that I also decided to not gamble on it, as I see the following diff in the commit where I changed it:

- kubelet_enforce_node_allocatable: "pods,kube-reserved,system-reserved"
+ kubelet_enforce_node_allocatable: "pods"

Additionally at the same time

- system_master_memory_reserved: 512Mi
+ system_master_memory_reserved: 4Gi

This was more than 1.5 years ago however, so I unfortunately cannot recall whether both were necessary or not or if I was simply trying to get the issue to go away.

[rptaylor]

kubelet_enforce_node_allocatable: "pods,kube-reserved,system-reserved" would explain the killing of the system daemons. The default is just pods.

#11367 will fix the issue of the disappearing reservation.

[Tristan971]

Thanks for the PR and pushing through with this! 👍