Graduate Evented PLEG to Beta #39913

harche · 2023-03-10T17:20:19Z

Signed-off-by: Harshal Patil [email protected]

Add EventedPLEG to FeatureGates doc.

Corresponding PR - kubernetes/kubernetes#115967
Enhancement - kubernetes/enhancements#3386

content/en/docs/reference/command-line-tools-reference/feature-gates.md

sftim

(I think that) you should also update https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/ to mention this beta feature gate and why you might care about it.

If that's not the right page to update, we should find which is the right page and update that instead. For beta features, even those disabled by default, we nearly document why you might want to turn them on and the work you need to do associated with enabling the feature.

If you think there are tasks to document - eg: migrate to using CRI notifications for Pod events - then please add that task page either in this PR or in a separate PR that also mentions the relevant KEP.

content/en/docs/reference/command-line-tools-reference/feature-gates.md

harche · 2023-03-13T16:07:05Z

Thanks @harche

(I think that) you should also update https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/ to mention this beta feature gate and why you might care about it.

If that's not the right page to update, we should find which is the right page and update that instead. For beta features, even those disabled by default, we nearly document why you might want to turn them on and the work you need to do associated with enabling the feature.

IMO, the right place for such page would https://kubernetes.io/docs/concepts/configuration/

I will add a page describing Evented PLEG and its usage. Thanks.

sftim · 2023-03-13T16:10:10Z

IMO, the right place for such page would https://kubernetes.io/docs/concepts/configuration/

That isn't the best home. If you want to document pod lifecycle events as a concept, put that in https://kubernetes.io/docs/concepts/containers/ or https://kubernetes.io/docs/concepts/workloads/pods/

If you want to explain the cluster operator angle, the content should live in either https://kubernetes.io/docs/concepts/cluster-administration/ or https://kubernetes.io/docs/tasks/administer-cluster/ (make the choice depending on whether you are explaining how to perform a task, such as migrating to CRI notifications, or providing general context).

harche · 2023-03-13T16:19:42Z

IMO, the right place for such page would https://kubernetes.io/docs/concepts/configuration/

That isn't the best home. If you want to document pod lifecycle events as a concept, put that in https://kubernetes.io/docs/concepts/containers/ or https://kubernetes.io/docs/concepts/workloads/pods/

If you want to explain the cluster operator angle, the content should live in either https://kubernetes.io/docs/concepts/cluster-administration/ or https://kubernetes.io/docs/tasks/administer-cluster/ (make the choice depending on whether you are explaining how to perform a task, such as migrating to CRI notifications, or providing general context).

It's a configuration of the kubelet (and CRI Runtime) that affects how it maintains its internal cache of pod statuses. That's the reason I thought configuration made sense. This is not directly connected to any concepts related to containers or pods, neither is it related to any operator or cluster administration.

harche · 2023-03-13T16:21:52Z

Coming to think of it, maybe https://kubernetes.io/docs/tasks/administer-cluster/ makes sense.

sftim · 2023-03-13T16:43:32Z

/sig node

harche · 2023-03-14T15:16:33Z

/hold

harche · 2023-03-14T17:30:41Z

/hold cancel

sftim

Thanks. This will need some work before we can merge it, I'm afraid. For example, I think we should advice readers to drain the nodes they're reconfiguring, or at least empower them to consider doing that.

What do you think @harche ?

content/en/docs/tasks/administer-cluster/switch-to-evented-pleg.md

sftim · 2023-03-14T17:33:48Z

content/en/docs/tasks/administer-cluster/switch-to-evented-pleg.md

+1. Start the `CRI Runtime` with the `Evented PLEG` support. e.g.
+   1. Containerd version 1.7+ 
+   2. CRI-O v1.26+ and also, start the cri-o daemon with the flag `--enable-pod-events=true` or using a drop in config like,
+      ```toml
+      [crio.runtime]
+      enable_pod_events: true
+      ```


When we're documenting for different container runtimes, we quite often use tabs to let readers click to what they're using.

Updated. Thanks.

sftim · 2023-03-14T17:34:05Z

content/en/docs/tasks/administer-cluster/switch-to-evented-pleg.md

+## Why switch to Evented PLEG?
+
+* The current `Generic PLEG` incurs non-negligible overhead due to frequent polling of container statuses.
+* This overhead is exacerbated by Kubelet's parallelism, limiting its scalability and causing poor performance and reliability problems.
+* The goal of `Evented PLEG` is to reduce unnecessary work during inactivity by replacing periodic polling.


This should go before the list of prerequisites.

sftim · 2023-03-14T17:34:32Z

content/en/docs/tasks/administer-cluster/switch-to-evented-pleg.md

+
+
+{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
+


Do you need to enable any feature gates on the control plane?

Do you need a specific Kubernetes version?

Do you need administrator-level access to the node(s) you're managing?

Do you need a container runtime that has support for a particular CRI feature?

and we should also cover, somewhere in this page:

do you need to roll this out to all nodes at once, or can you phase it?

Do you need to enable any feature gates on the control plane?
No.

Do you need a specific Kubernetes version?

1.26+

Do you need administrator-level access to the node(s) you're managing?

Yes

Do you need a container runtime that has support for a particular CRI feature?

Yes

and we should also cover, somewhere in this page:

do you need to roll this out to all nodes at once, or can you phase it?

You can phase it. It is only a node specific feature that affects how the given kubelet manages the pod status cache internally.

sftim · 2023-03-14T17:34:47Z

content/en/docs/tasks/administer-cluster/switch-to-evented-pleg.md

+      $ grep  EventedPLEG /tmp/kubelet.log 
+      I0314 11:10:13.909915 1105457 feature_gate.go:249] feature gates: &{map[EventedPLEG:true]}


That's not a shell script. See https://kubernetes.io/docs/contribute/style/style-guide/#separate-commands-from-output

content/en/docs/tasks/administer-cluster/switch-to-evented-pleg.md

sftim · 2023-03-14T17:38:33Z

content/en/docs/tasks/administer-cluster/switch-to-evented-pleg.md

+* The current `Generic PLEG` incurs non-negligible overhead due to frequent polling of container statuses.
+* This overhead is exacerbated by Kubelet's parallelism, limiting its scalability and causing poor performance and reliability problems.
+* The goal of `Evented PLEG` is to reduce unnecessary work during inactivity by replacing periodic polling.
+


Should you drain the node first?

harche · 2023-03-15T15:47:47Z

Thanks. This will need some work before we can merge it, I'm afraid. For example, I think we should advice readers to drain the nodes they're reconfiguring, or at least empower them to consider doing that.

What do you think @harche ?

yes, it makes sense.

harche · 2023-03-15T15:52:25Z

@sftim Thanks for your valuable feedback. I have updated the PR with those recommendations.

sftim

Some more feedback

sftim · 2023-03-15T15:52:18Z

content/en/docs/tasks/administer-cluster/switch-to-evented-pleg.md

@@ -0,0 +1,76 @@
+---
+title: Switching from polling to CRI event-based updates to container status


Suggested change

title: Switching from polling to CRI event-based updates to container status

title: Switching From Polling to CRI Event-based Updates to Container Status

sftim · 2023-03-15T15:54:27Z

content/en/docs/tasks/administer-cluster/switch-to-evented-pleg.md

+{{< feature-state for_k8s_version="v1.26" state="beta" >}}
+
+<!-- overview -->
+This page shows how to use `Evented PLEG` (CRI event-based updates to container status) in Kubelet which attempts to lower the resource consumption by avoiding frequent polling for container statuses.


Suggested change

This page shows how to use `Evented PLEG` (CRI event-based updates to container status) in Kubelet which attempts to lower the resource consumption by avoiding frequent polling for container statuses.

This page shows how to migrate nodes to use event based updates for container status. The event-based

implementation reduces node resource consumption by the kubelet, compared to the legacy approach

that relies on polling.

You may know this feature as _evented Pod lifecycle event generator (PLEG)_. That's the name used

internally within the Kubernetes project for a key implementation detail.

?

sftim · 2023-03-15T15:56:10Z

content/en/docs/tasks/administer-cluster/switch-to-evented-pleg.md

+* This support for this feature is available in Kubernetes {{< skew currentVersion >}};
+  Kubernetes v1.26 as a beta feature, disabled by default.


Suggested change

* This support for this feature is available in Kubernetes {{< skew currentVersion >}};

Kubernetes v1.26 as a beta feature, disabled by default.

You need to run a version of Kubernetes that provides this feature.

Kubernetes {{< skew currentVersion >}} includes beta support for event-based container

status updates. The feature is beta and is disabled by default.

{{< version-check >}}

If you are running a different version of Kubernetes, check the documentation for that release.

sftim · 2023-03-15T15:56:24Z

content/en/docs/tasks/administer-cluster/switch-to-evented-pleg.md

+
+* This support for this feature is available in Kubernetes {{< skew currentVersion >}};
+  Kubernetes v1.26 as a beta feature, disabled by default.
+* Please make sure the node is [drained](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/) before proceeding. 


This should be part of the steps and not a prerequisite.

sftim · 2023-03-15T15:57:09Z

content/en/docs/tasks/administer-cluster/switch-to-evented-pleg.md

+* Start the `CRI Runtime` with the `Evented PLEG` support. 
+      {{< tabs name="tab_with_code" >}}
+      {{< tab name="Containerd" codelang="bash" >}}
+      Version 1.7+
+      {{< /tab >}}
+      {{< tab name="CRI-O" codelang="bash" >}}
+      v1.26+ and also, start the cri-o daemon with the flag `--enable-pod-events=true` or using a drop in config like,
+
+      [crio.runtime]
+      enable_pod_events: true
+
+      {{< /tab >}}
+      {{< /tabs >}}


At this point, we should have the reader check that their runtime can support events. They don't have to have already reconfigured it (eg CRI-O and enable_pod_events).

sftim · 2023-03-15T15:57:42Z

content/en/docs/tasks/administer-cluster/switch-to-evented-pleg.md

+
+
+1. Start the Kubelet with the [feature gate](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) `EventedPLEG` enabled. 
+


We could hint that you do this by editing its configuration file, and then you restart the kubelet.

sftim · 2023-03-15T15:57:55Z

content/en/docs/tasks/administer-cluster/switch-to-evented-pleg.md

+2. Verify that `Evented PLEG` is in use: 
+
+      ```
+      $ grep  EventedPLEG /tmp/kubelet.log 


Can we rely on the log path being /tmp?

Signed-off-by: Harshal Patil <[email protected]> Co-authored-by: Tim Bannister <[email protected]>

harche · 2023-03-16T15:23:05Z

Thanks @sftim again for those suggestions.

Considering the significant amount of suggestions provided by I have added you as a Co-author in the commit, hope that is fine by you.

tengqm · 2023-03-31T03:01:48Z

/approve

k8s-ci-robot · 2023-03-31T03:02:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tengqm

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~content/en/OWNERS~~ [tengqm]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

reylejano · 2023-04-03T15:25:19Z

/lgtm

k8s-ci-robot · 2023-04-03T15:25:27Z

LGTM label has been added.

Git tree hash: 04ce26686dd5fdeee2f809aad148abd193f496dd

k8s-ci-robot added this to the 1.27 milestone Mar 10, 2023

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. language/en Issues or PRs related to English language sig/docs Categorizes an issue or PR as relevant to SIG Docs. labels Mar 10, 2023

k8s-ci-robot requested review from sftim and shannonxtreme March 10, 2023 17:20

harche mentioned this pull request Mar 10, 2023

Kubelet Evented PLEG for Better Performance kubernetes/enhancements#3386

Open

25 tasks

tengqm reviewed Mar 12, 2023

View reviewed changes

content/en/docs/reference/command-line-tools-reference/feature-gates.md Show resolved Hide resolved

sftim reviewed Mar 13, 2023

View reviewed changes

content/en/docs/reference/command-line-tools-reference/feature-gates.md Show resolved Hide resolved

k8s-ci-robot added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Mar 13, 2023

harche force-pushed the evented_pleg_beta branch from f9e8e9d to f9df2ac Compare March 14, 2023 15:16

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Mar 14, 2023

harche force-pushed the evented_pleg_beta branch from f9df2ac to dfd2fdc Compare March 14, 2023 17:30

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 14, 2023

sftim reviewed Mar 14, 2023

View reviewed changes

harche force-pushed the evented_pleg_beta branch from dfd2fdc to 643746c Compare March 15, 2023 15:47

sftim reviewed Mar 15, 2023

View reviewed changes

Graduate Evented PLEG to Beta

efa8181

Signed-off-by: Harshal Patil <[email protected]> Co-authored-by: Tim Bannister <[email protected]>

harche force-pushed the evented_pleg_beta branch from 643746c to efa8181 Compare March 16, 2023 15:21

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 31, 2023

k8s-ci-robot assigned reylejano Apr 3, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 3, 2023

k8s-ci-robot merged commit aff98a0 into kubernetes:dev-1.27 Apr 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Graduate Evented PLEG to Beta #39913

Graduate Evented PLEG to Beta #39913

harche commented Mar 10, 2023

sftim left a comment

harche commented Mar 13, 2023

sftim commented Mar 13, 2023

harche commented Mar 13, 2023

harche commented Mar 13, 2023

sftim commented Mar 13, 2023

harche commented Mar 14, 2023

harche commented Mar 14, 2023

sftim left a comment

sftim Mar 14, 2023

harche Mar 15, 2023

sftim Mar 14, 2023

sftim Mar 14, 2023

harche Mar 15, 2023

sftim Mar 14, 2023

sftim Mar 14, 2023

harche Mar 15, 2023

harche commented Mar 15, 2023

harche commented Mar 15, 2023

sftim left a comment

sftim Mar 15, 2023

sftim Mar 15, 2023 •

edited

Loading

sftim Mar 15, 2023

sftim Mar 15, 2023

sftim Mar 15, 2023

sftim Mar 15, 2023

sftim Mar 15, 2023

harche commented Mar 16, 2023

tengqm commented Mar 31, 2023

k8s-ci-robot commented Mar 31, 2023

reylejano commented Apr 3, 2023

k8s-ci-robot commented Apr 3, 2023



		{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}

		$ grep EventedPLEG /tmp/kubelet.log
		I0314 11:10:13.909915 1105457 feature_gate.go:249] feature gates: &{map[EventedPLEG:true]}

		@@ -0,0 +1,76 @@
		---
		title: Switching from polling to CRI event-based updates to container status

-This page shows how to use `Evented PLEG` (CRI event-based updates to container status) in Kubelet which attempts to lower the resource consumption by avoiding frequent polling for container statuses.
+This page shows how to migrate nodes to use event based updates for container status. The event-based
+implementation reduces node resource consumption by the kubelet, compared to the legacy approach
+that relies on polling.
+You may know this feature as _evented Pod lifecycle event generator (PLEG)_. That's the name used
+internally within the Kubernetes project for a key implementation detail.

		* This support for this feature is available in Kubernetes {{< skew currentVersion >}};
		Kubernetes v1.26 as a beta feature, disabled by default.

-* This support for this feature is available in Kubernetes {{< skew currentVersion >}};
-  Kubernetes v1.26 as a beta feature, disabled by default.
+You need to run a version of Kubernetes that provides this feature.
+Kubernetes {{< skew currentVersion >}} includes beta support for event-based container
+status updates. The feature is beta and is disabled by default.
+{{< version-check >}}
+If you are running a different version of Kubernetes, check the documentation for that release.



		1. Start the Kubelet with the [feature gate](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) `EventedPLEG` enabled.

Graduate Evented PLEG to Beta #39913

Graduate Evented PLEG to Beta #39913

Conversation

harche commented Mar 10, 2023

sftim left a comment

Choose a reason for hiding this comment

harche commented Mar 13, 2023

sftim commented Mar 13, 2023

harche commented Mar 13, 2023

harche commented Mar 13, 2023

sftim commented Mar 13, 2023

harche commented Mar 14, 2023

harche commented Mar 14, 2023

sftim left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

harche commented Mar 15, 2023

harche commented Mar 15, 2023

sftim left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sftim Mar 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

harche commented Mar 16, 2023

tengqm commented Mar 31, 2023

k8s-ci-robot commented Mar 31, 2023

reylejano commented Apr 3, 2023

k8s-ci-robot commented Apr 3, 2023

sftim Mar 15, 2023 •

edited

Loading