Remove containers from deleted pods once containers have exited

[dashpole]

Issue #51899
Since container deletion is currently done through periodic garbage collection every 30 seconds, it takes a long time for pods to be deleted, and causes the kubelet to send all delete pod requests at the same time, which has performance issues. This PR makes the kubelet actively remove containers of deleted pods rather than wait for them to be removed in periodic garbage collection.

Fixes a performance issue (#51899) identified in large-scale clusters when deleting thousands of pods simultaneously across hundreds of nodes, by actively removing containers of deleted pods, rather than waiting for periodic garbage collection and batching resulting pod API deletion requests.

[dashpole]

/release-note-none

[dashpole]

I tested this PR by doing the following:
I disabled periodic container gc by commenting out that code (just for my testing). I did this to ensure this PR is what is causing containers to be cleaned up.
I ran a node-e2e test which creates and deletes a pod.

I checked the kubelet log during the test, and found that the pod is stopped, cleaned up, and removed within 2 seconds of observing the deletion timestamp:
16:55:59 SyncLoop (DELETE ...
16:56:00 GenericPLEG ... running -> exited
16:56:00 Removing container ...
16:56:01 Volume detached for volume ...
16:56:01 GenericPLEG ... exited -> non-existent
16:56:01 ... fully terminated and removed from etcd

[dashpole]

flake seems unrelated:
error: failed to run Kubelet: cannot create certificate signing request: Post https://35.184.83.219/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 35.184.83.219:443: getsockopt: connection refused

[dashpole]

/test pull-kubernetes-e2e-gce-etcd3

[shyamjvs]

Seems to help quite a bit:

without the PR:

Oct  3 17:14:21.822: INFO: WARNING Top latency metric: {Resource:pods Subresource: Verb:DELETE Scope:namespace Latency:{Perc50:4.445ms Perc90:612.189ms Perc99:5.041177s Perc100:0s} Count:124002}

with the PR:

Oct  3 14:12:06.169: INFO: Top latency metric: {Resource:pods Subresource: Verb:DELETE Scope:namespace Latency:{Perc50:3.77ms Perc90:8.846ms Perc99:426.462ms Perc100:0s} Count:124000}

[shyamjvs]

@dashpole @dchen1107 Can we have this PR in soon'ish? Need to get the scale tests back to green.

[smarterclayton]

No further comments from me?

[wojtek-t]

LGTM

/approve no-issue

[wojtek-t]

Once merged we should cherrypick it to 1.8

[dchen1107]

/lgtm

[k8s-github-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dashpole, dchen1107, wojtek-t

Associated issue: 51899

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

• pkg/kubelet/OWNERS [dchen1107,wojtek-t]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[k8s-github-robot]

/test all [submit-queue is verifying that this PR is safe to merge]

[k8s-github-robot]

Automatic merge from submit-queue (batch tested with PRs 53403, 53233). If you want to cherry-pick this change to another branch, please follow the instructions here.

[liggitt]

pick to 1.8 opened at #53422, and added release note (please review/tweak language as desired)

[k8s-cherrypick-bot]

Commit found in the "release-1.8" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this is an error find help to get your PR picked.