Karpenter can't delete node if Kubelet crashes

[tomjohnburton]

Description

Observed Behavior:
We had a pod that as soon as it launched (and due to not setting resource limits) ate up the entire CPU and memory of the node it was running on. The node would go into the NotReady state and the pods would fail to be terminated. In the Karpenter logs, it was as if nothing happened. Karpenter doesn't make any attempt to recover the node.

Expected Behavior:
Karpenter should intervene in this case and try to spin up a new node.

Reproduction Steps (Please include YAML):
Create a pod without resource limits that immediately steals all the memory and cpu of the node. Kubelet should crash. Karpenter won't even register there is an issue.

Versions:

• Chart Version: 0.36.0
• Kubernetes Version (kubectl version): 1.29

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[tzneal]

Closing as a duplicate of the kubernetes-sigs/karpenter#750 issue.