Check DoNotEvict after filtering evictable pods to ensure termination can complete.

[ellistarn]

1. Issue, if available:
#1166

2. Description of changes:
Previously, thedo-not-evict annotation could block termination logic, even if the node was unreachable. This change modifies the do-not-evict logic to only apply to evictable pods. Currently, evictable pods exclude

• Pods owned by the node (a.k.a. static pods)
• Pods with deletion time stamp set past their graceful termination period (a.k.a. stuck termination)
• Pods that tolerate the nodes taints (i.e. would reschedule immediately)

This means that the do-not-evict annotation will no longer work for pods in the above categories. Given that we already did not attempt to evict these pods, the semantic holds. However, previously, pods in these categories with the do-not-evict annotation could block node termination.

3. How was this change tested?

Reproduced

1. Added do not evict annotation to pod
2. Manually terminated a node in EC2
3. Observe Node and Pod stuck Terminating

Fixed

1. Start with step 3 of above scenario
2. Apply PR changes to local environment
3. Node is gracefully terminated

4. Does this change impact docs?

• Yes, PR includes docs updates
• Yes, issue opened: link to issue
• No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[netlify]

✔️ Deploy Preview for karpenter-docs-prod canceled.

🔨 Explore the source changes: 7d00e1e

🔍 Inspect the deploy log: https://app.netlify.com/sites/karpenter-docs-prod/deploys/6203106ef4a6030008a3d54e

[njtran]

Consider testing static pods and a pod that tolerates NoSchedule, should just be two more pods creation lines, and adding to the respective lines below.

[njtran]

Thanks for the fix! Looks good!