[FEATURE]: CSM Resiliency supports evacuation of pods during NoExecute taint on node

[eanjtab]

Describe the feature

The original Resiliency design refused to force delete pods if they are potentially doing I/O to the array. The customer's request is a valid one, although not quite as safe as the current behavior of resiliency. I would propose that we make this behavior an option, i.e:

• A new configuration variable is introduced. "forceDeleteOnNoExecuteTaint" or something similar, that would change the current behavior so that if podmon receives a notification that the pods is Not Ready and the node has a "NoExecute" taint, we force delete the pod. The default would be false, resulting in the current behavior.
• If the pod has a grace period toleration for NoExecute (as most do, the default is 5 minutes), we need to do it before the grace period expires so that no replacement pods will be created on the same node being evacuated because of pod affinity. You could do it immediately upon receiving the NoExecute notification, or perhaps wait 1/2 the duration of the toleration (normally 2-3 minutes) in case the node becomes ready rather quickly.

Additional context
This feature has been converted from a bug. Logs have been attached.

session-logs.txt

[rbo54]

Hello, my analysis shows that podmon is working as designed. Please see the analysis write up in
CSM-87-analysis.txt

[gallacher]

Closing issue since it works as designed.

[eanjtab]

Hello Tom,

I saw the response, so since NoExec taint was not placed the podmon did not actually migrate the pods. If NoExec was placed then it will migrate ?

[rbo54]

Hello,
If you look at my analysis text, you will see podmon received two notifications from kubernetes, the first pod was not ready and there was a nosched taint but not a noexec taint: time="2021-11-02T12:11:31Z" level=info msg="podMonitorHandler: namespace: pmtv1 name: podmontest-85fcc7957c-c94mg nodename: twk8s-10-247-102-219 initialized: true ready: false taints [nosched: true noexec: false podmon: false ]". The second had both nosched and noexec true: time="2021-11-02T12:16:35Z" level=info msg="podMonitorHandler: namespace: pmtv1 name: podmontest-85fcc7957c-c94mg nodename: twk8s-10-247-102-219 initialized: true ready: false taints [nosched: true noexec: true podmon: false ]"
Podmon responds the same either way, if it detects the node is connected to array, and especially if I/O is going on, it aborts the cleanup, because this is the safest course.
An option we could consider would be to force a reboot of the node if a noexec taint was received and we needed to clean a pod but it had connection or I/O. Effectively that is what k8s is sort of trying to do anyway by sending noexec, i.e. get things off the node.
Let me know your thoughts.
Tom

[eanjtab]

Thanks Tom for the explanation, so I see that in either case NoSched or NoExec since the podmon detects that array connectivity is established and good it does not take any action. But I feel we need to change this now. Because there could be different situations where kube-api would place these taints in which we will end up pods stuck in terminating phase. Like for eg: - if the internal HB K8 network goes down then also we may end up with similar situation.

So definitely a good option here would be to look at the reboot option when the NoExec taint is placed on that node. Who will trigger the reboot ? Kube-api or can podmon do it ?

Please see my messages on Team, do you have time for a quick call ?

[rbo54]

We are currently testing a change to address this and plan to ship it in the next release at end Q1 2022. At the current time, we are not planning to automatically reboot the node to fix this, but rather allow the pods to be deleted if we receive a NoExecute taint and SkipArrayConnectionValidation option is set to true.

[alikdell]

Functionality will be available in CSM for Resiliency next release