[FEATURE]: Monitor CSI Driver node pods failure in CSM for Resiliency so that pods are not scheduled on that node

[eanjtab]

Describe the feature
The CSI node driver when not running on a WN (worker node). the CSI controller should check and taint that node so that pods are not scheduled on that node.

Feature functionality
When CSI Driver node pods are not running, CSM for Resiliency should taint that node so that new pods are not scheduled in that node. This will prohibits pods to fail where the starting or running of that pod/s need CSI Driver node pods to be running on that node.

Customer observation:

I am not sure how to exactly create this situation, but we do have a situation currently where CSI node driver pods are in "CrashLoopBackOff" state on WN 01 and 04 but there is no Taint on those nodes as a result the pods are scheduled on that node and fails due to unable to mount the volume. Please refer to the below output: -

(blccdmm02)[ccdadmin@BLcCCDADM01 custom-scripts]$ kubectl describe node mm-pool01-blccdmm02-wn001 |grep -i Taint
Taints:
(blccdmm02)[ccdadmin@BLcCCDADM01 custom-scripts]$ kubectl describe node mm-pool02-blccdmm02-wn004 |grep -i Taint
Taints:
(blccdmm02)[ccdadmin@BLcCCDADM01 custom-scripts]$
(blccdmm02)[ccdadmin@BLcCCDADM01 custom-scripts]$
(blccdmm02)[ccdadmin@BLcCCDADM01 custom-scripts]$ kubectl get pods -A |grep -i csi
kube-system csi-powerflex-controller-6b56d8fc9c-cgkvp 6/6 Running 0 3d15h
kube-system csi-powerflex-controller-6b56d8fc9c-fpr6j 6/6 Running 2 3d15h
kube-system csi-powerflex-node-5r5gq 3/3 Running 0 3d15h
kube-system csi-powerflex-node-cm6d5 3/3 Running 0 3d15h
kube-system csi-powerflex-node-dddqb 3/3 Running 0 3d15h
kube-system csi-powerflex-node-jmbh8 2/3 CrashLoopBackOff 482 2d
kube-system csi-powerflex-node-qjc84 3/3 Running 7 3d15h
kube-system csi-powerflex-node-w4hpl 2/3 CrashLoopBackOff 482 2d
kube-system csi-powerflex-node-w9lmf 3/3 Running 0 3d15h
(blccdmm02)[ccdadmin@BLcCCDADM01 custom-scripts]$

Expected behavior
CSI controller pod should check the status of CSI node pods and place a taint on the node if the CSI node pod is not running or crashed.

Screenshots
NA

Logs
Attached

System Information (please complete the following information):

• OS/Version: SUSE, 15-SP2
• Kubernetes Version: v1.21.1

Additional context
For some reason I have the logs collected by the collect_log,.sh script but unable to attach it here. I can always share it on mail. Please contact me on mail.

[hoppea2]

Updated the label to reflect correct module. We are reviewing this and will update shortly

[hoppea2]

We have added this to our product roadmap. No milestone has yet been determined.

[shanmydell]

@hoppea2 : If no milestone has been determined, we need to remove the milestone associated. Reflecting correct status