Additional discussions for CSI volume health proposal

[Aber4Nod]

Additional discussions for CSI volume health proposal: #677

[mishoyama]

@Aber4Nod several questions about your proposal:

1. Do we need to change existing disk replacement procedure?
2. What is the API to detect PV failure for the user? Only event or annotation as well?
3. When health of drive is UNKNOWN should we also report failure?
4. What are the conditions to move to the FAILED CSIStatus?
5. When OperationalStatus is MISSING do you suggest to report abnormal value right away or with some delay?
6. Usage FAILED is reported during disk replacement procedure. For example when application cannot finish data recovery. It looks strange that such volumes will be reported abnormal
7. Do we need to add mount status to Volumes CRD? How PV can be unmounted when pod is running? Are you referring to the VOL_READY or PUBLISHED state? What is the flow for checking that volume is mounted? Does presence of mount point guarantee mounted status?
8. Is ReadDir is enough to detect IO errors? Won't you read from system cache?
9. How often do you suggest to check FS consistency?

[Aber4Nod]

1. Currently there is no need in changing DR procedure. But in future it may be preferable to unbind other components from CSI Volume CR and use standard k8s mechanisms for interaction. E.g. objectscale-operator receives notification of unhealthy volume via CSI volume health mechanisms (which proposal we are currently reviewing) and sends the similar events about DR procedure status, by which CSI updates Volume CR by itself.
2. There are two sources of failure:
   2.1. By kubelet: If a PVC has an abnormal health condition, an event will to reported on the pod object that is using the PVC;
   2.2. By external monitoring controller: Monitoring controller reports events on the PVCs directly.
3. We should send the notification if we cannot find already existent drive (it's status will be offline and so volume operational status will be missing). Another scenario of unknown drive health is when we failed to get drive health (via nvme cli) of failed to find drive's health field via idrac - in this case we should send CSI internal notification/event (as we are doing already), user has nothing to deal with this.
4. There are plenty a lot of cases when volume moves to FAILED CSIStatus (e.g. volume preparation/mount failed).
5. Yes, because we are reporting the actual volume status. What policy should be applied to the volume - it is user's concern.
6. Yes, it's not related to this feature - will remove it from proposal.
7. Do we need to add mount status to Volumes CRD? - It may be implemented additionally as separate flow, in case of it's presence we should only check this status for get mount information.
   How PV can be unmounted when pod is running? - e.g. manually.
   Are you referring to the VOL_READY or PUBLISHED state? - If the question is about NodeGetVolumeStats RPC, then this RPC is only called for mounted PV and so PUBLISHED state.
   What is the flow for checking that volume is mounted? - we should lookup procfs mountinfo. Alternatively we could use https://github.com/dell/gofsutil this lib. As far as I know, if it exists at procfs, then we should be satisfied by it. Also all of the CSI drivers (which I viewed) implemented this check by this way.
8. Is ReadDir is enough to detect IO errors? - It's one of the ways to detect IO errors. As I seen only one CSI driver used it to check volume health: https://github.com/dell/csi-powerstore. Others do not even use this (only check mount point).
   Won't you read from system cache? - It may use system cache for some content, but the probability, that it will be used for all the content is low.
9. If the question about fsck or xfs_repair runs - then in basic case it should be done with small intervals (dozens of ms or less - should be additionally evaluated by standard workloads). Also this command will be more suitable with separate mount status for volume, than executing this evaluation by every run of k8s volume health checking mechanism. We can introduce this type of status by separate proposal or it's better in this one?

[mishoyama]

3. If we can't find existing drive - this is covered by MISSING operational status. We can generate event only when drive health was detected as GOOD and then transferred to UNKNOWN state.
4. Are all of them require exposing through volume health APIs?
5. The system might be under heavy load and commands sent to the devices might failed by timeout. In this case you will get bunch of false positive alerts.
6. Please try to unmount volume manually when container is running - you'll get "busy" error. What state do you suggest to report when volume is not staged (bind mount)? We need to investigate question about reliable mount check logic.
7. Try to hide scsi device and read dir on related block device. Your test will show you that everything is ok.
8. How running of these tools affect IO performance? What is the benefit of running them with small intervals?