fix corrupted mount issue when driver daemonset restarted

[andyzhangx]

What type of PR is this?
/kind bug

What this PR does / why we need it:
This PR together with an k8s upstream PR(kubernetes/kubernetes#88569) would fix the corrupted mount issue when fuse based CSI driver daemonset is restarted on the node:
after daemonset is restarted, original blobfuse mount is broken, this PR would handle broken mount in both NodeStage and NodePublish

• detect the broken mount path
• unmount broken mount path
• remount mount path

Which issue(s) this PR fixes:

Fixes #115

Special notes for your reviewer:
Main fix is in ensureMountPoint func:

func (d *Driver) ensureMountPoint(target string) error {
	notMnt, err := d.mounter.IsLikelyNotMountPoint(target)
	if err != nil && !os.IsNotExist(err) {
		if IsCorruptedDir(target) {
			notMnt = false
			klog.Warningf("detected corrupted mount for targetPath [%s]", target)
		} else {
			return err
		}
	}
...

main code exection logic for this PR:

I0226 03:37:10.191769       1 utils.go:112] GRPC call: /csi.v1.Node/NodeStageVolume
I0226 03:37:10.191800       1 utils.go:113] GRPC request: volume_id:"andy-1180alpha5#fuse9e6bc1063ad742c8a12#pvc-0433847e-03fd-422f-b053-5534510eb338" staging_target_path:"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0433847e-03fd-422f-b053-5534510eb338/globalmount" volume_capability:<mount:<fs_type:"ext4" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > > volume_context:<key:"skuName" value:"Standard_LRS" > volume_context:<key:"storage.kubernetes.io/csiProvisionerIdentity" value:"1582552362270-8081-blobfuse.csi.azure.com" >
I0226 03:37:10.191810       1 nodeserver.go:105] NodeStageVolume: called with args {VolumeId:andy-1180alpha5#fuse9e6bc1063ad742c8a12#pvc-0433847e-03fd-422f-b053-5534510eb338 PublishContext:map[] StagingTargetPath:/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0433847e-03fd-422f-b053-5534510eb338/globalmount VolumeCapability:mount:<fs_type:"ext4" > access_mode:<mode:MULTI_NODE_MULTI_WRITER >  Secrets:map[] VolumeContext:map[skuName:Standard_LRS storage.kubernetes.io/csiProvisionerIdentity:1582552362270-8081-blobfuse.csi.azure.com] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
W0226 03:37:10.191901       1 nodeserver.go:241] detected corrupted mount for targetPath [/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0433847e-03fd-422f-b053-5534510eb338/globalmount]
W0226 03:37:10.191917       1 nodeserver.go:255] ReadDir /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0433847e-03fd-422f-b053-5534510eb338/globalmount failed with open /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0433847e-03fd-422f-b053-5534510eb338/globalmount: transport endpoint is not connected, unmount this directory
I0226 03:37:10.191925       1 mount_linux.go:209] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0433847e-03fd-422f-b053-5534510eb338/globalmount
I0226 03:37:10.674579       1 nodeserver.go:141] target /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0433847e-03fd-422f-b053-5534510eb338/globalmount
fstype ext4

volumeId andy-1180alpha5#fuse9e6bc1063ad742c8a12#pvc-0433847e-03fd-422f-b053-5534510eb338
context map[skuName:Standard_LRS storage.kubernetes.io/csiProvisionerIdentity:1582552362270-8081-blobfuse.csi.azure.com]
mountflags []
mountOptions [--use-https=true]
args /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0433847e-03fd-422f-b053-5534510eb338/globalmount --tmp-path=/mnt/andy-1180alpha5#fuse9e6bc1063ad742c8a12#pvc-0433847e-03fd-422f-b053-5534510eb338 --container-name=pvc-0433847e-03fd-422f-b053-5534510eb338 --use-https=true

I0226 03:37:10.813063       1 utils.go:113] GRPC request: volume_id:"andy-1180alpha5#fuse9e6bc1063ad742c8a12#pvc-0433847e-03fd-422f-b053-5534510eb338" staging_target_path:"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0433847e-03fd-422f-b053-5534510eb338/globalmount" target_path:"/var/lib/kubelet/pods/8a2a3fdd-f52c-460b-b270-d7cc00ae7ed5/volumes/kubernetes.io~csi/pvc-0433847e-03fd-422f-b053-5534510eb338/mount" volume_capability:<mount:<fs_type:"ext4" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > > volume_context:<key:"skuName" value:"Standard_LRS" > volume_context:<key:"storage.kubernetes.io/csiProvisionerIdentity" value:"1582552362270-8081-blobfuse.csi.azure.com" >
I0226 03:37:10.813074       1 nodeserver.go:39] NodePublishVolume: called with args {VolumeId:andy-1180alpha5#fuse9e6bc1063ad742c8a12#pvc-0433847e-03fd-422f-b053-5534510eb338 PublishContext:map[] StagingTargetPath:/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0433847e-03fd-422f-b053-5534510eb338/globalmount TargetPath:/var/lib/kubelet/pods/8a2a3fdd-f52c-460b-b270-d7cc00ae7ed5/volumes/kubernetes.io~csi/pvc-0433847e-03fd-422f-b053-5534510eb338/mount VolumeCapability:mount:<fs_type:"ext4" > access_mode:<mode:MULTI_NODE_MULTI_WRITER >  Readonly:false Secrets:map[] VolumeContext:map[skuName:Standard_LRS storage.kubernetes.io/csiProvisionerIdentity:1582552362270-8081-blobfuse.csi.azure.com] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
W0226 03:37:10.813124       1 nodeserver.go:241] detected corrupted mount for targetPath [/var/lib/kubelet/pods/8a2a3fdd-f52c-460b-b270-d7cc00ae7ed5/volumes/kubernetes.io~csi/pvc-0433847e-03fd-422f-b053-5534510eb338/mount]
W0226 03:37:10.813142       1 nodeserver.go:255] ReadDir /var/lib/kubelet/pods/8a2a3fdd-f52c-460b-b270-d7cc00ae7ed5/volumes/kubernetes.io~csi/pvc-0433847e-03fd-422f-b053-5534510eb338/mount failed with open /var/lib/kubelet/pods/8a2a3fdd-f52c-460b-b270-d7cc00ae7ed5/volumes/kubernetes.io~csi/pvc-0433847e-03fd-422f-b053-5534510eb338/mount: transport endpoint is not connected, unmount this directory
I0226 03:37:10.813152       1 mount_linux.go:209] Unmounting /var/lib/kubelet/pods/8a2a3fdd-f52c-460b-b270-d7cc00ae7ed5/volumes/kubernetes.io~csi/pvc-0433847e-03fd-422f-b053-5534510eb338/mount
I0226 03:37:10.819437       1 nodeserver.go:69] NodePublishVolume: mounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0433847e-03fd-422f-b053-5534510eb338/globalmount at /var/lib/kubelet/pods/8a2a3fdd-f52c-460b-b270-d7cc00ae7ed5/volumes/kubernetes.io~csi/pvc-0433847e-03fd-422f-b053-5534510eb338/mount with mountOptions: [bind]
I0226 03:37:10.819461       1 mount_linux.go:142] Mounting cmd (mount) with arguments ([-o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0433847e-03fd-422f-b053-5534510eb338/globalmount /var/lib/kubelet/pods/8a2a3fdd-f52c-460b-b270-d7cc00ae7ed5/volumes/kubernetes.io~csi/pvc-0433847e-03fd-422f-b053-5534510eb338/mount])
I0226 03:37:10.821022       1 mount_linux.go:142] Mounting cmd (mount) with arguments ([-o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0433847e-03fd-422f-b053-5534510eb338/globalmount /var/lib/kubelet/pods/8a2a3fdd-f52c-460b-b270-d7cc00ae7ed5/volumes/kubernetes.io~csi/pvc-0433847e-03fd-422f-b053-5534510eb338/mount])
I0226 03:37:10.824341       1 nodeserver.go:76] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0433847e-03fd-422f-b053-5534510eb338/globalmount at /var/lib/kubelet/pods/8a2a3fdd-f52c-460b-b270-d7cc00ae7ed5/volumes/kubernetes.io~csi/pvc-0433847e-03fd-422f-b053-5534510eb338/mount successfully

Release note:

fix corrupted mount issue when driver deamonset restarted

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andyzhangx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [andyzhangx]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ZeroMagic]

/lgtm

[hhstu]

@andyzhangx Is this support subpath?

[andyzhangx]

@andyzhangx Is this support subpath?

@hhstu the mount path won't be broken now since we are using blobfuse-proxy by default: https://github.com/kubernetes-sigs/blob-csi-driver/tree/master/deploy/blobfuse-proxy

[hhstu]

@andyzhangx ok，Thank you.