[QUESTION]: Persistent volume not getting detached after pod deletion

[rajendraindukuri]

Reported by customer :
I have a problem with csi-unity - once a PersistentVolume is attached to a node, it does not get detached after the pod using it is deleted
later on, when a different pod referring to that PV via PersistentVolumeClaim gets scheduled to a different node, CSI driver creates new VolumeAttachment which cannot be provisioned: Volume has been published to a different host already.

[karthikk92]

could you please let us know the bellow details

1. environment info
2. driver version
3. installed via helm/operator ? its version ?
4. k8s/openshift version
5. steps to reproduce this issue

[mkej]

1. Deployed on a VM-based Kubernetes cluster, each VM runs Ubuntu 20.04 with 5.11 kernel.
2. Observed on v1.6.0 and v2.6.0
3. Installed via Helm, using dell-csi-helm-installer/csi-install.sh script.
4. K8s cluster installed using kubeadm, issues observed on v1.21.3 and v1.22.4 (same cluster, upgraded)
5. Steps to reproduce:
   Apply these resources to cluster:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: volume-test
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: unity-iscsi
  volumeMode: Filesystem
---
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: volume-test
  labels:
    app: volume-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: volume-test
  template:
    metadata:
      labels:
        app: volume-test
    spec:
      containers:
      - name: nginx
        image: nginx
        volumeMounts:
        - mountPath: /usr/share/nginx/html
          name: volume-test
      volumes:
      - name: volume-test
        persistentVolumeClaim:
          claimName: volume-test

This will create PersistentVolume and a VolumeAttachment for it and a specific node.

In order to force rescheduling pod to a different node, add spec.template.spec.nodeName: some-different-node to the ReplicaSet and delete existing pod.

Another VolumeAttachment will be created, with the previous one still in place, with status.attachError.message = 'rpc error: code = Aborted desc = runid=7652 Volume has been published to a different host already.'. The previous attachement is still in place and active. This will keep the Pod stuck with AttachVolume.Attach failed error from attachdetach-controller.

[mkej]

Also, when the above ReplicaSet and PersistentVolumeClaim are deleted, the PersistentVolume is not deleted, even though it has Delete reclaim policy. It is stuck in Released status, with both VolumeAttachments still existing. When I manually delete VolumeAttachments the Host access for the original node gets removed from the block LUN on Unity, but PV is still there on the Kubernetes cluster. If I delete PV manually once attachements are gone, the PV does get deleted from the cluster, but related block LUN is still there on Unity.

This seems similar to issue described on https://www.dell.com/community/Containers/csi-unity-volumeattachments-pv-will-not-detach-and-delete-LUN-if/td-p/7847221, but "tear down the whole environment (nodes and all) and rebuild" sounds horrible and I would love to find a different solution.

[karthikk92]

could you please share the cli id from unity console. It will be of format "sv_XXXX" and also please attach the complete controller driver and node logs with enabled debug logs

and also PVC, PV and volumeattachment of pod

[karthikk92]

We have seen similar issue earlier when length of prefix+cliid is less than 15.

i.e
When total number of characters in volume-name-prefix + CLI-ID is 15 characters or more, then the volumeattachment issue goes away as mentioned in the github issue( for operator put this entry in sample files and if installation via helm , please update with values.yaml file)

So we recommend the customer to have longer prefix names, say for example "csiunitystorageclass" as a workaround.

There is a [PR |https://github.com/kubernetes/kubernetes/pull/106557] raised in kubernetes to fix the issue.

Hope this will resolve the issue. Please let us know if you are still facing the issue and will debug further

Thanks and Regards,
Karthik K

[mkej]

@karthikk92 thank you so much for your suggestion! I've changed volume name prefix to be longer - it was 7 characters long before, it is 18 chars long now. The scenario I described above, which was consistently failing before, now works like a charm. At most one VolumeAttachment is now present for a PersistentVolume at any given time. When deleting pod, VolumeAttachment is deleted as well and a new one is created when the pod is scheduled again, possibly on different node.

[mkej]

By the way, isn't default csi-unity configuration affected by this as well? The default prefix is csivol, which is even shorter than mine was. Or maybe I was unlucky enough to hit a particular problematic length range.

[karthikk92]

Once the fix PR by kubernetes is merged(kubernetes/kubernetes#106557), this will be taken care. SO till than we recommend the customer to have longer prefix names.

[karthikk92]

Since the issue resolved , will be closing this issue. Thanks @mkej