PV is not getting cleaned up for the restored volumes

[pawanpraka1]

What steps did you take and what happened:

Restored the namespace which has 2 pods and corresponding pvcs using openebs/velero-plugin. When I am deleting the restored namespace, pv remains there in the system with the error

  status:
    message: 'Error getting deleter volume plugin for volume "pvc-b4b70408-49d7-4921-a88e-80c1fcd981aa":
      no deletable volume plugin matched'
    phase: Failed

What did you expect to happen:

PV to be cleand up. PVC and pods gets cleaned up but not the PV.

Anything else you would like to add:

I think this change (#3007) caused the issue. k8s checks the annotation pv.kubernetes.io/provisioned-by to find the plugin related information which has been cleaned up in this PR. Now k8s assumes that pv is statically provisioned and throws error.

This works fine with 1.5.1 and lower releases. The issue is there with 1.5.2 and 1.5.3.

[pawanpraka1]

cc: @nrb

[ywk253100]

Get the same error on Azure, here is the original PV:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: disk.csi.azure.com
  creationTimestamp: "2021-11-04T07:42:52Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-887ac40b-5a27-4763-9e99-b908410d6ae1
  resourceVersion: "14468306"
  uid: 508c0b31-7a96-491a-be7c-fb166c9ecb09
spec:
  accessModes:
  - ReadWriteOnce
  azureDisk:
    cachingMode: ReadWrite
    diskName: pvc-887ac40b-5a27-4763-9e99-b908410d6ae1
    diskURI: /subscriptions/2261f3e7-d159-48fe-95a3-0e6a96e11159/resourceGroups/mc_yinw-resource-group-01_yinw-cluster-01_westus/providers/Microsoft.Compute/disks/pvc-887ac40b-5a27-4763-9e99-b908410d6ae1
    fsType: ""
    kind: Managed
    readOnly: false
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: etcd0-pv-claim
    namespace: default
    resourceVersion: "14468289"
    uid: 887ac40b-5a27-4763-9e99-b908410d6ae1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.disk.csi.azure.com/zone
          operator: In
          values:
          - ""
  persistentVolumeReclaimPolicy: Delete
  storageClassName: managed
  volumeMode: Filesystem
status:
  phase: Bound

And here is the one after restoring:

apiVersion: v1
kind: PersistentVolume
metadata:
  creationTimestamp: "2021-11-04T07:44:51Z"
  finalizers:
  - kubernetes.io/pv-protection
  labels:
    velero.io/backup-name: azure-csi-13
    velero.io/restore-name: azure-csi-13
  name: pvc-887ac40b-5a27-4763-9e99-b908410d6ae1
  resourceVersion: "14468697"
  uid: b522071e-5e5a-4288-8609-35414b80f0b3
spec:
  accessModes:
  - ReadWriteOnce
  azureDisk:
    cachingMode: ReadWrite
    diskName: restore-cd7632ca-13b0-45c6-b6a4-eb5e7e614c94
    diskURI: /subscriptions/2261f3e7-d159-48fe-95a3-0e6a96e11159/resourceGroups/MC_yinw-resource-group-01_yinw-cluster-01_westus/providers/Microsoft.Compute/disks/restore-cd7632ca-13b0-45c6-b6a4-eb5e7e614c94
    fsType: ""
    kind: Managed
    readOnly: false
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: etcd0-pv-claim
    namespace: default
    resourceVersion: "14468695"
    uid: 1cb76a5f-f3d1-44f4-981f-e2a48bd9307c
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.disk.csi.azure.com/zone
          operator: In
          values:
          - ""
  persistentVolumeReclaimPolicy: Delete
  storageClassName: managed
  volumeMode: Filesystem
status:
  phase: Bound

[ywk253100]

This issue only happens on the clusters that have no in-tree volume drivers.

When a volume is being deleted, the func (ctrl *PersistentVolumeController) findDeletablePlugin(volume *v1.PersistentVolume) (vol.DeletableVolumePlugin, error) is called to find the corresponding plugin/driver.

The function tries to get the driver name from the annotation pv.kubernetes.io/provisioned-by of the PV first. If the annotation doesn't exist, the function fallback to go through all the registered in-tree drivers and to check whether anyone can handle this PV.

For the PV restored by velero, as velero removes the annotation pv.kubernetes.io/provisioned-by during the restore, the function always fallback to get the volume plugin from the in-tree drivers. So on the cluster that has no in-tress driver enabled, a similar error Error getting deleter volume plugin for volume "pvc-35974a18-4dfd-4f16-ad0a-5fecade61252": no deletable volume plugin matched is returned.

So taking the annotation pv.kubernetes.io/provisioned-by back fixes this issue and this will not affect the rebinding process for PV and PVC