DeleteVolume() fails to cleanup volume if it contains read-only subdir

[mpatlasov]

What happened:

When deleting a Pod with PV with "Delete" reclaim policy, DeleteVolume() attempts to cleanup volume, but fails if it contains a directory without write permissions. PV got stuck in "Released" status and reports error:

$ kubectl describe pv pvc-793ed532-1297-4a12-845f-23b804a4b304
...
  Warning  VolumeFailedDelete  62s (x8 over 3m9s)  smb.csi.k8s.io_smb-csi-driver-controller-7767f6685-rvdgl_6ca9501c-0edf-43a5-b6fa-9c9f5627b96b  rpc error: code = Internal desc = failed to delete subdirectory: unlinkat /tmp/pvc-793ed532-1297-4a12-845f-23b804a4b304/pvc-793ed532-1297-4a12-845f-23b804a4b304/dd: permission denied

Inside SMB server, corresponding volume is also not deleted:

[root@samba-0 /]# ls -l /share
total 24
drwx------. 2 root      root      16384 Aug 30 10:36 lost+found
drwxr-xr-x. 3 sambauser sambauser  4096 Aug 30 13:13 pvc-793ed532-1297-4a12-845f-23b804a4b304

What you expected to happen:

Both PV object and a share on SMB server must be deleted (the same way as it happens without read-only subdir).

How to reproduce it:

Create StorageClass:

kubectl apply -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: samba
provisioner: smb.csi.k8s.io
parameters:
  source: //samba-server.samba-server.svc.cluster.local/share
  csi.storage.k8s.io/provisioner-secret-name: smbcreds
  csi.storage.k8s.io/provisioner-secret-namespace: samba-server
  csi.storage.k8s.io/node-stage-secret-name: smbcreds
  csi.storage.k8s.io/node-stage-secret-namespace: samba-server
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
  - dir_mode=0777
  - file_mode=0777
  - noperm
  - mfsymlinks
  - cache=strict
  - noserverino
EOF

for a samba server from https://github.com/samba-in-kubernetes/samba-container .

Create a Pod requesting samba PV:

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Mi
  storageClassName: samba
---
apiVersion: v1
kind: Pod
metadata:
  name: fedora
  labels:
    app: fedora
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1002
    runAsGroup: 1002
    fsGroup: 1002
  volumes:
  - name: claim2-vol
    persistentVolumeClaim:
      claimName: claim
  containers:
  - image: registry.fedoraproject.org/fedora-minimal
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    name: fedora
    securityContext:
      allowPrivilegeEscalation: false
      seccompProfile:
        type: RuntimeDefault
      capabilities:
        drop:
        - "ALL"
    volumeMounts:
      - mountPath: "/mnt/claim"
        name: claim2-vol
  restartPolicy: Always

Create read-only subdir inside the Pod, then delete Pod and PVC:

$ kubectl exec -it fedora -- bash
bash-5.2$ mkdir /mnt/claim/dd
bash-5.2$ chmod a-w /mnt/claim/dd

$ kubectl delete pod fedora
$ kubectl delete pvc --all

PV is not deleted:

$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                       STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-793ed532-1297-4a12-845f-23b804a4b304   1Mi        RWO            Delete           Released   default/claim               samba          <unset>                          3m3s

It reports error:

$ kubectl describe pv pvc-793ed532-1297-4a12-845f-23b804a4b304
...
  Warning  VolumeFailedDelete  62s (x8 over 3m9s)  smb.csi.k8s.io_smb-csi-driver-controller-7767f6685-rvdgl_6ca9501c-0edf-43a5-b6fa-9c9f5627b96b  rpc error: code = Internal desc = failed to delete subdirectory: unlinkat /tmp/pvc-793ed532-1297-4a12-845f-23b804a4b304/pvc-793ed532-1297-4a12-845f-23b804a4b304/dd: permission denied

And read-only dir still exists inside samba server:

[root@samba-0 /]# ls -l /share/
total 24
drwx------. 2 root      root      16384 Aug 30 10:36 lost+found
drwxr-xr-x. 3 sambauser sambauser  4096 Aug 30 13:13 pvc-793ed532-1297-4a12-845f-23b804a4b304
[root@samba-0 /]# ls -l /share/pvc-793ed532-1297-4a12-845f-23b804a4b304
total 8
drwxr-xr-x. 2 sambauser sambauser 4096 Aug 30 13:13 dd

Anything else we need to know?:

Environment:

• CSI Driver version: Image manually built by docker build ... -f ./cmd/smbplugin/Dockerfile . from the top of upstream master branch (commit 4d10e97 of https://github.com/kubernetes-csi/csi-driver-smb.git)

• Kubernetes version (use kubectl version):

Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.3

[andyzhangx]

we are using os.RemoveAll to remove the dir, how should this driver delete the dir without permission?

csi-driver-smb/pkg/smb/controllerserver.go

Line 227 in 54c6421

if err = os.RemoveAll(internalVolumePath); err != nil {

[mpatlasov]

we are using os.RemoveAll to remove the dir, how should this driver delete the dir without permission?

csi-driver-smb/pkg/smb/controllerserver.go

Line 227 in 54c6421

if err = os.RemoveAll(internalVolumePath); err != nil {

chmod -R a+rwx <volume-dir> would fix the issue. See #840 please.