Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] Workload Identity

[MarkTallentire]

What happened:
I am attempting to connect to a storage account (Standard v2, public network access) but receiving

MountVolume.SetUp failed for volume "pv-blob-vcurrent" : rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 *** blobfuse2: A new version [2.3.2] is available. Consider upgrading to latest version for bug-fixes & new features. *** Visit https://aka.ms/blobfuse2warnings#230 to see the list of vulnerabilities associated with your current version [2.3.0] Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.

I'm currently on AKS v1.30.3 which according to their release notes should be on 1.24

Azure Blob Disk CSI driver has been upgraded to v1.24.3 on AKS 1.30, v1.23.7 on AKS 1.29 and 1.28.

https://github.com/Azure/AKS/releases

I've confirmed this in my logs

And also by running kubectl get po -n kube-system -o yaml | FINDSTR blob (I'm on windows)

I can also confirm that my csidriver has the correct tokenrequests field

Heres my manifest files:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: blob.csi.azure.com
  name: pv-blob-vcurrent
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: azureblob-fuse-premium
  mountOptions:
    - -o allow_other
    - --file-cache-timeout-in-seconds=120
  csi:
    driver: blob.csi.azure.com
    readOnly: false
    volumeHandle: vcurrentexis
    volumeAttributes:       
      containerName: exis
      storageAccount: weusavcurrenttest
      resourceGroup: weu-rsg-vcurrent-test
      clientID: 49bf2853-d2a2-4f92-9fee-e859c21457b6

kind: Pod
apiVersion: v1
metadata:
  name: nginx-blob
  labels:
    azure.workload.identity/use: "true"
spec:
  serviceAccountName: vcurrent-sa
  nodeSelector:
    "kubernetes.io/os": linux
  containers:
    - image: nginx:latest
      name: nginx-blob
      volumeMounts:
        - name: blob01
          mountPath: "/mnt/blob"
          readOnly: false
  volumes:
    - name: blob01
      persistentVolumeClaim:
        claimName: pvc-blob
        
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-blob
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  volumeName: pv-blob-vcurrent
  storageClassName: azureblob-fuse-premium

I have checked RBAC for the service account and confirm it has Blob Contributor

and Contributor rights

and the network is set to public access (for now)

Access keys are turned on and have not been changed in anyway, this is a brand new storage account that hasnt had any defaults changed.

I can also see requests being successfully made to retrieve the storage account keys

What you expected to happen:
According to https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/docs/workload-identity-static-pv-mount.md I should be able to use workload identity by adding clientId to PersistentVolume.volumeAttributes as of 1.24 and AKS > 1.29

[MarkTallentire]

After a lot of debugging, I found that my container name was wrong. There was an error buried on the nodes logs (blobfuse2.log) indicating this, would be great if we could surface this better.