Unable to create PVC from snapshot on EKS 1.21+

[mconigliaro]

I'm following the instructions from here using the volume snapshot CRDs and volume snapshot controller from the 6.0 pre-release. Snapshots are created just fine, but when I try to create a new PVC from one, I get the following two errors:

failed to provision volume with StorageClass "gp3": rpc error: code = Internal desc = Could not create volume "pvc-0766c0f1-febf-42e1-aed3-6099a8b05e35": failed to get an available volume in EC2: InvalidVolume.NotFound: The volume 'vol-00fb50f0f0c0da1a8' does not exist. status code: 400, request id: 12a4f77a-de2a-4010-8b7f-8b435c531032

failed to provision volume with StorageClass "gp3": rpc error: code = AlreadyExists desc = Could not create volume "pvc-0766c0f1-febf-42e1-aed3-6099a8b05e35": Parameters on this idempotent request are inconsistent with parameters used in previous request(s)

This is on a recent version of EKS:

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.5-eks-bc4871b", GitCommit:"5236faf39f1b7a7dabea8df12726f25608131aa9", GitTreeState:"clean", BuildDate:"2021-10-29T23:32:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}

That InvalidVolume.NotFound: The volume 'vol-00fb50f0f0c0da1a8' does not exist seems odd. Where is that volume id coming from? Is that supposed to be the new volume created from the snapshot? I noticed I get a new one every time I try this, but I never actually see a new volume appear in the AWS console.

That Parameters on this idempotent request are inconsistent with parameters used in previous request(s) also seems odd, because for what it's worth, I literally copied and pasted the original PVC to create the new one for the test restore. The spec is the same except for the addition of the dataSource to specify the source snapshot:

Source PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app.kubernetes.io/name: looker
  name: looker
  namespace: looker
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: gp3

Test restore PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app.kubernetes.io/name: looker
  name: looker-restore
  namespace: looker
spec:
  accessModes:
    - ReadWriteOnce
  dataSource:
    name: looker-test-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  resources:
    requests:
      storage: 10Gi
  storageClassName: gp3

How I created the test snapshot

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: looker-test-snapshot
spec:
  source:
    persistentVolumeClaimName: looker

[mconigliaro]

Related:

• Parameters on this idempotent request are inconsistent with parameters used in previous request kubernetes-sigs/aws-ebs-csi-driver#1036
• Example IAM policy is insufficient kubernetes-sigs/aws-ebs-csi-driver#935

[mconigliaro]

Sorry for the distraction. I now believe this was caused by using an old version of the IAM policy for the EBS CSI driver and use of encrypted volumes.