Unable to create CSI snapshot-EBS csi driver #1249
Comments
|
Hi @keerthana1608 👋 Have you installed the external-snapshotter sidecar? If you intend to use the csi-snapshotter functionality, you will need to first install the CSI Snapshotter. |
@torredil Yes, the external snapshotter was installed. The same tests were working fine with CSI driver 1.5.1 when this was tested previously a while back. |
|
Looks like the snapshot CRDs are not installed. Can you try installing them with: |
|
What we observed is the VolumeSnapshot was created and the associated VolumeSnapshotContent also created but the EBS CSI Driver does not show any action (in the log) that it starts working on the VolumeSnapshotContent so the VolumeSnapshot and VolumeSnapshotContent are just hanging there. |
|
Same behaviour for me as reported by @phuongatemc but I am using the managed add-on Creating PVCs works fine (type gp3) but not taking snapshots via K8S/Kubectl Version: Log from container CRDs (related to Volume Snapshots) taken from https://github.com/kubernetes-csi/external-snapshotter/tree/v6.0.0/client/config/crd |
|
Hi @phuongatemc @sgatdell @tgip-work, This PR will fix the API version issue on sidecar. Updating the snapshot controller CRDs after this PR will work, for long term we will bump up our snapshotter sidecar version to make sure the API V1 works. |
|
@gtxu CSI snapshotter logs: |
|
Hi @gtxu as @keerthana1608 did, I also used the latest CRDs from https://github.com/kubernetes-csi/external-snapshotter/tree/master/client/config/crd. and I still see the same issue as before. My setup:
Maybe I am totally wrong, but the sidecar container csi-snapshotter (for the managed addon) is using image According to the docs this is an outdated version! Most current image is I will try to run the sidecar container with In addition to the CSI Snapshotter isse, there is an event at the |
SHORT UPDATE and good news!A snapshot has been created: Summary: What I needed to change:
Step by Step1. My helm values file 2. Helm Installation 3. ClusterRole ebs-external-snapshotter-role Due to error I added the verb 4. Then I had to restart the pods AppendixIn case you want the helm release exactly look like the EKS managed addon (except 1. aws-ebs-csi-driver 1.6.2 instead of 1.6.1 and 2. csi-snapshotter 6.0.1 instead of 3.0.3) , you can use these helm values. image:
repository: 602401143452.dkr.ecr.<REGION>.amazonaws.com/eks/aws-ebs-csi-driver
controller:
k8sTagClusterId: <CLUSTER_ID>
region: <REGION>
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: <ROLE_ARN>
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- ebs-csi-controller
topologyKey: kubernetes.io/hostname
weight: 100
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25%
maxUnavailable: 1
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 10m
memory: 40Mi
node:
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 10m
memory: 40Mi
sidecars:
provisioner:
image:
repository: 602401143452.dkr.ecr.<REGION>.amazonaws.com/eks/csi-provisioner
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 10m
memory: 40Mi
attacher:
image:
repository: 602401143452.dkr.ecr.<REGION>.amazonaws.com/eks/csi-attacher
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 10m
memory: 40Mi
snapshotter:
image:
#repository: 602401143452.dkr.ecr.<REGION>.amazonaws.com/eks/csi-snapshotter
repository: k8s.gcr.io/sig-storage/csi-snapshotter
tag: "v6.0.1"
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 10m
memory: 40Mi
resizer:
image:
repository: 602401143452.dkr.ecr.<REGION>.amazonaws.com/eks/csi-resizer
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 10m
memory: 40Mi
livenessProbe:
image:
repository: 602401143452.dkr.ecr.<REGION>.amazonaws.com/eks/livenessprobe
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 10m
memory: 40Mi
nodeDriverRegistrar:
image:
repository: 602401143452.dkr.ecr.<REGION>.amazonaws.com/eks/csi-node-driver-registrar
|
|
Hi @tgip-work, thanks for the attempt and detailed description on the solution. The latest snapshot controller(deployed with CRDs) includes several changes that does not works well with the external-snapshotter sidecar lower than v6.0.0+. We will update the helm to use latest external-snapshotter after testing. |
|
Plus if you install & manage your EBS CSI Driver with Helm, I recommend pulling sidecar images from GCR currently. |
|
@tgip-work I try this. Still seeing the same issue. ebs-csi-controller/csi-snapshotter can't found snapshot
|
|
I try this, it works well. apply CRD old version. v6.0.0-rc4 csi-snapshotter image use v3.0.3. |
|
Updated to latest snapshotter sidecars and added verb patch to resource volumesnaphsotcontents. |
Signed-off-by: Eddie Torres <[email protected]>
Signed-off-by: Eddie Torres <[email protected]>
ghost
mentioned this issue
/kind bug
What happened?
Unable to create CSI snapshot
What you expected to happen?
Should be able to create CSI driver snapshot without any failure.
How to reproduce it (as minimally and precisely as possible)?
Install the latest 1.5(1.5.3) or 1.6(1.6.2) ebs csi driver on EKS clsuter. Try creating the volumesnapshot.
Anything else we need to know?:
We are seeing following constant error in controller logs:
E0523 16:45:25.101262 1 reflector.go:127] k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *v1beta1.VolumeSnapshotContent: failed to list *v1beta1.VolumeSnapshotContent: the server could not find the requested resource (get volumesnapshotcontents.snapshot.storage.k8s.io)
E0523 16:45:25.101335 1 reflector.go:127] k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *v1beta1.VolumeSnapshotClass: failed to list *v1beta1.VolumeSnapshotClass: the server could not find the requested resource (get volumesnapshotclasses.snapshot.storage.k8s.io)
Environment
Kubernetes version (use
kubectl version):Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.9-eks-14c7a48", GitCommit:"717bfb2b8ceb809a42a6c0baabde59fae28637ef", GitTreeState:"clean", BuildDate:"2022-04-01T03:17:28Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}
Driver version: 1.5.3, 1.6.2
csi-snapshotter_logs.txt
The text was updated successfully, but these errors were encountered: