Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PVC are in Pending state #954

Closed
dolisss opened this issue Jun 24, 2021 · 29 comments
Closed

PVC are in Pending state #954

dolisss opened this issue Jun 24, 2021 · 29 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.

Comments

@dolisss
Copy link

dolisss commented Jun 24, 2021

/kind bug

I deployed CSI Driver as stated here: https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html

What I expect? : Pod should run using PVC that are provisioned dynamically

how to reproduce this issue : I followed the document to install this and I can see the below error

Below are the logs describing PVC:
Warning ProvisioningFailed 93s (x2 over 93s) persistentvolume-controller storageclass.storage.k8s.io "ebs-sc" not found
Normal Provisioning 9s (x6 over 90s) ebs.csi.aws.com_ebs-csi-controller-f5d9c9475-wh2t9_e2eea260-a1f6-4b74-9250-baf43ba03780 External provisioner is provisioning volume for claim "default/ebs-claim"
Normal ExternalProvisioning 3s (x8 over 90s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
Warning ProvisioningFailed 0s (x6 over 80s) ebs.csi.aws.com_ebs-csi-controller-f5d9c9475-wh2t9_e2eea260-a1f6-4b74-9250-baf43ba03780 failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded


kubectl logs ebs-csi-controller-f5d9c9475-wh2t9 -c csi-provisioner -n kube-system:

CreateVolume failed, supports topology = true, node selected true => may reschedule = true => state = Background: rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0624 09:55:16.287191 1 controller.go:1106] Temporary error received, adding PVC 3f6ec7b5-2a25-4f42-babb-d808dd464535 to claims in progress
W0624 09:55:16.287200 1 controller.go:958] Retrying syncing claim "3f6ec7b5-2a25-4f42-babb-d808dd464535", failure 8
E0624 09:55:16.287222 1 controller.go:981] error syncing claim "3f6ec7b5-2a25-4f42-babb-d808dd464535": failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0624 09:55:16.287251 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-claim", UID:"3f6ec7b5-2a25-4f42-babb-d808dd464535", APIVersion:"v1", ResourceVersion:"6828149", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded


I cant see the right error for this issue to troubleshoot.

AWS-EKS
-eks: 1.19

  • Driver version: aws-ebs-csi-driver:v1.1.0
@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Jun 24, 2021
@AndyXiangLi
Copy link
Contributor

Hi @dolisss, Sorry I'm not able to reproduce the issue on my side.. looks like provisioner have some trouble talk tot csi driver. Can you also paste the log from driver?

@dolisss
Copy link
Author

dolisss commented Jun 28, 2021

@AndyXiangLi I can see daemonset.apps/ebs-csi-node, deployment.apps/ebs-csi-controller, statefulset.apps/ebs-snapshot-controller Kubernetes object running as a part of deploying CSI driver. Please can you help me understand what refers to the driver logs here?

@AndyXiangLi
Copy link
Contributor

@AndyXiangLi I can see daemonset.apps/ebs-csi-node, deployment.apps/ebs-csi-controller, statefulset.apps/ebs-snapshot-controller Kubernetes object running as a part of deploying CSI driver. Please can you help me understand what refers to the driver logs here?

I'm referring to the controller log, basically controller is responsible for any aws request like CreateVolume, and you can grab the log by running kubectl logs -n kube-system -c ebs-plugin ${your-controller-pod-id}

@dolisss
Copy link
Author

dolisss commented Jun 29, 2021

@AndyXiangLi Ohh I see, please find the below logs:

W0629 14:33:29.966118 1 metadata.go:191] Failed to parse the outpost arn:
E0629 14:33:59.940825 1 driver.go:119] GRPC error: rpc error: code = Internal desc = RequestCanceled: request context canceled
caused by: context canceled
E0629 14:33:59.940813 1 driver.go:119] GRPC error: rpc error: code = Internal desc = RequestCanceled: request context canceled
caused by: context deadline exceeded
E0629 14:34:10.942234 1 driver.go:119] GRPC error: rpc error: code = Internal desc = RequestCanceled: request context canceled
caused by: context canceled
E0629 14:34:10.942265 1 driver.go:119] GRPC error: rpc error: code = Internal desc = RequestCanceled: request context canceled
caused by: context canceled
E0629 14:34:22.942976 1 driver.go:119] GRPC error: rpc error: code = Internal desc = RequestCanceled: request context canceled
caused by: context canceled
E0629 14:34:22.943004 1 driver.go:119] GRPC error: rpc error: code = Internal desc = RequestCanceled: request context canceled
caused by: context canceled

@stevehipwell
Copy link
Contributor

We've seen the same behaviour on multiple EKS clusters, usually after upgrading cluster version meaning that the nodes are being replaced.

@schollii
Copy link

schollii commented Jul 7, 2021

@AndyXiangLi I can get you all the logs you want, I am seeing this error with vanilla install according to https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html in an EKS 1.19 cluster. Here are some of the errors I'm seeing:

# controller/ebs-plugin:
 E0707 02:04:59.545307       1 driver.go:119] GRPC error: rpc error: code = Internal desc = Could not create volume "pvc-79212e61-3325-4d5d-a137-aa51bf408378": 
     status code: 403, request id: b62ee447-b137-46d4-a186-c98b9fda79e7

# controller/csi-snapshotter:
 E0707 02:44:33.481699       1 reflector.go:127] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: Failed to  
 watch *v1beta1.VolumeSnapshotContent: failed to list *v1beta1.VolumeSnapshotContent: the server could not find the requested resource (get volumesnapshotconte 
 nts.snapshot.storage.k8s.io)                                                                                                                                   

# node/ebs-plugin:
W0707 01:44:40.067049       1 metadata.go:191] Failed to parse the outpost arn: 
(nothing else after that)

#  I0707 02:46:34.018948       1 reflector.go:243] Listing and watching *v1beta1.VolumeSnapshotContent from github.com/kubernetes-csi/external-snapshotter/client 
 /v3/informers/externalversions/factory.go:117                                                                                                                  
 E0707 02:46:34.020942       1 reflector.go:127] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: Failed to  
 watch *v1beta1.VolumeSnapshotContent: failed to list *v1beta1.VolumeSnapshotContent: the server could not find the requested resource (get volumesnapshotconte 
 nts.snapshot.storage.k8s.io)                                                                                                                                   

It seems to me that the CRD were not created, eg kubectl get crd does not show VolumeSnapshotContent.

Based on helm docs for helm 3, helm install and helm upgrade --install should both install the CRDs. I used both helm 3.5 and 3.6, like on the AWS docs referenced earlier ie

helm install aws-ebs-csi-driver aws-ebs-csi-driver/aws-ebs-csi-driver \
                --namespace kube-system \
                --set enableVolumeResizing=true \
                --set enableVolumeSnapshot=true \
                --set serviceAccount.controller.create=true \
                --set serviceAccount.controller.name=ebs-csi-controller-sa

Now when if I install the chart from source the CRD are created:

$ git clone https://github.com/kubernetes-sigs/aws-ebs-csi-driver.git
...
$ cd aws-ebs-csi-driver/charts
$ helm install aws-ebs-csi-driver ./aws-ebs-csi-driver --namespace kube-system --set ...
...
$ kg crd
NAME                                             CREATED AT
...
volumesnapshotclasses.snapshot.storage.k8s.io    2021-07-07T03:38:25Z
volumesnapshotcontents.snapshot.storage.k8s.io   2021-07-07T03:38:25Z
volumesnapshots.snapshot.storage.k8s.io          2021-07-07T03:38:25Z

ie there are 3 new CRDs.

The helm chart in charts repo (not git repo) is this as of right now:

$ helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver
...
$ helm repo update
...
$ helm search repo aws-ebs-csi-driver
NAME                                 	CHART VERSION	APP VERSION	DESCRIPTION                        
aws-ebs-csi-driver/aws-ebs-csi-driver	1.2.3        	1.1.0      	A Helm chart for AWS EBS CSI Driver

@schollii
Copy link

schollii commented Jul 7, 2021

So further investigation shows that the chart 1.2.3 package does not contain a crds folder:

$ helm pull aws-ebs-csi-driver/aws-ebs-csi-driver

$ tar tvf aws-ebs-csi-driver-1.2.3.tgz
-rw-r--r-- 0/0             416 2021-06-17 20:22 aws-ebs-csi-driver/Chart.yaml
-rw-r--r-- 0/0            4695 2021-06-17 20:22 aws-ebs-csi-driver/values.yaml
-rw-r--r-- 0/0             852 2021-06-17 20:22 aws-ebs-csi-driver/templates/NOTES.txt
-rw-r--r-- 0/0            2542 2021-06-17 20:22 aws-ebs-csi-driver/templates/_helpers.tpl
-rw-r--r-- 0/0             740 2021-06-17 20:22 aws-ebs-csi-driver/templates/clusterrole-attacher.yaml
-rw-r--r-- 0/0             241 2021-06-17 20:22 aws-ebs-csi-driver/templates/clusterrole-csi-node.yaml
-rw-r--r-- 0/0            1312 2021-06-17 20:22 aws-ebs-csi-driver/templates/clusterrole-provisioner.yaml
-rw-r--r-- 0/0            1008 2021-06-17 20:22 aws-ebs-csi-driver/templates/clusterrole-resizer.yaml
-rw-r--r-- 0/0            1147 2021-06-17 20:22 aws-ebs-csi-driver/templates/clusterrole-snapshot-controller.yaml
-rw-r--r-- 0/0             782 2021-06-17 20:22 aws-ebs-csi-driver/templates/clusterrole-snapshotter.yaml
-rw-r--r-- 0/0             416 2021-06-17 20:22 aws-ebs-csi-driver/templates/clusterrolebinding-attacher.yaml
-rw-r--r-- 0/0             404 2021-06-17 20:22 aws-ebs-csi-driver/templates/clusterrolebinding-csi-node.yaml
-rw-r--r-- 0/0             422 2021-06-17 20:22 aws-ebs-csi-driver/templates/clusterrolebinding-provisioner.yaml
-rw-r--r-- 0/0             414 2021-06-17 20:22 aws-ebs-csi-driver/templates/clusterrolebinding-resizer.yaml
-rw-r--r-- 0/0             427 2021-06-17 20:22 aws-ebs-csi-driver/templates/clusterrolebinding-snapshot-controller.yaml
-rw-r--r-- 0/0             422 2021-06-17 20:22 aws-ebs-csi-driver/templates/clusterrolebinding-snapshotter.yaml
-rw-r--r-- 0/0           10265 2021-06-17 20:22 aws-ebs-csi-driver/templates/controller.yaml
-rw-r--r-- 0/0             308 2021-06-17 20:22 aws-ebs-csi-driver/templates/csidriver.yaml
-rw-r--r-- 0/0            5842 2021-06-17 20:22 aws-ebs-csi-driver/templates/node.yaml
-rw-r--r-- 0/0             476 2021-06-17 20:22 aws-ebs-csi-driver/templates/poddisruptionbudget-controller.yaml
-rw-r--r-- 0/0             377 2021-06-17 20:22 aws-ebs-csi-driver/templates/poddisruptionbudget-snapshot-controller.yaml
-rw-r--r-- 0/0             328 2021-06-17 20:22 aws-ebs-csi-driver/templates/role-snapshot-controller-leaderelection.yaml
-rw-r--r-- 0/0             426 2021-06-17 20:22 aws-ebs-csi-driver/templates/rolebinding-snapshot-controller-leaderelection.yaml
-rw-r--r-- 0/0             537 2021-06-17 20:22 aws-ebs-csi-driver/templates/serviceaccount-csi-controller.yaml
-rw-r--r-- 0/0             336 2021-06-17 20:22 aws-ebs-csi-driver/templates/serviceaccount-csi-node.yaml
-rw-r--r-- 0/0             350 2021-06-17 20:22 aws-ebs-csi-driver/templates/serviceaccount-snapshot-controller.yaml
-rw-r--r-- 0/0            2230 2021-06-17 20:22 aws-ebs-csi-driver/templates/snapshot-controller.yaml
-rw-r--r-- 0/0             443 2021-06-17 20:22 aws-ebs-csi-driver/templates/storageclass.yaml
-rw-r--r-- 0/0             342 2021-06-17 20:22 aws-ebs-csi-driver/.helmignore

whereas the crds folder is present in the charts folder of source in github:

$ cd aws-ebs-csi-driver/charts
$ ls aws-ebs-csi-driver -l
total 20
-rw-r--r-- 1 tooluser tooluser  434 Jul  7 01:46 Chart.yaml
drwxr-xr-x 2 tooluser tooluser 4096 Jul  7 01:46 crds
drwxr-xr-x 2 tooluser tooluser 4096 Jul  7 01:46 templates
-rw-r--r-- 1 tooluser tooluser 4695 Jul  7 01:46 values.yaml

So this is the issue: the chart was incorrectly packaged. I could not find how the chart is created, I looked for helm package command, none in Makefile or anywhere else, so I am unable to submit a PR.

Meanwhile, 2 workarounds:

  1. git clone this repo, cd to charts folder, and run helm install on the folder (rather than the package from ebs-csi helm charts repo).
  2. (better) run kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/charts/aws-ebs-csi-driver/crds/snapshot-controller-crds.yaml to create the CRDs, then helm upgrade --install ... per above.

Once the CRDs created, the errors that I described in previous comment go away.

Note that if you use IRSA (AWS IAM Role for Service Accounts), you need to set the annotation on the service account when calling helm install and although it can be done via --set, it is rather messy and much better to do it via a values.yaml. Eg

# values.yaml
enableVolumeResizing: true
enableVolumeSnapshot: true
serviceAccount:
  controller:
    create: true
    name: ebs-csi-controller-sa
    annotations:
      eks.amazonaws.com/role-arn: EBS_CONTROLLER_ROLE_ARN

where EBS_CONTROLLER_ROLE_ARN is ARN of role created per https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html. Then assuming you created the CRDs via kubectl as described above, you can just do

helm upgrade --install aws-ebs-csi-driver \
    aws-ebs-csi-driver/aws-ebs-csi-driver  \
    --namespace kube-system \
    -f /path/to/values.yaml

@krmichel
Copy link
Contributor

krmichel commented Jul 7, 2021

This is related to issue #635

@schollii
Copy link

schollii commented Jul 7, 2021

Related, but the root cause is that the crds are not in the chart so anyone installing the helm chart via instructions elsewhere (such as AWS docs) will miss that critical step, and moreover will assume the crds are in the chart since they are in the charts folder in github.

Summary solution here, at least for now until #635 is resolved (and probably even after, based on the discussion there), is:

  1. run kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/charts/aws-ebs-csi-driver/crds/snapshot-controller-crds.yaml before installing the helm chart
  2. if using AWS IRSA, include the controller's service account annotation in the values file used by helm upgrade --install ....

@robsonvn
Copy link

robsonvn commented Jul 8, 2021

Not sure if is 100% related, but I ran into an issue similar which was missing KMS permissions. #935 (comment)

@krmichel
Copy link
Contributor

Version 2.0.0 of the helm chart no longer installs the snapshot controller or its CRDs. The snapshot controller is now listed as a pre-req if you want to use snapshot functionality. The enableVolumeSnapshot value no longer exists and the sidecar is enabled based on whether the snapshot API is available in the target cluster. The AWS docs need to be updated. I am not sure if there is someone on this project that has access to edit those docs. @wongma7 or @ayberk do you know of someone who can update those docs?

@arnitkun
Copy link

arnitkun commented Sep 23, 2021

PVCs pending for me as well. Even the dynamic provisioning example does not seem to work for me. I am using manifests. I suspect I might be doing something wrong, but I have tried the instructions on the AWS EBS CSI docs multiple times.

EDIT: Apparently if I make a PV before deploying the example for dynamic provisioning, it works. I suspect that kind of defeats the purpose of dynamic provisioning because in the docs it does not say to create a PV.

@noamgreen
Copy link

noamgreen commented Sep 28, 2021

HI i have same issue

"
ebs-csi-controller-955cc5f55-9ttgg csi-provisioner I0928 07:03:47.386456 1 controller.go:1332] provision "default/ebs-snapshot-restored-claim" class "ebs-sc": started
ebs-csi-controller-955cc5f55-9ttgg csi-provisioner I0928 07:03:47.386630 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-snapshot-restored-claim", UID:"ec7c16a2-58ea-46d9-824b-25db61bcfae7", APIVersion:"v1", ResourceVersion:"89953569", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/ebs-snapshot-restored-claim"
ebs-csi-controller-955cc5f55-9ttgg csi-provisioner W0928 07:03:47.392698 1 warnings.go:70] snapshot.storage.k8s.io/v1beta1 VolumeSnapshot is deprecated; use snapshot.storage.k8s.io/v1 VolumeSnapshot
ebs-csi-controller-955cc5f55-9ttgg csi-provisioner W0928 07:03:47.392838 1 controller.go:958] Retrying syncing claim "ec7c16a2-58ea-46d9-824b-25db61bcfae7", failure 7
ebs-csi-controller-955cc5f55-9ttgg csi-provisioner E0928 07:03:47.392861 1 controller.go:981] error syncing claim "ec7c16a2-58ea-46d9-824b-25db61bcfae7": failed to provision volume with StorageClass "ebs-sc": error getting handle for DataSource Type VolumeSnapshot by Name static-snapshot-demo: snapshot static-snapshot-demo not bound

reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ebs-sc": error getting handle for DataSource Type VolumeSnapshot by Name static-snapshot-demo: snapshot static-snapshot-demo not bound

"

i dont know why , using EKS 1.21
Driver version: aws-ebs-csi-driver:v1.2.0

any one have idea ? ( the CRD is install and i checked)

@dolisss
Copy link
Author

dolisss commented Oct 6, 2021

@AndyXiangLi I still face this issue , do we know how can I resolve this?

@nikitsrj
Copy link

Hello, I am facing the same issue as well.
Could you pls let us know, how to resolve this?

I1027 05:55:53.938657       1 controller.go:1332] provision "default/ebs-claim" class "ebs-sc": started
I1027 05:55:53.938718       1 controller.go:519] skip translation of storage class for plugin: ebs.csi.aws.com
I1027 05:55:53.938957       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-claim", UID:"e414dd30-4f2d-438f-a5c1-c938a1f1c710", APIVersion:"v1", ResourceVersion:"23767", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/ebs-claim"
I1027 05:55:54.034519       1 reflector.go:381] sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller/controller.go:869: forcing resync
I1027 05:56:03.938992       1 controller.go:1106] Temporary error received, adding PVC e414dd30-4f2d-438f-a5c1-c938a1f1c710 to claims in progress
W1027 05:56:03.939021       1 controller.go:958] Retrying syncing claim "e414dd30-4f2d-438f-a5c1-c938a1f1c710", failure 10
E1027 05:56:03.939043       1 controller.go:981] error syncing claim "e414dd30-4f2d-438f-a5c1-c938a1f1c710": failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I1027 05:56:03.939222       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-claim", UID:"e414dd30-4f2d-438f-a5c1-c938a1f1c710", APIVersion:"v1", ResourceVersion:"23767", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I1027 05:56:39.942417       1 reflector.go:530] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PersistentVolumeClaim total 0 items received
I1027 05:58:39.773754       1 controller.go:1332] provision "default/ebs-claim" class "ebs-sc": started
I1027 05:58:39.773809       1 controller.go:519] skip translation of storage class for plugin: ebs.csi.aws.com
I1027 05:58:39.774219       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-claim", UID:"e414dd30-4f2d-438f-a5c1-c938a1f1c710", APIVersion:"v1", ResourceVersion:"23767", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/ebs-claim"
I1027 05:58:49.774093       1 controller.go:1106] Temporary error received, adding PVC e414dd30-4f2d-438f-a5c1-c938a1f1c710 to claims in progress
W1027 05:58:49.774124       1 controller.go:958] Retrying syncing claim "e414dd30-4f2d-438f-a5c1-c938a1f1c710", failure 11
E1027 05:58:49.774144       1 controller.go:981] error syncing claim "e414dd30-4f2d-438f-a5c1-c938a1f1c710": failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I1027 05:58:49.774163       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-claim", UID:"e414dd30-4f2d-438f-a5c1-c938a1f1c710", APIVersion:"v1", ResourceVersion:"23767", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded

@nikitsrj
Copy link

@dolisss

I was able to resolve my error, which is almost similar to yours. Although my EKS is fully private, so I am using VPCE for all the requirements. In the AWS docs, I saw in the cluster autoscaler they asked to mention to add the environment variable which is AWS_REGION=yourRegion and AWS_STS_REGIONAL_ENDPOINTS=regional. I added both the env variable in the ebs-csi-controller deployment and it worked

The only difference is cluster-autoscaler fails to deploy if we don't give these two env variables in the private cluster, but ebs-csi-controller deploys successfully.

@AndyXiangLi Is it possible to do a network and permission check validation before ebs-csi-controller deploys successfully. I saw one issue #216 but not sure it's the same :)

@ki0
Copy link

ki0 commented Nov 29, 2021

Hi, i got similar problem and the way to see what is happening was to activate the AWS logs as they say here: ##830

In my case, i got this error:

│ ebs-plugin ---[ RESPONSE ]--------------------------------------
│ ebs-plugin HTTP/1.1 400
│ ebs-plugin Connection: close
│ ebs-plugin Content-Length: 276
│ ebs-plugin Content-Type: text/xml
│ ebs-plugin Date: Mon, 29 Nov 2021 15:50:07 GMT
│ ebs-plugin X-Amzn-Requestid: cabfb5a6-3c8a-43e0-9a89-189087570b0f
│ ebs-plugin -----------------------------------------------------
│ ebs-plugin 2021/11/29 15:50:07 DEBUG: Validate Response sts/AssumeRoleWithWebIdentity failed, attempt 8/8, error InvalidIdentityToken: Incorrect token audience

So it seems it is something related with my EKS configuration.
Cheers.

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 27, 2022
@iomarcovalente
Copy link

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 7, 2022
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 5, 2022
@swetli
Copy link

swetli commented Jun 10, 2022

Sharing what solved my issue: Turns out that we need to add OIDC IDP in the IAM of the account for the OIDC issuer of the EKS cluster.

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jul 10, 2022
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue or PR with /reopen
  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

@k8s-ci-robot
Copy link
Contributor

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue or PR with /reopen
  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@arshavardan
Copy link

Has anyone able to solve this issue? Getting the same no matter how many times i try following the document @dolisss

@nikitsrj
Copy link

@arshavardan Did you setup IRSA and the environment variables properly in manifest file?

@vsivas
Copy link

vsivas commented Mar 27, 2024

@AndyXiangLi In my case its same error but I am using EFS CSI Driver.
@dolisss and @nikitsrj any suggestions?
I have done as per: https://aws.amazon.com/blogs/containers/introducing-efs-csi-dynamic-provisioning/

errors in EFS Controller logs :
—————————————————————————————
I0326 09:06:06.820914 1 controller.go:1082] Temporary error received, adding PVC 3f096f7a-9b79-4ad5-bd1c-992c66b73069 to claims in progress
W0326 09:06:06.820948 1 controller.go:934] Retrying syncing claim "3f096f7a-9b79-4ad5-bd1c-992c66b73069", failure 67
E0326 09:06:06.820968 1 controller.go:957] error syncing claim "3f096f7a-9b79-4ad5-bd1c-992c66b73069": failed to provision volume with StorageClass "aws-efs-csi-sc-cevoc-dev": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0326 09:06:06.820994 1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"cevoc-1-dev", Name:"efs-claim", UID:"3f096f7a-9b79-4ad5-bd1c-992c66b73069", APIVersion:"v1", ResourceVersion:"119237073", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "aws-efs-csi-sc-cevoc-dev": rpc error: code = DeadlineExceeded desc = context deadline exceeded

@nikitsrj
Copy link

nikitsrj commented Mar 28, 2024

hello @vsivas ,
Pls let me know whether you are spinning up private EKS cluster or Public, and if private, then you might need to check the endpoints and the you have to add those env variable that i mentioned above in my previous comments. Let me know if that works. Also make sure about IRAS config and everything is in place properly

@vsivas
Copy link

vsivas commented Mar 28, 2024

Hi @nikitsrj Its private EKS cluster(customer) and I dont think we are using helm charts for this.
This is the change right ?: https://github.com/kubernetes-sigs/aws-efs-csi-driver/pull/585/files - Conditionally added AWS_STS_REGIONAL_ENDPOINTS flag #585
Then how can i add these flags to controller- efs-csi-controller? I dont have any deployment yaml files becasue this was added by Cluster AddOn

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Projects
None yet
Development

No branches or pull requests