-
Notifications
You must be signed in to change notification settings - Fork 807
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PVC are in Pending state #954
Comments
Hi @dolisss, Sorry I'm not able to reproduce the issue on my side.. looks like provisioner have some trouble talk tot csi driver. Can you also paste the log from driver? |
@AndyXiangLi I can see daemonset.apps/ebs-csi-node, deployment.apps/ebs-csi-controller, statefulset.apps/ebs-snapshot-controller Kubernetes object running as a part of deploying CSI driver. Please can you help me understand what refers to the driver logs here? |
I'm referring to the controller log, basically controller is responsible for any aws request like CreateVolume, and you can grab the log by running kubectl logs -n kube-system -c ebs-plugin ${your-controller-pod-id} |
@AndyXiangLi Ohh I see, please find the below logs: W0629 14:33:29.966118 1 metadata.go:191] Failed to parse the outpost arn: |
We've seen the same behaviour on multiple EKS clusters, usually after upgrading cluster version meaning that the nodes are being replaced. |
@AndyXiangLi I can get you all the logs you want, I am seeing this error with vanilla install according to https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html in an EKS 1.19 cluster. Here are some of the errors I'm seeing:
It seems to me that the CRD were not created, eg Based on helm docs for helm 3,
Now when if I install the chart from source the CRD are created:
ie there are 3 new CRDs. The helm chart in charts repo (not git repo) is this as of right now:
|
So further investigation shows that the chart 1.2.3 package does not contain a
whereas the
So this is the issue: the chart was incorrectly packaged. I could not find how the chart is created, I looked for Meanwhile, 2 workarounds:
Once the CRDs created, the errors that I described in previous comment go away. Note that if you use IRSA (AWS IAM Role for Service Accounts), you need to set the annotation on the service account when calling helm install and although it can be done via
where
|
This is related to issue #635 |
Related, but the root cause is that the crds are not in the chart so anyone installing the helm chart via instructions elsewhere (such as AWS docs) will miss that critical step, and moreover will assume the crds are in the chart since they are in the charts folder in github. Summary solution here, at least for now until #635 is resolved (and probably even after, based on the discussion there), is:
|
Not sure if is 100% related, but I ran into an issue similar which was missing KMS permissions. #935 (comment) |
Version 2.0.0 of the helm chart no longer installs the snapshot controller or its CRDs. The snapshot controller is now listed as a pre-req if you want to use snapshot functionality. The enableVolumeSnapshot value no longer exists and the sidecar is enabled based on whether the snapshot API is available in the target cluster. The AWS docs need to be updated. I am not sure if there is someone on this project that has access to edit those docs. @wongma7 or @ayberk do you know of someone who can update those docs? |
PVCs pending for me as well. Even the dynamic provisioning example does not seem to work for me. I am using manifests. I suspect I might be doing something wrong, but I have tried the instructions on the AWS EBS CSI docs multiple times. EDIT: Apparently if I make a PV before deploying the example for dynamic provisioning, it works. I suspect that kind of defeats the purpose of dynamic provisioning because in the docs it does not say to create a PV. |
HI i have same issue " reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ebs-sc": error getting handle for DataSource Type VolumeSnapshot by Name static-snapshot-demo: snapshot static-snapshot-demo not bound " i dont know why , using EKS 1.21 any one have idea ? ( the CRD is install and i checked) |
@AndyXiangLi I still face this issue , do we know how can I resolve this? |
Hello, I am facing the same issue as well.
|
I was able to resolve my error, which is almost similar to yours. Although my EKS is fully private, so I am using VPCE for all the requirements. In the AWS docs, I saw in the cluster autoscaler they asked to mention to add the environment variable which is AWS_REGION=yourRegion and AWS_STS_REGIONAL_ENDPOINTS=regional. I added both the env variable in the ebs-csi-controller deployment and it worked The only difference is cluster-autoscaler fails to deploy if we don't give these two env variables in the private cluster, but ebs-csi-controller deploys successfully. @AndyXiangLi Is it possible to do a network and permission check validation before ebs-csi-controller deploys successfully. I saw one issue #216 but not sure it's the same :) |
Hi, i got similar problem and the way to see what is happening was to activate the AWS logs as they say here: ##830 In my case, i got this error:
So it seems it is something related with my EKS configuration. |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
/remove-lifecycle stale |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
Sharing what solved my issue: Turns out that we need to add OIDC IDP in the IAM of the account for the OIDC issuer of the EKS cluster. |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close |
@k8s-triage-robot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Has anyone able to solve this issue? Getting the same no matter how many times i try following the document @dolisss |
@arshavardan Did you setup IRSA and the environment variables properly in manifest file? |
@AndyXiangLi In my case its same error but I am using EFS CSI Driver. errors in EFS Controller logs : |
hello @vsivas , |
Hi @nikitsrj Its private EKS cluster(customer) and I dont think we are using helm charts for this. |
/kind bug
I deployed CSI Driver as stated here: https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html
What I expect? : Pod should run using PVC that are provisioned dynamically
how to reproduce this issue : I followed the document to install this and I can see the below error
Below are the logs describing PVC:
Warning ProvisioningFailed 93s (x2 over 93s) persistentvolume-controller storageclass.storage.k8s.io "ebs-sc" not found
Normal Provisioning 9s (x6 over 90s) ebs.csi.aws.com_ebs-csi-controller-f5d9c9475-wh2t9_e2eea260-a1f6-4b74-9250-baf43ba03780 External provisioner is provisioning volume for claim "default/ebs-claim"
Normal ExternalProvisioning 3s (x8 over 90s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
Warning ProvisioningFailed 0s (x6 over 80s) ebs.csi.aws.com_ebs-csi-controller-f5d9c9475-wh2t9_e2eea260-a1f6-4b74-9250-baf43ba03780 failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded
kubectl logs ebs-csi-controller-f5d9c9475-wh2t9 -c csi-provisioner -n kube-system:
CreateVolume failed, supports topology = true, node selected true => may reschedule = true => state = Background: rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0624 09:55:16.287191 1 controller.go:1106] Temporary error received, adding PVC 3f6ec7b5-2a25-4f42-babb-d808dd464535 to claims in progress
W0624 09:55:16.287200 1 controller.go:958] Retrying syncing claim "3f6ec7b5-2a25-4f42-babb-d808dd464535", failure 8
E0624 09:55:16.287222 1 controller.go:981] error syncing claim "3f6ec7b5-2a25-4f42-babb-d808dd464535": failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0624 09:55:16.287251 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-claim", UID:"3f6ec7b5-2a25-4f42-babb-d808dd464535", APIVersion:"v1", ResourceVersion:"6828149", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I cant see the right error for this issue to troubleshoot.
AWS-EKS
-eks: 1.19
The text was updated successfully, but these errors were encountered: