PVCs in EKS 1.24 stuck in Pending state

[ddl-lnarayanan]

/kind bug

What happened?
Pods stuck in pending state, PVCs not getting bound with this error:

E0927 15:58:28.337744       1 driver.go:120] GRPC error: rpc error: code = Internal desc = Could not create volume "pvc-aa928776-736e-4b4f-9e5d-b19998b8df58": could not create volume in EC2: RequestCanceled: request context canceled
caused by: context canceled

The csi provisioner has this:
E0927 15:58:28.337564 1 controller.go:957] error syncing claim "aa928776-736e-4b4f-9e5d-b19998b8df58": failed to provision volume with StorageClass "gp3-encrypted": rpc error: code = DeadlineExceeded desc = context deadline exceeded

This seems to be similar to this issue #954

What you expected to happen?
Pod should run with PVC bound and EBS Volumes Created when provisioned dynamically

How to reproduce it (as minimally and precisely as possible)?

• Upgrade to EKS 1.24
• Use EC2 instance role and the role has AmazonEBSCSIDriverPolicy.

Anything else we need to know?:
I've multiple other clusters running using instance role where as AWS docs suggest using service accounts is the best option. Is using instance role no longer supported?

Environment

• Kubernetes version (use kubectl version): 1.24
• Driver version: 1.19

any guidance would be much appreciated!

[ddl-lnarayanan]

Issue with CoreDNS add-on that was manifesting in a weird way. Fixing the coredns addon resolved this issue