AssumeRoleForPodIdentity context canceled errors

[zvickery]

Hi, I am testing EKS Pod Identity for production rollout. However, I have noticed that when many of our EKS cron jobs start, we see throttling errors in the EKS Pod Identity agent logs. With the most recent release (v0.1.21), I now see the below errors during peak moments of pod startup:

{"client-addr":"10.10.42.5:36146","cluster-name":"my-eks","level":"info","msg":"Calling EKS Auth to fetch credentials","time":"2025-02-19T19:30:37Z"}
{"client-addr":"10.10.34.0:52004","cluster-name":"my-eks","level":"info","msg":"handling new request request from 10.10.34.0:52004","time":"2025-02-19T19:30:37Z"}
{"client-addr":"10.10.34.0:52004","cluster-name":"my-eks","level":"info","msg":"Calling EKS Auth to fetch credentials","time":"2025-02-19T19:30:38Z"}
{"client-addr":"10.10.42.1:43120","cluster-name":"my-eks","level":"info","msg":"handling new request request from 10.10.42.1:43120","time":"2025-02-19T19:30:37Z"}
{"client-addr":"10.10.42.1:43120","cluster-name":"my-eks","level":"info","msg":"Calling EKS Auth to fetch credentials","time":"2025-02-19T19:30:38Z"}
{"client-addr":"10.10.42.5:36146","cluster-name":"my-eks","level":"error","msg":"Error fetching credentials: error getting credentials to cache: unable to fetch credentials from EKS Auth: operation error EKS Auth: AssumeRoleForPodIdentity, https response error Stat
usCode: 0, RequestID: , canceled, context canceled","operation":"AssumeRoleForPodIdentity","request-id":"","service":"EKS Auth","time":"2025-02-19T19:30:38Z"}
{"client-addr":"10.10.42.1:43120","cluster-name":"my-eks","level":"error","msg":"Error fetching credentials: error getting credentials to cache: unable to fetch credentials from EKS Auth: operation error EKS Auth: AssumeRoleForPodIdentity, https response error Stat
usCode: 0, RequestID: , canceled, context canceled","operation":"AssumeRoleForPodIdentity","request-id":"","service":"EKS Auth","time":"2025-02-19T19:30:38Z"}
{"client-addr":"10.10.34.162:54070","cluster-name":"my-eks","level":"error","msg":"Error fetching credentials: error getting credentials to cache: unable to fetch credentials from EKS Auth: operation error EKS Auth: AssumeRoleForPodIdentity, https response error St
atusCode: 0, RequestID: , canceled, context canceled","operation":"AssumeRoleForPodIdentity","request-id":"","service":"EKS Auth","time":"2025-02-19T19:30:39Z"}
{"client-addr":"10.10.34.0:52006","cluster-name":"my-eks","level":"info","msg":"handling new request request from 10.10.34.0:52006","time":"2025-02-19T19:30:39Z"}

While the message is unclear, I am pretty confident these errors are throttling related. They go away on their own but leave me feeling uneasy about this solution compared to IRSA. Is there anything that can be done to combat them?