ECS Probe - Rate Limit

[pecigonzalo]

I think scope is polling ECS too often and this could trigger rate limiting issues. Unfortunately, I could not find the exact rate for ECS, but I had problems with that API already in the past when implementing some event based actions.

If you have many failed task which keeps polling from ECS instead of cache at the current rate, times the number of nodes in the cluster in a cluster with 4+ nodes I have experienced hitting the rate limit with rotating tasks or similar

As a suggestion scope could cache the failed hit count and if it reaches X stop polling for some time.

[rade]

Some more general discussion on ECS API throttling, and how to avoid it, are in #2050.

[pecigonzalo]

Good ideas, I actually use CW Event Streams for some ECS magic but still believe we might need to throttle back our querying for failed events as they happen every couple of seconds and they are not covered by cache.

[carsten-neuendorf]

That answer I got from AWS support:

Hi there,
This is Marcos from AWS Support and I will help you today. I understand you're seeing "Rate Exceeded" when trying to work with some services from the ECS Dashboard. The "Rate Exceeded" means your API call "DescribeServices" are being throttled.
This results in a decrease of the rate you can execute it. Unfortunately, I am not able to check how many API calls are being made, as this is not logged by CloudTrail [2].

We throttle API requests for each AWS account on a per-region basis to help the performance of the service. We ensure that API calls don't exceed the maximum allowed API request rate. The maximum API request rate may vary across regions and can be automatically adjusted based a number of factors. This usually occurs when some external requester tries to execute the "DescribeServices" repeatedly.

Would you have any monitoring service or automation script that could be executing many calls in short period of time? If yes, a general recommendation would be to implement an exponential backoff and retry strategy within your code to allow them to go through. The pseudo-code below is an example how this can be implemented and you can see more information on the link [1].

MakeAwsApiCall if throttled: wait 2 sec;
MakeAwsApiCall if throttled: wait 4 sec;
MakeAwsApiCall if throttled: wait 8 sec;
MakeAwsApiCall if throttled: wait 16 sec;
MakeAwsApiCall if throttled: wait 32 sec;
MakeAwsApiCall if throttled: wait 32 sec;
MakeAwsApiCall if throttled: wait 32 sec;

The limit can be increased, but it would temporarily delay the issue which you might see again when you hit the increased limits. So I suggest trying the solution above first and contacting us again if that doesn't solve your issue.