Enable retries for Fargate Agent

[lukeorland]

Current behavior

We invoked client.create_flow_run in a for loop with 365 different parameter values. This caused 258 of them to fail with one of two different errors. In each case there was only run log message for the whole flow run.

March 17th 2020 at 2:32:51am | agent
ERROR 
An error occurred (ThrottlingException) when calling the RunTask operation (reached max retries: 4): Rate exceeded.

March 17th 2020 at 2:32:59am | agent
ERROR 
list index out of rangeMarch 17th 2020 at 2:32:59am | agent
ERROR 
list index out of range

Apparently, for the flow runs that the FargateAgent received an error from the AWS API, the FargateAgent quit and sent the error back to Prefect Cloud.

Proposed behavior

When the FargateAgent receives a throttling error response from AWS, it could retry the request (perhaps using backoff). It appears that botocore already has retry functionality:

boto/botocore#1260

Example

If this were implemented, it would cover the case when a team is running a backfill workflow for every day in a given year, all at once.

[joshmeek]

👍 Thanks for opening. Yeah I think we should take advantage of boto's retry configuration