Use AWSRetry on ec2 launch failures due to lack of capacity (InsuffienctInstanceCapacity)

[briantist]

Summary

Similar to #927

I've hit some errors in launching ec2 instances where AWS was out of capacity for the specific instance type in that AZ (InsuffienctInstanceCapacity), however, it failed with the boto default of 4 retries. Like other operations, it could be useful to have more than 4, and it would be nice to have consistency with the other operations.

I am not sure exactly where in the code this condition would be caught so I don't have a line to point to right now.

Issue Type

Feature Idea

Component Name

ec2_instance.py
ec2.py

Additional Information

The error was: botocore.exceptions.ClientError: An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 4): We currently do not have sufficient m6i.xlarge capacity in the Availability Zone you requested (us-east-1c). Our system will be working on provisioning additional capacity. You can currently get m6i.xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-1a, us-east-1b, us-east-1d, us-east-1f.

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibullbot]

Files identified in the description:

• [plugins/doc_fragments/ec2.py](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/doc_fragments/ec2.py)
• [plugins/modules/ec2_instance.py](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/modules/ec2_instance.py)

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibullbot]

cc @jillr @ryansb @s-hertel @tremble
click here for bot help

[tremble]

My experience with InsuffienctInstanceCapacity is that waiting a few minutes wouldn't be sufficient. Have you experienced cases where you've been able to grab capacity after retrying?

[tremble]

An example of how to inject additional codes to retry can be found in ec2_vpc_net

    connection = module.client(
        'ec2',
        retry_decorator=AWSRetry.jittered_backoff(
            retries=8, delay=3, catch_extra_error_codes=['InvalidVpcID.NotFound']
        ),
    )

( Different error code needed ;) )

[briantist]

My experience with InsuffienctInstanceCapacity is that waiting a few minutes wouldn't be sufficient. Have you experienced cases where you've been able to grab capacity after retrying?

Yes, this came from a CI pipeline I have (internal), after several tries, most of them did find capacity (it was probably launching a few dozen instances). In the end, these are test/molecule instances and the instance type isn't terrible important, so I changed from m6i to m6a and it was fine.

I do agree that retrying is not always going to work (or perhaps not even usually), but as long as the number of retries is not configurable by end users, it's difficult to deal with these things without making internal changes to the code, and I'd prefer that the number of internal retries is generally consistent.