Add Retry Mechanism to E2E EC2 Terraform Deployment

[harrryr]

Issue #, if available:
The EC2 Canary occasionally fails due to transitivity issues. Some of the recurring errors are Max attempts reached in the Step : Wait for Endpoint to Come Online and the Step: Timeout while waiting for state to become running. This occurs due to the endpoint and the ec2 instances sometime taking longer than expected to become ready.

Description of changes:

• Change the endpoint wait time from 5 minutes to 10 minutes.
• Add a retry mechanism such that if either the terraform deployment or the endpoint connection fails, then destroy the terraform deployment and try again. This will ensure that the endpoint is deleted and recreated.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov-commenter]

Codecov Report

Attention: 102 lines in your changes are missing coverage. Please review.

Comparison is base (09e6487) 85.71% compared to head (184678f) 50.73%.
Report is 177 commits behind head on main.

Files	Patch %	Lines
...ent/providers/AwsAppSignalsCustomizerProvider.java	24.00%	35 Missing and 3 partials ⚠️
...gent/providers/AwsSpanMetricsProcessorBuilder.java	0.00%	20 Missing ⚠️
...ders/AttributePropagatingSpanProcessorBuilder.java	0.00%	16 Missing ⚠️
...viders/AwsMetricAttributesSpanExporterBuilder.java	0.00%	11 Missing ⚠️
...try/javaagent/providers/AwsSpanProcessingUtil.java	90.16%	1 Missing and 5 partials ⚠️
...vaagent/providers/AwsMetricAttributeGenerator.java	96.89%	2 Missing and 3 partials ⚠️
...y/javaagent/providers/AwsSpanMetricsProcessor.java	91.48%	0 Missing and 4 partials ⚠️
...t/providers/AttributePropagatingSpanProcessor.java	94.59%	2 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@              Coverage Diff              @@
##               main     #635       +/-   ##
=============================================
- Coverage     85.71%   50.73%   -34.99%     
- Complexity       19      264      +245     
=============================================
  Files             3       39       +36     
  Lines            49     1301     +1252     
  Branches          5      141      +136     
=============================================
+ Hits             42      660      +618     
- Misses            3      609      +606     
- Partials          4       32       +28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[srprash]

Success should be made 0 after the terraform apply command has completed. That's is the assumption in the following code.

[harrryr]

Success of 1 indicates that the setting up App Signals on the sample app failed, while 0 indicates that everything ran successfully.

The logic is:

1. Set Success to 1 (Set initial value to 1 so that the while loop runs)
2. While Success is 1 (Indicates that terraform deployment/endpoint connection failed and will try again):
     2a: Set Success to 0 (Set the value to 0 and if there were any failures change it to 1)
     2b: Run Terraform apply (If the deployment failed, then success will change to 1)
     2c: If Success is still 0, then install App Signals and check endpoint connection
     2d: If endpoint connection failed, change success to 1
     2e: If Success is 1 at this point, then either the deployment or connection failed and run the while loop again. If it is still 0, then the code ran successfully and exit the while loop

If the success is made 0 after the terraform apply, then it will override whether terraform deployment succeeded or not. If after the terraform deployment the success is 1, we want to skip the endpoint connection step and redeploy the terraform again.

[srprash]

Is there a sample run where we can see this change in action?