fix(s3): added retry logic for replication policy race condition

[edwardofclt]

Description

Adds retry logic to provider to ensure AWS has time to return updated info about the S3 replication configuration.

Relations

Closes #30646

Output from Acceptance Testing

TF_ACC=1 go test ./internal/service/s3/... -v -count 1 -parallel 20 -run='TestAccS3BucketReplicationConfiguration'  -timeout 180m
=== RUN   TestAccS3BucketReplicationConfiguration_basic
=== PAUSE TestAccS3BucketReplicationConfiguration_basic
=== RUN   TestAccS3BucketReplicationConfiguration_disappears
=== PAUSE TestAccS3BucketReplicationConfiguration_disappears
=== RUN   TestAccS3BucketReplicationConfiguration_multipleDestinationsEmptyFilter
=== PAUSE TestAccS3BucketReplicationConfiguration_multipleDestinationsEmptyFilter
=== RUN   TestAccS3BucketReplicationConfiguration_multipleDestinationsNonEmptyFilter
=== PAUSE TestAccS3BucketReplicationConfiguration_multipleDestinationsNonEmptyFilter
=== RUN   TestAccS3BucketReplicationConfiguration_twoDestination
=== PAUSE TestAccS3BucketReplicationConfiguration_twoDestination
=== RUN   TestAccS3BucketReplicationConfiguration_configurationRuleDestinationAccessControlTranslation
=== PAUSE TestAccS3BucketReplicationConfiguration_configurationRuleDestinationAccessControlTranslation
=== RUN   TestAccS3BucketReplicationConfiguration_configurationRuleDestinationAddAccessControlTranslation
=== PAUSE TestAccS3BucketReplicationConfiguration_configurationRuleDestinationAddAccessControlTranslation
=== RUN   TestAccS3BucketReplicationConfiguration_replicationTimeControl
=== PAUSE TestAccS3BucketReplicationConfiguration_replicationTimeControl
=== RUN   TestAccS3BucketReplicationConfiguration_replicaModifications
=== PAUSE TestAccS3BucketReplicationConfiguration_replicaModifications
=== RUN   TestAccS3BucketReplicationConfiguration_withoutId
=== PAUSE TestAccS3BucketReplicationConfiguration_withoutId
=== RUN   TestAccS3BucketReplicationConfiguration_withoutStorageClass
=== PAUSE TestAccS3BucketReplicationConfiguration_withoutStorageClass
=== RUN   TestAccS3BucketReplicationConfiguration_schemaV2
=== PAUSE TestAccS3BucketReplicationConfiguration_schemaV2
=== RUN   TestAccS3BucketReplicationConfiguration_schemaV2SameRegion
=== PAUSE TestAccS3BucketReplicationConfiguration_schemaV2SameRegion
=== RUN   TestAccS3BucketReplicationConfiguration_schemaV2DestinationMetrics
=== PAUSE TestAccS3BucketReplicationConfiguration_schemaV2DestinationMetrics
=== RUN   TestAccS3BucketReplicationConfiguration_existingObjectReplication
    bucket_replication_configuration_test.go:766: skipping test: AWS Technical Support request required to allow ExistingObjectReplication
--- SKIP: TestAccS3BucketReplicationConfiguration_existingObjectReplication (0.00s)
=== RUN   TestAccS3BucketReplicationConfiguration_filter_emptyConfigurationBlock
=== PAUSE TestAccS3BucketReplicationConfiguration_filter_emptyConfigurationBlock
=== RUN   TestAccS3BucketReplicationConfiguration_filter_emptyPrefix
=== PAUSE TestAccS3BucketReplicationConfiguration_filter_emptyPrefix
=== RUN   TestAccS3BucketReplicationConfiguration_filter_tagFilter
=== PAUSE TestAccS3BucketReplicationConfiguration_filter_tagFilter
=== RUN   TestAccS3BucketReplicationConfiguration_filter_andOperator
=== PAUSE TestAccS3BucketReplicationConfiguration_filter_andOperator
=== RUN   TestAccS3BucketReplicationConfiguration_filter_withoutId
=== PAUSE TestAccS3BucketReplicationConfiguration_filter_withoutId
=== RUN   TestAccS3BucketReplicationConfiguration_withoutPrefix
=== PAUSE TestAccS3BucketReplicationConfiguration_withoutPrefix
=== RUN   TestAccS3BucketReplicationConfiguration_migrate_noChange
=== PAUSE TestAccS3BucketReplicationConfiguration_migrate_noChange
=== RUN   TestAccS3BucketReplicationConfiguration_migrate_withChange
=== PAUSE TestAccS3BucketReplicationConfiguration_migrate_withChange
=== CONT  TestAccS3BucketReplicationConfiguration_basic
=== CONT  TestAccS3BucketReplicationConfiguration_schemaV2
=== CONT  TestAccS3BucketReplicationConfiguration_configurationRuleDestinationAddAccessControlTranslation
=== CONT  TestAccS3BucketReplicationConfiguration_withoutId
=== CONT  TestAccS3BucketReplicationConfiguration_filter_andOperator
=== CONT  TestAccS3BucketReplicationConfiguration_filter_tagFilter
=== CONT  TestAccS3BucketReplicationConfiguration_withoutStorageClass
=== CONT  TestAccS3BucketReplicationConfiguration_replicaModifications
=== CONT  TestAccS3BucketReplicationConfiguration_replicationTimeControl
=== CONT  TestAccS3BucketReplicationConfiguration_multipleDestinationsNonEmptyFilter
=== CONT  TestAccS3BucketReplicationConfiguration_schemaV2SameRegion
=== CONT  TestAccS3BucketReplicationConfiguration_configurationRuleDestinationAccessControlTranslation
=== CONT  TestAccS3BucketReplicationConfiguration_filter_emptyPrefix
=== CONT  TestAccS3BucketReplicationConfiguration_filter_emptyConfigurationBlock
=== CONT  TestAccS3BucketReplicationConfiguration_schemaV2DestinationMetrics
=== CONT  TestAccS3BucketReplicationConfiguration_multipleDestinationsEmptyFilter
=== CONT  TestAccS3BucketReplicationConfiguration_migrate_withChange
=== CONT  TestAccS3BucketReplicationConfiguration_disappears
=== CONT  TestAccS3BucketReplicationConfiguration_migrate_noChange
=== CONT  TestAccS3BucketReplicationConfiguration_twoDestination
--- PASS: TestAccS3BucketReplicationConfiguration_disappears (99.75s)
=== CONT  TestAccS3BucketReplicationConfiguration_filter_withoutId
--- PASS: TestAccS3BucketReplicationConfiguration_schemaV2SameRegion (187.33s)
=== CONT  TestAccS3BucketReplicationConfiguration_withoutPrefix
--- PASS: TestAccS3BucketReplicationConfiguration_filter_emptyConfigurationBlock (312.59s)
--- PASS: TestAccS3BucketReplicationConfiguration_withoutId (316.80s)
--- PASS: TestAccS3BucketReplicationConfiguration_twoDestination (320.33s)
--- PASS: TestAccS3BucketReplicationConfiguration_multipleDestinationsNonEmptyFilter (322.57s)
--- PASS: TestAccS3BucketReplicationConfiguration_filter_emptyPrefix (323.72s)
--- PASS: TestAccS3BucketReplicationConfiguration_withoutStorageClass (323.78s)
--- PASS: TestAccS3BucketReplicationConfiguration_filter_tagFilter (324.86s)
--- PASS: TestAccS3BucketReplicationConfiguration_multipleDestinationsEmptyFilter (324.88s)
--- PASS: TestAccS3BucketReplicationConfiguration_schemaV2 (325.31s)
--- PASS: TestAccS3BucketReplicationConfiguration_schemaV2DestinationMetrics (325.76s)
--- PASS: TestAccS3BucketReplicationConfiguration_replicationTimeControl (325.95s)
--- PASS: TestAccS3BucketReplicationConfiguration_replicaModifications (325.97s)
--- PASS: TestAccS3BucketReplicationConfiguration_migrate_withChange (335.33s)
--- PASS: TestAccS3BucketReplicationConfiguration_migrate_noChange (336.82s)
--- PASS: TestAccS3BucketReplicationConfiguration_configurationRuleDestinationAddAccessControlTranslation (344.08s)
--- PASS: TestAccS3BucketReplicationConfiguration_configurationRuleDestinationAccessControlTranslation (345.14s)
--- PASS: TestAccS3BucketReplicationConfiguration_filter_andOperator (345.28s)
--- PASS: TestAccS3BucketReplicationConfiguration_basic (355.91s)
--- PASS: TestAccS3BucketReplicationConfiguration_filter_withoutId (267.14s)
--- PASS: TestAccS3BucketReplicationConfiguration_withoutPrefix (264.27s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/s3	458.775s

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this pull request by adding a 👍 reaction to the original post to help the community and maintainers prioritize this pull request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

For Submitters

• Review the contribution guide relating to the type of change you are making to ensure all of the necessary steps have been taken.
• For new resources and data sources, use skaff to generate scaffolding with comments detailing common expectations.
• Whether or not the branch has been rebased will not impact prioritization, but doing so is always a welcome surprise.

[edwardofclt]

@justinretzolk When can this get reviewed & merged? TF building resources with newly configured S3 replication requires two runs before it applies successfully.

[justinretzolk]

Hey @edwardofclt 👋 Thank you for taking the time to raise this, and for checking in on it! Unfortunately I can't provide an ETA on when this will be reviewed/merged due to the potential of shifting priorities. We prioritize by count of 👍 reactions and a few other things (more information on our prioritization guide if you're interested).

[johnsonaj]

LGTM 🚀

make testacc TESTARGS='-run=TestAccS3BucketReplicationConfiguration_' PKG=s3

==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/s3/... -v -count 1 -parallel 20  -run=TestAccS3BucketReplicationConfiguration_ -timeout 180m
    bucket_replication_configuration_test.go:766: skipping test: AWS Technical Support request required to allow ExistingObjectReplication
--- SKIP: TestAccS3BucketReplicationConfiguration_existingObjectReplication (0.00s)
--- PASS: TestAccS3BucketReplicationConfiguration_disappears (78.72s)
--- PASS: TestAccS3BucketReplicationConfiguration_schemaV2SameRegion (175.10s)
--- PASS: TestAccS3BucketReplicationConfiguration_filter_emptyPrefix (310.91s)
--- PASS: TestAccS3BucketReplicationConfiguration_withoutStorageClass (314.58s)
--- PASS: TestAccS3BucketReplicationConfiguration_schemaV2DestinationMetrics (315.20s)
--- PASS: TestAccS3BucketReplicationConfiguration_twoDestination (315.37s)
--- PASS: TestAccS3BucketReplicationConfiguration_replicaModifications (316.25s)
--- PASS: TestAccS3BucketReplicationConfiguration_withoutId (316.33s)
--- PASS: TestAccS3BucketReplicationConfiguration_filter_tagFilter (316.66s)
--- PASS: TestAccS3BucketReplicationConfiguration_replicationTimeControl (316.97s)
--- PASS: TestAccS3BucketReplicationConfiguration_multipleDestinationsEmptyFilter (316.95s)
--- PASS: TestAccS3BucketReplicationConfiguration_filter_emptyConfigurationBlock (317.22s)
--- PASS: TestAccS3BucketReplicationConfiguration_schemaV2 (317.72s)
--- PASS: TestAccS3BucketReplicationConfiguration_multipleDestinationsNonEmptyFilter (318.27s)
--- PASS: TestAccS3BucketReplicationConfiguration_migrate_withChange (328.02s)
--- PASS: TestAccS3BucketReplicationConfiguration_migrate_noChange (331.50s)
--- PASS: TestAccS3BucketReplicationConfiguration_filter_andOperator (338.83s)
--- PASS: TestAccS3BucketReplicationConfiguration_configurationRuleDestinationAddAccessControlTranslation (338.94s)
--- PASS: TestAccS3BucketReplicationConfiguration_configurationRuleDestinationAccessControlTranslation (340.81s)
--- PASS: TestAccS3BucketReplicationConfiguration_withoutPrefix (268.52s)
--- PASS: TestAccS3BucketReplicationConfiguration_basic (350.59s)
--- PASS: TestAccS3BucketReplicationConfiguration_filter_withoutId (265.47s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/s3	448.209s

[johnsonaj]

@edwardofclt thanks for the contribution! 🎉

[github-actions]

This functionality has been released in v5.1.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.