[Bug]: s3_bucket_versioning fails to delete due to a "A conflicting conditional operation"

[jasonarewhy]

Terraform Core Version

1.3.9

AWS Provider Version

5.17.0

Affected Resource(s)

• s3_bucket_versioning
• (possibly other bucket-based resources)

Expected Behavior

s3_bucket_versioning can be destroyed without error.

An example of the resources being destroyed:

Terraform will perform the following actions:

  # module.s3_logging.aws_s3_bucket.bucket will be destroyed
  - resource "aws_s3_bucket" "bucket" {
      - arn                         = "arn:aws:s3:::example-bucket" -> null
      - bucket                      = "example-bucket" -> null
      - bucket_domain_name          = "example-bucket.s3.amazonaws.com" -> null
      - bucket_regional_domain_name = "example-bucket.s3.us-west-2.amazonaws.com" -> null
      - force_destroy               = true -> null
      - hosted_zone_id              = "REDACTED" -> null
      - id                          = "example-bucket" -> null
      - object_lock_enabled         = false -> null
      - region                      = "us-west-2" -> null
      - request_payer               = "BucketOwner" -> null

      - grant {
          - id          = "id" -> null
          - permissions = [
              - "FULL_CONTROL",
            ] -> null
          - type        = "CanonicalUser" -> null
        }

      - lifecycle_rule {
          - abort_incomplete_multipart_upload_days = 0 -> null
          - enabled                                = false -> null
          - id                                     = "glacier" -> null
          - tags                                   = {} -> null

          - transition {
              - days          = 90 -> null
              - storage_class = "GLACIER" -> null
            }
        }
      - lifecycle_rule {
          - abort_incomplete_multipart_upload_days = 0 -> null
          - enabled                                = false -> null
          - id                                     = "expiration" -> null
          - tags                                   = {} -> null

          - expiration {
              - days                         = 180 -> null
              - expired_object_delete_marker = false -> null
            }
        }
      - lifecycle_rule {
          - abort_incomplete_multipart_upload_days = 0 -> null
          - enabled                                = false -> null
          - id                                     = "expire_non_current_versions" -> null
          - tags                                   = {} -> null

          - noncurrent_version_expiration {
              - days = 180 -> null
            }
        }

      - server_side_encryption_configuration {
          - rule {
              - bucket_key_enabled = false -> null

              - apply_server_side_encryption_by_default {
                  - kms_master_key_id = "arn:aws:kms:us-west-2:REDACTED:key/kms-key-id" -> null
                  - sse_algorithm     = "aws:kms" -> null
                }
            }
        }

      - versioning {
          - enabled    = false -> null
          - mfa_delete = false -> null
        }
    }

  # module.s3_logging.aws_s3_bucket_lifecycle_configuration.s3_bucket_lifecycle_rules will be destroyed
  - resource "aws_s3_bucket_lifecycle_configuration" "s3_bucket_lifecycle_rules" {
      - bucket = "example-bucket" -> null
      - id     = "example-bucket" -> null

      - rule {
          - id     = "glacier" -> null
          - status = "Disabled" -> null

          - filter {
            }

          - transition {
              - days          = 90 -> null
              - storage_class = "GLACIER" -> null
            }
        }
      - rule {
          - id     = "expiration" -> null
          - status = "Disabled" -> null

          - expiration {
              - days                         = 180 -> null
              - expired_object_delete_marker = false -> null
            }

          - filter {
            }
        }
      - rule {
          - id     = "expire_non_current_versions" -> null
          - status = "Disabled" -> null

          - filter {
            }

          - noncurrent_version_expiration {
              - noncurrent_days = 180 -> null
            }
        }
    }

  # module.s3_logging.aws_s3_bucket_ownership_controls.bucket_ownership will be destroyed
  - resource "aws_s3_bucket_ownership_controls" "bucket_ownership" {
      - bucket = "example-bucket" -> null
      - id     = "example-bucket" -> null

      - rule {
          - object_ownership = "BucketOwnerEnforced" -> null
        }
    }

  # module.s3_logging.aws_s3_bucket_server_side_encryption_configuration.s3_bucket_ss_encryption will be destroyed
  - resource "aws_s3_bucket_server_side_encryption_configuration" "s3_bucket_ss_encryption" {
      - bucket = "example-bucket" -> null
      - id     = "example-bucket" -> null

      - rule {
          - bucket_key_enabled = false -> null

          - apply_server_side_encryption_by_default {
              - kms_master_key_id = "arn:aws:kms:us-west-2:REDACTED:key/kms-key-id" -> null
              - sse_algorithm     = "aws:kms" -> null
            }
        }
    }

  # module.s3_logging.aws_s3_bucket_versioning.s3_bucket_versioning will be destroyed
  - resource "aws_s3_bucket_versioning" "s3_bucket_versioning" {
      - bucket = "example-bucket" -> null
      - id     = "example-bucket" -> null

      - versioning_configuration {
          - status = "Suspended" -> null
        }
    }

Actual Behavior

s3_bucket_versioning destroy operation errors out with:

Error: deleting S3 Bucket Versioning (example-bucket): operation error S3: PutBucketVersioning, https response error StatusCode: 409... api error OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again.

This seems to have been introduced in 5.17.0. I've tested with both 5.16.0 and 5.14.0 and was able to destroy without an issues. Looks like a reintroduction of #7628.

Relevant Error/Panic Output Snippet

Error: deleting S3 Bucket Versioning (example-bucket): operation error S3: PutBucketVersioning, https response error StatusCode: 409... api error OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again.

Terraform Configuration Files

example.zip

Steps to Reproduce

• Create a bucket with multiple s3_bucket... resources (see plan above as an example)
• attempt to destroy all the resources

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[ewbankkit]

@jasonarewhy Thanks for raising this issue 👏.
You are correct, I neglected to migrate this functionality

terraform-provider-aws/internal/service/s3/service_package.go

Lines 35 to 43 in 51c0d52

	func (p *servicePackage) CustomizeConn(ctx context.Context, conn *s3_sdkv1.S3) (*s3_sdkv1.S3, error) {
	conn.Handlers.Retry.PushBack(func(r *request_sdkv1.Request) {
	if tfawserr.ErrMessageContains(r.Error, errCodeOperationAborted, "A conflicting conditional operation is currently in progress against this resource. Please try again.") {
	r.Retryable = aws_sdkv1.Bool(true)
	}
	})
	return conn, nil
	}

when migrating the aws_s3_bucket_versioning resource to AWS SDK for Go v2.

Relevant SDK documentation.

[rsheldon-sincro]

I have run into similar issues with other S3 related resources while upgrading some modules to AWS Provider 5. I will be holding off doing any more until these get resolved. e.g.

This one was on an S3 bucket with a aws_s3_bucket_policy also defined.

error 16-Sep-2023 01:14:09 Error: putting S3 Bucket (mybucket) Policy: operation error S3: PutBucketPolicy, https response error StatusCode: 409, ..., api error OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again.

I got these when destroying my module

│ Error: deleting S3 Bucket (mybucket) Policy: operation error S3: DeleteBucketPolicy, https response error StatusCode: 409, ..., api error OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again.

│ Error: deleting S3 Bucket Versioning (mybucket): operation error S3: PutBucketVersioning, https response error StatusCode: 409, ..., api error OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again.

Hopefully these issues will also be addressed by the code you didn't migrate.

[morremeyer]

Can confirm that this also happens when creating entirely new resources. We do so when testing our terraform modules with terratest:

TestS3BucketBackupOffVersioningOn 2023-09-20T09:44:16Z logger.go:66: aws_s3_bucket.this: Creating...
TestS3BucketBackupOffVersioningOn 2023-09-20T09:44:19Z logger.go:66: aws_s3_bucket.this: Creation complete after 3s [id=5b1b1b77-6736-47be-96d9-7393e18fee9b]
TestS3BucketBackupOffVersioningOn 2023-09-20T09:44:19Z logger.go:66: aws_s3_bucket_public_access_block.this: Creating...
TestS3BucketBackupOffVersioningOn 2023-09-20T09:44:19Z logger.go:66: aws_s3_bucket_versioning.this: Creating...
TestS3BucketBackupOffVersioningOn 2023-09-20T09:44:19Z logger.go:66: aws_s3_bucket_server_side_encryption_configuration.this: Creating...
TestS3BucketBackupOffVersioningOn 2023-09-20T09:44:19Z logger.go:66: aws_s3_bucket_public_access_block.this: Creation complete after 1s [id=5b1b1b77-6736-47be-96d9-7393e18fee9b]
TestS3BucketBackupOffVersioningOn 2023-09-20T09:44:20Z logger.go:66: aws_s3_bucket_server_side_encryption_configuration.this: Creation complete after 1s [id=5b1b1b77-6736-47be-96d9-7393e18fee9b]
TestS3BucketBackupOffVersioningOn 2023-09-20T09:44:20Z logger.go:66: 
TestS3BucketBackupOffVersioningOn 2023-09-20T09:44:20Z logger.go:66: Error: creating S3 Bucket (5b1b1b77-6736-47be-96d9-7393e18fee9b) Versioning: operation error S3: PutBucketVersioning, https response error StatusCode: 409, RequestID: FYX97164ND0TQK5A, HostID: d5wI7RgQuVFU7Bp5U1q8bjZDmd6LR0mBZIJM4cYEZSM8vbZtnMEZiHDWD/H7buFQGbzS//hEeDQ=, api error OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again.
TestS3BucketBackupOffVersioningOn 2023-09-20T09:44:20Z logger.go:66: 
TestS3BucketBackupOffVersioningOn 2023-09-20T09:44:20Z logger.go:66:   with aws_s3_bucket_versioning.this,
TestS3BucketBackupOffVersioningOn 2023-09-20T09:44:20Z logger.go:66:   on main.tf line 43, in resource "aws_s3_bucket_versioning" "this":
TestS3BucketBackupOffVersioningOn 2023-09-20T09:44:20Z logger.go:66:   43: resource "aws_s3_bucket_versioning" "this" {

[kneekeel]

I am experiencing the same error when using terraform-provider-aws v5.17.0

[github-actions]

This functionality has been released in v5.18.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.