GCS Bucket - delete operation stuck in never-ending loop when it can't delete objects

[stepanstipl]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

$ terraform -v
Terraform v0.12.24
+ provider.google v3.15.0

Affected Resource(s)

• google_storage_bucket

Terraform Configuration Files

resource "google_storage_bucket" "test" {
  project       = "stepan-playground"
  name          = "stepan-playground-tfdelete"
  location      = "EU"
  force_destroy = true

  bucket_policy_only = true
}

Debug Output

https://gist.github.com/390796b9d6563d21571b1d2c28cae892

Panic Output

Expected Behavior

Terraform will stop after non-retryable error and fail.

Actual Behavior

Terraform keeps running, stuck in the object delete loop, till interrupted.

Steps to Reproduce

1. Apply above shared code

   terraform apply
   

2. Create and upload object to newly created bucket

   echo "hello" > test.txt
   gsutil cp test.txt gs://stepan-playground-tfdelete/
   

3. Apply hold on object, to prevent it's deletion

   gsutil retention temp set gs://stepan-playground-tfdelete/test.txt
   

4. Run destroy

   terraform destroy
   

Important Factoids

Terraform will try to delete the object, and fails, correctly with non-retryable error 403 Forbidden, as it's not able to delete the object because of the temp hold. However this error is not surfaced [1] and the operation is stuck in never-ending for loop [2] and will keep trying to delete the object, eventually hitting rate limit and continue failing with 429 Too Many Requests errors, until manually interrupted.

References

• [1] https://github.com/terraform-providers/terraform-provider-google/blob/fc68ab472dda82b22dc95ce6d9312f2401cac33e/google/resource_storage_bucket.go#L674-L679

• [2] https://github.com/terraform-providers/terraform-provider-google/blob/fc68ab472dda82b22dc95ce6d9312f2401cac33e/google/resource_storage_bucket.go#L620

[edwardmedia]

I can repro it. It does make sense to handle it in a better way.

[stepanstipl]

Thanks @edwardmedia, @danawillow - I'm putting together a PR against magic-modules, should have that ready shortly.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!