Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test/resource/aws_s3_bucket: Add EMR bootstrap prefix to sweeper list #10106

Merged
merged 1 commit into from
Sep 24, 2019

Conversation

nywilken
Copy link
Contributor

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" comments, they generate extra noise for pull request followers and do not help prioritize the request

Relates OR Closes #0000

Release note for CHANGELOG:

none

Currently the EMR cluster test configuration creates a s3 bucket that is publicly accessible. These buckets are not currently cleaned up after testing so this change updates the s3 sweeper to remove test buckets starting with the prefix tf-emr-bootstrap. In the long term, we should determine if EMR bootstrapping supports non-public buckets and update the test configurations with the proper bucket permissions.

Before the change:

> go test ./aws -v -sweep=us-gov-west-1 -sweep-run=aws_s3_bucket -timeout 10h
2019/09/13 13:22:08 [DEBUG] Running Sweepers for region (us-gov-west-1):
2019/09/13 13:22:08 [DEBUG] Sweeper (aws_s3_bucket) has dependency (aws_s3_bucket_object), running..
2019/09/13 13:22:08 [INFO] Building AWS auth structure
2019/09/13 13:22:08 [INFO] Setting AWS metadata API timeout to 100ms
2019/09/13 13:22:08 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
...
2019/09/13 13:22:11 [INFO] Skipping S3 Bucket: tf-emr-bootstrap-396768694530128871
...
2019/09/13 13:22:13 [DEBUG] Sweeper (aws_s3_bucket_object) already ran in region (us-gov-west-1)
2019/09/13 13:22:13 Sweeper Tests ran:
	- aws_s3_bucket_object
	- aws_s3_bucket
ok  	github.com/terraform-providers/terraform-provider-aws/aws	5.138s

After the change

> go test ./aws -v -sweep=us-gov-west-1 -sweep-run=aws_s3_bucket -timeout 10h
2019/09/13 13:34:30 [DEBUG] Running Sweepers for region (us-gov-west-1):
2019/09/13 13:34:30 [DEBUG] Sweeper (aws_s3_bucket) has dependency (aws_s3_bucket_object), running..
2019/09/13 13:34:30 [INFO] Building AWS auth structure
2019/09/13 13:34:30 [INFO] Setting AWS metadata API timeout to 100ms
2019/09/13 13:34:30 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
...
2019/09/13 13:34:33 [INFO] Deleting S3 Bucket (tf-emr-bootstrap-396768694530128871) Object (testscript.sh) Version: null
...
2019/09/13 13:34:38 [DEBUG] Sweeper (aws_s3_bucket_object) already ran in region (us-gov-west-1)
2019/09/13 13:34:38 Sweeper Tests ran:
	- aws_s3_bucket_object
	- aws_s3_bucket
ok  	github.com/terraform-providers/terraform-provider-aws/aws	8.158s

Currently the EMR cluster test configuration creates a s3 bucket that is publicly accessible. These buckets are not currently cleaned up after testing so this change updates the s3 sweeper to remove test buckets starting with the prefix `tf-emr-bootstrap`. In the long term, we should determine if EMR bootstrapping supports non-public buckets and update the test configurations with the proper bucket permissions.

Before the change:
```
> go test ./aws -v -sweep=us-gov-west-1 -sweep-run=aws_s3_bucket -timeout 10h
2019/09/13 13:22:08 [DEBUG] Running Sweepers for region (us-gov-west-1):
2019/09/13 13:22:08 [DEBUG] Sweeper (aws_s3_bucket) has dependency (aws_s3_bucket_object), running..
2019/09/13 13:22:08 [INFO] Building AWS auth structure
2019/09/13 13:22:08 [INFO] Setting AWS metadata API timeout to 100ms
2019/09/13 13:22:08 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
...
2019/09/13 13:22:11 [INFO] Skipping S3 Bucket: tf-emr-bootstrap-396768694530128871
...
2019/09/13 13:22:13 [DEBUG] Sweeper (aws_s3_bucket_object) already ran in region (us-gov-west-1)
2019/09/13 13:22:13 Sweeper Tests ran:
	- aws_s3_bucket_object
	- aws_s3_bucket
ok  	github.com/terraform-providers/terraform-provider-aws/aws	5.138s
```

After the change
```
> go test ./aws -v -sweep=us-gov-west-1 -sweep-run=aws_s3_bucket -timeout 10h
2019/09/13 13:34:30 [DEBUG] Running Sweepers for region (us-gov-west-1):
2019/09/13 13:34:30 [DEBUG] Sweeper (aws_s3_bucket) has dependency (aws_s3_bucket_object), running..
2019/09/13 13:34:30 [INFO] Building AWS auth structure
2019/09/13 13:34:30 [INFO] Setting AWS metadata API timeout to 100ms
2019/09/13 13:34:30 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
...
2019/09/13 13:34:33 [INFO] Deleting S3 Bucket (tf-emr-bootstrap-396768694530128871) Object (testscript.sh) Version: null
...
2019/09/13 13:34:38 [DEBUG] Sweeper (aws_s3_bucket_object) already ran in region (us-gov-west-1)
2019/09/13 13:34:38 Sweeper Tests ran:
	- aws_s3_bucket_object
	- aws_s3_bucket
ok  	github.com/terraform-providers/terraform-provider-aws/aws	8.158s
```
@nywilken nywilken requested a review from a team September 13, 2019 17:48
@ghost ghost added size/XS Managed by automation to categorize the size of a PR. service/s3 Issues and PRs that pertain to the s3 service. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. labels Sep 13, 2019
Copy link
Contributor

@bflad bflad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

@nywilken nywilken merged commit 6fe866b into master Sep 24, 2019
@nywilken nywilken deleted the t-aws_s3_bucket-emr-sweeper-prefix branch September 24, 2019 22:51
@nywilken nywilken added this to the v2.30.0 milestone Sep 25, 2019
@ghost
Copy link

ghost commented Sep 26, 2019

This has been released in version 2.30.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

@ghost
Copy link

ghost commented Nov 1, 2019

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Nov 1, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
service/s3 Issues and PRs that pertain to the s3 service. size/XS Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants