Failing test(s): TestAccAccessApprovalSettings

[melinath]

Failure rates

• 22.6% in 2023-01
• 22.6% in 2023-03

Impacted tests:

• TestAccAccessApprovalSettings

Affected Resource(s)

• google_project_access_approval_settings

Nightly builds

• https://ci-oss.hashicorp.engineering/buildConfiguration/GoogleCloudBeta_ProviderGoogleCloudBetaGoogleProject/374144?buildTab=tests&pager.currentPage=1&name=TestAccAccessApprovalSettings&expandedTest=1176907843211229648

Message(s)

Error: Provider produced inconsistent result after apply
When applying changes to
google_project_access_approval_settings.project_access_approval, provider
"provider[\"registry.terraform.io/hashicorp/google\"]" produced an unexpected
new value: Root resource was present, but now absent.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.

b/304725248

[SarahFrench]

Other error messages appear like:

vcr_utils.go:144: Step 5/6 error: After applying this test step and performing a `terraform refresh`, the plan was not empty.
stdout
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# google_organization_access_approval_settings.organization_access_approval will be updated in-place
~ resource "google_organization_access_approval_settings" "organization_access_approval" {
id                              = "organizations/*******/accessApprovalSettings"
name                            = "organizations/*******/accessApprovalSettings"
# (6 unchanged attributes hidden)
- enrolled_services {
- cloud_product    = "App Engine" -> null
- enrollment_level = "BLOCK_ALL" -> null
}
- enrolled_services {
- cloud_product    = "Cloud Dataflow" -> null
- enrollment_level = "BLOCK_ALL" -> null
}
+ enrolled_services {
+ cloud_product    = "all"
+ enrollment_level = "BLOCK_ALL"
}
}
Plan: 0 to add, 1 to change, 0 to destroy.

I think that the underlying cause could be >1 test affecting the resource at once

• I think Root resource was present, but now absent (link) means that Terraform just planned to make a resource but found that it's missing from state afterwards - ie deleted from GCP by third party
• I think the non-empty plan error is due to two tests changing the same resource?

The diff shown above seems to be a result of these two test steps:

terraform-provider-google/google/resource_access_approval_organization_settings_test.go

Lines 77 to 93 in c79e58e

	func testAccAccessApprovalOrganizationSettings_full(context map[string]interface{}) string {
	return Nprintf(`
	resource "google_organization_access_approval_settings" "organization_access_approval" {
	organization_id = "%{org_id}"
	notification_emails = ["[email protected]"]
	enrolled_services {
	cloud_product = "App Engine"
	}
	enrolled_services {
	cloud_product = "dataflow.googleapis.com"
	enrollment_level = "BLOCK_ALL"
	}
	}
	`, context)
	}

terraform-provider-google/google/resource_access_approval_organization_settings_test.go

Lines 95 to 107 in c79e58e

	func testAccAccessApprovalOrganizationSettings_update(context map[string]interface{}) string {
	return Nprintf(`
	resource "google_organization_access_approval_settings" "organization_access_approval" {
	organization_id = "%{org_id}"
	notification_emails = ["[email protected]", "[email protected]"]
	enrolled_services {
	cloud_product = "all"
	enrollment_level = "BLOCK_ALL"
	}
	}
	`, context)
	}

[melinath]

This test has only succeeded once out of the last 120 runs in GA and 4 of the last 120 runs in Beta.