Failing test(s): Folder not found issues

[melinath]

Failure rate: ???

Impacted tests:

• TestAccLoggingFolderSink_described
• TestAccLoggingFolderSink_disabled
• TestAccLoggingFolderSink_update
• TestAccLoggingFolderSink_removeOptionals
• TestAccLoggingFolderSink_folderAcceptsFullFolderPath
• TestAccLoggingFolderSink_updateBigquerySink
• TestAccLoggingFolderSink_heredoc
• TestAccLoggingFolderSink_basic
• TestAccFolderOrganizationPolicy_list_update
• TestAccFolderOrganizationPolicy_list_allowSome
• TestAccFolderOrganizationPolicy_list_denySome
• TestAccFolderOrganizationPolicy_restore_defaultTrue
• TestAccFolderOrganizationPolicy_boolean
• TestAccFolderIamPolicy_auditConfigs
• TestAccFolderIamMember_basic
• TestAccFolderIamMember_multiple
• TestAccFolderIamMember_remove
• TestAccFolderIamBinding_basic
• TestAccFolderIamBinding_update
• TestAccFolderIamBinding_remove
• TestAccFolderIamBinding_multiple
• TestAccFolderIamBinding_multipleAtOnce
• TestAccFolder_rename
• TestAccFolder_moveParent
• TestAccComputeFirewallPolicy_update
• TestAccComputeFirewallPolicyRule_multipleRules
• TestAccFolderIamAuditConfig_basic
• TestAccFolderIamAuditConfig_updateNoExemptMembers
• TestAccFolderIamAuditConfig_remove
• TestAccFolderIamAuditConfig_addFirstExemptMember
• TestAccFolderIamAuditConfig_removeLastExemptMember
• TestAccFolderIamAuditConfig_update
• TestAccFolderIamAuditConfig_multiple
• TestAccFolderIamAuditConfig_multipleAtOnce
• TestAccDataSourceGoogleFolders_basic
• TestAccDataSourceGoogleFolder_lookupOrganization
• TestAccDataSourceGoogleActiveFolder_dash
• TestAccDataSourceGoogleActiveFolder_space
• TestAccDataSourceGoogleActiveFolder_default

Nightly builds:

• https://ci-oss.hashicorp.engineering/buildConfiguration/GoogleCloud_ProviderGoogleCloudGoogleProject/254727?buildTab=tests&expandedTest=6317277238467567837

PR builds:

• https://ci-oss.hashicorp.engineering/buildConfiguration/GoogleCloudBeta_ProviderGoogleCloudBetaMmUpstreamVcr/209386?buildTab=tests&expandedTest=-7514914581837289937

Message:

Error: Error creating folder 'tf-test-folder-gkug6e9og8' in 'organizations/*******': Error waiting for creating folder: Error code 9, message: Folder reservation failed for parent [organizations/*******], folder [] due to constraint: The folder operation violates fanout constraints.

Fanout contraints are documented here: https://cloud.google.com/resource-manager/reference/rest/v3/folders/create

The addition of the folder must not cause the total number of folders under its parent to exceed 300.

This probably means something is going wrong with sweeping test folders?

[nat-henderson]

Also

• TestAccLoggingFolderExclusion
• TestAccFolderOrganizationPolicy_*
• TestAccFolderIamAuditConfig_*

and a whole ton of others. i'll just go delete em myself real quick.

[nat-henderson]

Hm, easier said than done. I got a few dozen with

for fid in $(gcloud resource-manager folders list --organization $ORG_ID | grep "^tf-test-" | awk '{print $3}'); do gcloud resource-manager folders delete $fid; done; 

but that's not enough, I mostly got "folder not empty". Which would explain why we're having sweeper problems! That's where I'd look next.

[rileykarson]

Marking as large because we probably need a bigger solution than we've currently got (and a manual cleanup will probably fix it for a while, but the problem will recur)

[rileykarson]

Note: fanout constraints are resolved, but we're seeing issues finding folders in tests

provider_test.go:309: Step 1/2 error: Check failed: Check 1/1 error: Failed to retrieve IAM Policy for folder "tf-test-7injbpe84c": expected exactly 1 folder, found 0

[ScottSuarez]

There looks to be some sort of latency between a folder being created and a folder being searchable on the API.

I've created b/228499462

[melinath]

based on the response - It looks like getFolderIamPolicyByParentAndDisplayName uses Folders.Search, which is eventually consistent. Switching to Folders.List might work better since it's strongly consistent. At least for tests that rely on that function.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.