add retries for eventual consistency with storage buckets

[megan07]

Fixes hashicorp/terraform-provider-google#10207

If this PR is for Terraform, I acknowledge that I have:

• Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
• Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
• Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
• Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
• Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

storage: fixed a bug to better handle eventual consistency among `google_storage_bucket` resources.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 32 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 2 files changed, 33 insertions(+), 2 deletions(-))

[modular-magician]

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccDataSourceGoogleCloudFunctionsFunction_basic|TestAccDataSourceComputeBackendBucket_basic|TestAccDatasourceGoogleServiceNetworkingPeeredDnsDomain_basic|TestAccDataSourceGoogleStorageBucket_basic|TestAccDataSourceStorageBucketObjectContent_Basic|TestAccCloudFunctionsCloudFunctionIamMemberGenerated|TestAccCloudFunctionsCloudFunctionIamPolicyGenerated|TestAccCloudFunctionsCloudFunctionIamBindingGenerated|TestAccIapAppEngineVersionIamMemberGenerated|TestAccIapAppEngineVersionIamPolicyGenerated_withCondition|TestAccIapAppEngineVersionIamBindingGenerated_withCondition|TestAccIapAppEngineVersionIamPolicyGenerated|TestAccIapAppEngineVersionIamMemberGenerated_withCondition|TestAccIapAppEngineVersionIamBindingGenerated|TestAccIapAppEngineServiceIamMemberGenerated|TestAccIapAppEngineServiceIamPolicyGenerated_withCondition|TestAccIapAppEngineServiceIamBindingGenerated_withCondition|TestAccIapAppEngineServiceIamMemberGenerated_withCondition|TestAccIapAppEngineServiceIamBindingGenerated|TestAccIapAppEngineServiceIamPolicyGenerated|TestAccStorageBucketIamMemberGenerated|TestAccStorageBucketIamPolicyGenerated_withCondition|TestAccStorageBucketIamBindingGenerated_withCondition|TestAccStorageBucketIamMemberGenerated_withCondition|TestAccStorageBucketIamPolicyGenerated|TestAccStorageBucketIamBindingGenerated|TestAccAppEngineApplicationUrlDispatchRules_appEngineApplicationUrlDispatchRulesBasicExample|TestAccAppEngineServiceNetworkSettings_appEngineServiceNetworkSettingsExample|TestAccAppEngineServiceSplitTraffic_appEngineServiceSplitTrafficExample|TestAccAppEngineStandardAppVersion_appEngineStandardAppVersionExample|TestAccAppEngineServiceNetworkSettings_update|TestAccAppEngineFlexibleAppVersion_appEngineFlexibleAppVersionExample|TestAccAppEngineFlexibleAppVersion_update|TestAccAppEngineStandardAppVersion_update|TestAccBigQueryJob_bigqueryJobExtractExample|TestAccBigQueryJob_bigqueryJobExtractTableReferenceExample|TestAccBigQueryTable_HivePartitioning|TestAccBigQueryTable_HivePartitioningCustomSchema|TestAccBigQueryExternalDataTable_CSV|TestAccCloudFunctionsFunction_basic|TestAccCloudFunctionsFunction_pubsub|TestAccCloudFunctionsFunction_firestore|TestAccCloudFunctionsFunction_serviceAccountEmail|TestAccCloudFunctionsFunction_update|TestAccCloudFunctionsFunction_bucket|TestAccCloudFunctionsFunction_vpcConnector|TestAccComputeBackendBucket_backendBucketFullExample|TestAccComputeBackendBucket_backendBucketBasicExample|TestAccComputeBackendBucketSignedUrlKey_basic|TestAccComputeBackendBucket_withCdnPolicy|TestAccComputeBackendBucket_basicModified|TestAccComputeRegionNetworkEndpointGroup_regionNetworkEndpointGroupFunctionsExample|TestAccComputeRegionNetworkEndpointGroup_regionNetworkEndpointGroupAppengineExample|TestAccComputeUrlMap_urlMapBasicExample|TestAccDataLossPreventionStoredInfoType_dlpStoredInfoTypeLargeCustomDictionaryExample|TestAccDataprocCluster_withInitAction|TestAccDataprocCluster_withStagingBucket|TestAccDataprocCluster_withTempBucket|TestAccDataprocCluster_withKerberos|TestAccFirebaseWebApp_firebaseWebAppBasicExample|TestAccLoggingBillingAccountSink_basic|TestAccLoggingBillingAccountSink_update|TestAccLoggingBillingAccountSink_described|TestAccLoggingBillingAccountSink_disabled|TestAccLoggingBillingAccountSink_heredoc|TestAccLoggingFolderSink_basic|TestAccLoggingFolderSink_described|TestAccLoggingFolderSink_disabled|TestAccLoggingFolderSink_folderAcceptsFullFolderPath|TestAccLoggingFolderSink_update|TestAccLoggingFolderSink_heredoc|TestAccLoggingFolderSink_removeOptionals|TestAccLoggingOrganizationSink_basic|TestAccLoggingOrganizationSink_disabled|TestAccLoggingOrganizationSink_described|TestAccLoggingOrganizationSink_update|TestAccLoggingOrganizationSink_heredoc|TestAccLoggingProjectSink_basic|TestAccLoggingProjectSink_described|TestAccLoggingProjectSink_disabled|TestAccLoggingProjectSink_heredoc|TestAccLoggingProjectSink_updatePreservesUniqueWriter|TestAccLoggingProjectSink_disabled_update|TestAccNetworkServicesEdgeCacheService_networkServicesEdgeCacheServiceBasicExample|TestAccNetworkServicesEdgeCacheService_networkServicesEdgeCacheServiceAdvancedExample|TestAccNetworkServicesEdgeCacheService_updateAndImport|TestAccPubsubLiteReservation_pubsubLiteReservationBasicExample|TestAccPubsubLiteTopic_pubsubLiteTopicBasicExample|TestAccStorageBucketAccessControl_storageBucketAccessControlPublicBucketExample|TestAccStorageBucketAcl_predefined|TestAccStorageBucketAcl_basic|TestAccStorageBucketAcl_unordered|TestAccStorageBucketAcl_RemoveOwner|TestAccStorageObject_basic|TestAccStorageBucketAccessControl_update|TestAccStorageObject_content|TestAccStorageObject_withContentCharacteristics|TestAccStorageObject_dynamicContent|TestAccStorageObject_cacheControl|TestAccStorageObject_storageClass|TestAccStorageBucketIamPolicy|TestAccStorageObject_recreate|TestAccStorageObject_metadata|TestAccStorageBucketAcl_downgrade|TestAccStorageBucketAcl_upgrade|TestAccStorageBucketAcl_upgradeSingleUser|TestAccStorageObjectKms|TestAccStorageObject_customerEncryption|TestAccStorageBucket_requesterPays|TestAccStorageBucket_basic|TestAccStorageBucket_lowercaseLocation|TestAccStorageBucket_lifecycleRuleStateLive|TestAccStorageBucket_customAttributes|TestAccStorageObject_holds|TestAccStorageBucket_update|TestAccStorageBucket_lifecycleRuleStateArchived|TestAccStorageBucket_forceDestroy|TestAccStorageBucket_versioning|TestAccStorageBucket_update_requesterPays|TestAccStorageBucket_defaultEventBasedHold|TestAccStorageBucket_cors|TestAccStorageBucket_forceDestroyObjectDeleteError|TestAccStorageBucket_forceDestroyWithVersioning|TestAccStorageDefaultObjectAccessControl_storageDefaultObjectAccessControlPublicExample|TestAccStorageBucket_logging|TestAccStorageBucket_retentionPolicy|TestAccStorageDefaultObjectAcl_basic|TestAccStorageBucket_bucketPolicyOnly|TestAccStorageBucket_uniformBucketAccessOnly|TestAccStorageDefaultObjectAcl_upgrade|TestAccStorageDefaultObjectAcl_downgrade|TestAccStorageBucket_lifecycleRuleStateAny|TestAccStorageDefaultObjectAcl_unordered|TestAccStorageDefaultObjectAcl_noRoleEntity|TestAccStorageBucket_website|TestAccStorageBucket_labels|TestAccStorageDefaultObjectAccessControl_update|TestAccStorageObjectAccessControl_storageObjectAccessControlPublicObjectExample|TestAccStorageNotification_withEventsAndAttributes|TestAccStorageNotification_basic|TestAccStorageObjectAcl_basic|TestAccStorageObjectAcl_predefined|TestAccStorageObjectAcl_unordered|TestAccStorageObjectAccessControl_update|TestAccStorageObjectAccessControl_updateWithSlashes|TestAccStorageTransferJob_omitScheduleEndDate|TestAccStorageObjectAcl_predefinedToExplicit|TestAccStorageObjectAcl_explicitToPredefined|TestAccStorageTransferJob_basic|TestAccStorageObjectAcl_upgrade|TestAccStorageObjectAcl_downgrade|TestAccComputeResourceUsageExportBucket You can view the result here: https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=209386

[megan07]

I think we're good? All the LoggingFolderSink tests passed locally for me, these seem to be issues with the VCR tests I think?

[modular-magician]

Tests failed during RECORDING mode: TestAccAppEngineServiceNetworkSettings_appEngineServiceNetworkSettingsExample|TestAccLoggingFolderSink_described|TestAccLoggingFolderSink_disabled|TestAccLoggingFolderSink_removeOptionals|TestAccLoggingFolderSink_folderAcceptsFullFolderPath|TestAccLoggingFolderSink_updateBigquerySink|TestAccLoggingFolderSink_heredoc|TestAccFirebaseWebApp_firebaseWebAppBasicExample Please fix these to complete your PR

[melinath]

It looks like the failures are due to fanout constraints, which are described here:

The addition of the folder must not cause the total number of folders under its parent to exceed 300.

That seems like we might be missing a sweeper? But I think we can probably handle that as a separate PR.

[melinath]

hashicorp/terraform-provider-google#10286 tracks the fanout constraint issue.