Added support for VPC Service Controls (VPC SC) Secure Data Exchange

[estebanbouza]

Added support for ingress and egress policies in a VPC Service Controls perimeter.
Context: https://cloud.google.com/access-context-manager/docs/reference/rest/v1/accessPolicies.servicePerimeters

If this PR is for Terraform, I acknowledge that I have:

• Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
• Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
• Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
• Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
• Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

accesscontextmanager: added support for ingress and egress policies to `google_access_context_manager_service_perimeter`

[modular-magician]

Hello! I am a robot who works on Magic Modules PRs.

I have detected that you are a community contributor, so your PR will be assigned to someone with a commit-bit on this repo for initial review.

Thanks for your contribution! A human will be with you soon.

@slevenick, please review this PR or find an appropriate assignee.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 5 files changed, 4122 insertions(+), 96 deletions(-))
Terraform Beta: Diff ( 5 files changed, 4122 insertions(+), 96 deletions(-))
TF Conversion: Diff ( 2 files changed, 1561 insertions(+), 25 deletions(-))
Inspec: Diff ( 26 files changed, 1096 insertions(+))

[modular-magician]

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=173138"

[modular-magician]

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccBigtableAppProfile_bigtableAppProfileMulticlusterExample|TestAccBigtableAppProfile_bigtableAppProfileSingleclusterExample|TestAccAccessContextManager|TestAccRedisInstance_redisInstancePrivateServiceExample You can view the result here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=173140"

[modular-magician]

Tests failed during RECORDING mode: TestAccRedisInstance_redisInstancePrivateServiceExample Please fix these to complete your PR

[estebanbouza]

@slevenick I've run tests for the failed one locally, with a blank project within my org. The test passes successfully, looks like the failure could be a transient error. Any suggestion on how to retry the PR's tests here or other way to proceed? Thanks!

➜  magic-modules-dev local-test/test.sh
==> Checking source code against gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google-beta -v -run=TestAccRedisInstance_redisInstancePrivateServiceExample -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google-beta/v3/version.ProviderVersion=acc"
=== RUN   TestAccRedisInstance_redisInstancePrivateServiceExample
=== PAUSE TestAccRedisInstance_redisInstancePrivateServiceExample
=== CONT  TestAccRedisInstance_redisInstancePrivateServiceExample
--- PASS: TestAccRedisInstance_redisInstancePrivateServiceExample (588.88s)
PASS
ok      github.com/hashicorp/terraform-provider-google-beta/v3/google-beta      588.896s
/home/esteban/magic-modules-dev
➜  magic-modules-dev

[slevenick]

We likely need to mark some of the subfields here as Required, or at least mark some as AtLeastOneOf/ExactlyOneOf. Probably Required because this is inside an array which makes the other two options not viable

We try to do this for nested objects because allowing users to specify empty object often causes problems. Can we mark at least one field on every level of the object as Required?

[slevenick]

Why skip test here?

[slevenick]

Well, these are all marked as skip test because they are only for example generation. Can we add a test for this to the handwritten test file so our CI will run them regularly?

I believe it is this file: https://github.com/GoogleCloudPlatform/magic-modules/blob/master/mmv1/third_party/terraform/tests/resource_access_context_manager_service_perimeter_test.go.erb

[slevenick]

@slevenick I've run tests for the failed one locally, with a blank project within my org. The test passes successfully, looks like the failure could be a transient error. Any suggestion on how to retry the PR's tests here or other way to proceed? Thanks!

➜  magic-modules-dev local-test/test.sh
==> Checking source code against gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google-beta -v -run=TestAccRedisInstance_redisInstancePrivateServiceExample -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google-beta/v3/version.ProviderVersion=acc"
=== RUN   TestAccRedisInstance_redisInstancePrivateServiceExample
=== PAUSE TestAccRedisInstance_redisInstancePrivateServiceExample
=== CONT  TestAccRedisInstance_redisInstancePrivateServiceExample
--- PASS: TestAccRedisInstance_redisInstancePrivateServiceExample (588.88s)
PASS
ok      github.com/hashicorp/terraform-provider-google-beta/v3/google-beta      588.896s
/home/esteban/magic-modules-dev
➜  magic-modules-dev

The redis test is flaky, don't worry about that.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 5 files changed, 4148 insertions(+), 102 deletions(-))
Terraform Beta: Diff ( 6 files changed, 4149 insertions(+), 103 deletions(-))
TF Conversion: Diff ( 2 files changed, 1561 insertions(+), 25 deletions(-))
Inspec: Diff ( 26 files changed, 1096 insertions(+))

[modular-magician]

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=174927"

[modular-magician]

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccServiceUsageConsumerQuotaOverride_regionConsumerQuotaOverrideExample You can view the result here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=174929"

[modular-magician]

Tests failed during RECORDING mode: TestAccServiceUsageConsumerQuotaOverride_regionConsumerQuotaOverrideExample Please fix these to complete your PR

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 5 files changed, 4152 insertions(+), 102 deletions(-))
Terraform Beta: Diff ( 5 files changed, 4152 insertions(+), 102 deletions(-))
TF Conversion: Diff ( 2 files changed, 1561 insertions(+), 25 deletions(-))
Inspec: Diff ( 26 files changed, 1096 insertions(+))

[modular-magician]

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=174933"

[estebanbouza]

@slevenick I have added the at_least_one_of constraint to avoid empty arrays/blocks to the following elements:

• ingressPolicies
• ingressPolicies.ingressFrom
• ingressPolicies.ingressTo
• egressPolicies
• egressPolicies.egressFrom
• egressPolicies.egressTo

In regards to testing you mentioned, I had originally submitted a test in the following file:
mmv1/third_party/terraform/tests/resource_access_context_manager_service_perimeter_test.go.erb
Let me know if you have any feedback on it.

Both the test for this feature and the one failing in CI/CD (TestAccServiceUsageConsumerQuotaOverride_regionConsumerQuotaOverrideExample) are passing correctly in my private organization.

[modular-magician]

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccServiceUsageConsumerQuotaOverride_regionConsumerQuotaOverrideExample You can view the result here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=174937"

[modular-magician]

Tests failed during RECORDING mode: TestAccServiceUsageConsumerQuotaOverride_regionConsumerQuotaOverrideExample Please fix these to complete your PR

[slevenick]

I think many of the fields can be marked as required: true in api.yaml. Marking required fields as required helps users correctly configure resources

I'm guessing a little here because I'm not very familiar with this resource, so if they are not actually required then we shouldn't add it. But most of these things seem like they must be filled out. If a field is marked as required it only must be specified if the object it belongs in is specified, so it won't force a resource that doesn't specify ingress_policies or egress_policies to include them

[slevenick]

Unfortunately I don't believe this will work. Because ingress_policies is an array we would have to hard-code the index here, but since it is an array that allows multiple entries we could only ever check the first entry with a hard coded index.

[slevenick]

Is there a reason this would ever be empty or not specified? Can we make this required: true?

[eadred]

The API allows you to specify either identities or identity_type, so I think what's needed is an exactly_one_of. The same goes for egress_policies.egressFrom.identities.

[eadred]

This was how I had the validation properties set on a PR that was trying to do the same thing as this one: 23baeae

[slevenick]

Can we make this required: true also?

[estebanbouza]

I have been reviewing the API and running some tests against it. The API does not seem to have any requirement of required fields or "at least / exactly one of".

For example

• it is possible to create a blank ingressPolicy without an ingressFrom or ingressTo.
• It is possible to create a blank ingressPolicy.ingressFrom without specifying an identityType, identities, allowedIdentity or sources .
• An ingressFrom can be empty, have only sources, or have a sources + identityType
• There are incompatible combinations, but those should be obvious to the user. If the user still goes ahead the error is self explanatory. E.g. can't mix identityType: ANY_IDENTITY and at thiese same time specify users in identities. The error message raised is: [...] Error in IngressFrom: 'identities' field should be empty when identity_type field is not set to IDENTITY_TYPE_UNSPECIFIED.
• The rules seem to apply equally for egress

My recommendation is to remove all the at_least_one_of in this PR. Any other suggestions?

[slevenick]

Yeah, I guess that is the only way to handle this

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 5 files changed, 4122 insertions(+), 96 deletions(-))
Terraform Beta: Diff ( 5 files changed, 4122 insertions(+), 96 deletions(-))
TF Conversion: Diff ( 2 files changed, 1561 insertions(+), 25 deletions(-))
Inspec: Diff ( 26 files changed, 1096 insertions(+))

[modular-magician]

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=177323"

[modular-magician]

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccDataSourceGoogleCloudRunLocations_basic|TestAccDataSourceDNSKeys_noDnsSec|TestAccDataSourceDnsManagedZone_basic|TestAccDataSourceGameServicesGameServerDeploymentRollout_basic|TestAccDataSourceGoogleActiveFolder_default|TestAccDataSourceComputeNetworkEndpointGroup|TestAccDataSourceDNSKeys_basic|TestAccDataSourceCloudIdentityGroupMemberships_basic|TestAccDataSourceCloudIdentityGroups_basic|TestAccDataSourceGoogleActiveFolder_space|TestAccDataSourceGoogleBigqueryDefaultServiceAccount_basic|TestAccDataSourceGoogleAppEngineDefaultServiceAccount_basic|TestAccDataSourceGoogleBillingAccount_byShortName|TestAccDataSourceGoogleBillingAccount_byFullName|TestAccDataSourceGoogleCloudRunService_optionalProject|TestAccDataSourceGoogleActiveFolder_dash|TestAccDataSourceGoogleCloudRunService_basic|TestAccDataSourceGoogleClientOpenIDUserinfo_basic|TestAccDataSourceGoogleCloudFunctionsFunction_basic|TestAccDataSourceComposerEnvironment_basic|TestAccDataSourceComposerImageVersions_basic|TestAccDataSourceGoogleComputeDefaultServiceAccount_basic|TestAccDataSourceComputeImage|TestAccDataSourceComputeBackendBucket_basic|TestAccDataSourceComputeBackendService_basic|TestAccDataSourceComputeAddress|TestAccDataSourceGoogleForwardingRule|TestAccDataSourceComputeImageFilter|TestAccDataSourceComputeGlobalAddress|TestAccDataSourceGoogleComputeInstanceGroup_basic|TestAccDataSourceGoogleComputeInstanceGroup_withNamedPort|TestAccInstanceTemplateDatasource_name|TestAccDataSourceComputeInstanceSerialPort_basic|TestAccInstanceTemplateDatasource_filter_mostRecent|TestAccDataSourceComputeNodeTypes_basic|TestAccComputeRegions_basic|TestAccInstanceTemplateDatasource_filter|TestAccDataSourceGoogleNetwork|TestAccDataSourceComputeInstance_basic|TestAccDataSourceComputeRegionSslCertificate|TestAccDataSourceGoogleComputeInstanceGroup_fromIGM|TestAccDataSourceComputeResourcePolicy|TestAccDataSourceComputeRouter|TestAccComputeZones_basic|TestAccDataSourceComputeSslCertificate|TestAccComputeZones_filter|TestAccDataSourceGoogleSslPolicy|TestAccDataSourceGoogleSubnetwork|TestAccDataSourceGoogleVpnGateway|TestAccContainerEngineVersions_basic|TestAccContainerClusterDatasource_zonal|TestAccContainerClusterDatasource_regional|TestAccContainerEngineVersions_filtered|TestAccDataSourceGoogleFolderOrganizationPolicy_basic|TestAccDataSourceGoogleFolder_byFullName|TestAccDataSourceGoogleFolder_lookupOrganization|TestAccDataSourceGoogleFolder_byShortName|TestAccDataSourceIAMRole|TestAccDataSourceGoogleKmsCryptoKey_basic|TestAccDataKmsSecretCiphertext_basic|TestAccDataSourceGoogleGlobalForwardingRule|TestAccDataSourceGoogleKmsKeyRing_basic|TestAccDataSourceGoogleMonitoringUptimeCheckIps_basic|TestAccDataSourceGoogleKmsCryptoKeyVersion_basic|TestAccDataSourceGoogleOrganization_byShortName|TestAccDataSourceGoogleOrganization_byFullName|TestAccDataSourceGoogleProjects_basic|TestAccDataSourceGoogleProjectOrganizationPolicy_basic|TestAccDatasourceGoogleServiceAccountKey_basic|TestAccDatasourceGoogleServiceAccount_basic|TestAccDataSourceGoogleProject_basic|TestAccDataSourceGoogleStorageProjectServiceAccount_basic|TestAccDataSourceGoogleSQLCaCerts_basic|TestAccDataSourceGoogleStorageTransferProjectServiceAccount_basic|TestAccDataSourceIAMBetaWorkloadIdentityPoolProvider_basic|TestAccDataSourceIAMBetaWorkloadIdentityPool_basic|TestAccDataSourceGoogleMonitoringNotificationChannel_byDisplayName|TestAccDataSourceGoogleMonitoringNotificationChannel_byTypeAndLabel|TestAccDataSourceGoogleMonitoringNotificationChannel_byDisplayNameAndType|TestAccDataSourceGoogleMonitoringNotificationChannel_UserLabel|TestAccDataSourceMonitoringService_AppEngine|TestAccRedisInstanceDatasource_basic|TestAccDataSourceGooglePubsubTopic_optionalProject|TestAccDataSourceGooglePubsubTopic_basic|TestAccRuntimeconfigConfigDatasource_basic|TestAccIapClient_Datasource_basic|TestAccRuntimeconfigVariableDatasource_basic|TestAccDatasourceSecretManagerSecretVersion_latest|TestAccDatasourceSecretManagerSecretVersion_basic|TestAccTPUTensorflowVersions_basic|TestAccDataSourceSqlDatabaseInstance_basic|TestAccDataSourceStorageBucketObjectContent_Basic|TestAccDataSourceGoogleIamTestablePermissions_basic|TestAccApiGatewayGateway_apigatewayGatewayBasicExampleUpdated|TestAccBigQueryDataset_cmek|TestAccBigQueryJob_bigqueryJobCopyExample|TestAccBigQueryJob_bigqueryJobCopyTableReferenceExample|TestAccBigqueryDataTransferConfig|TestAccBigQueryTable_Kms|TestAccBillingBudget_billingBudgetBasicExample You can view the result here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=177326"

[modular-magician]

Tests failed during RECORDING mode: TestAccRuntimeconfigConfigDatasource_basic|TestAccRuntimeconfigVariableDatasource_basic|TestAccBigQueryJob_bigqueryJobCopyTableReferenceExample|TestAccApiGatewayGateway_apigatewayGatewayBasicExampleUpdated Please fix these to complete your PR

[estebanbouza]

@slevenick mind taking a look at the status of this PR? I've reverted the at least one of requirements according to the API definition.

[slevenick]

Thanks for the addition!