Add `billing_project` configuration to the provider #3886

upodroid · 2020-08-19T15:19:11Z

Fixes: hashicorp/terraform-provider-google#6747

This PR introduces a new field in the provider billing_project and amends ACM Products to support User ADCs.

I took a few ideas from how the cloud_feed

I'm stuck on testing it properly.

If this PR is for Terraform, I acknowledge that I have:

Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

provider: added a new field `billing_project` to the provider that's associated as a billing/quota project with most requests when `user_project_override` true

Fork Update

modular-magician · 2020-08-19T15:20:34Z

Hello! I am a robot who works on Magic Modules PRs.

I have detected that you are a community contributor, so your PR will be assigned to someone with a commit-bit on this repo for initial review.

Thanks for your contribution! A human will be with you soon.

@rileykarson, please review this PR or find an appropriate assignee.

upodroid · 2020-08-19T15:21:27Z

The env variable is set in both provider.go and provider-test.go

REDACTED  MCW0CDP3YY  ~  go  …  github.com  hashicorp  terraform-provider-google-beta   master  41⬇  23✎  $    make testacc TEST=./google-beta TESTARGS='-run=TestAccAccessContextManager'
==> Checking source code against gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google-beta -v -run=TestAccAccessContextManager -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google-beta/version.ProviderVersion=acc"
=== RUN   TestAccAccessContextManager
=== RUN   TestAccAccessContextManager/access_level_full
    TestAccAccessContextManager/access_level_full: testing.go:674: Step 0 error: errors during apply:
        
        Error: billing_project: required field is not set
        
          on /var/folders/b1/dthn83bs2qbcrg38qszm22440000gn/T/tf-test046903338/main.tf line 2:
          (source code not available)
        
        
=== RUN   TestAccAccessContextManager/access_level_custom
    TestAccAccessContextManager/access_level_custom: testing.go:674: Step 0 error: errors during apply:
        
        Error: billing_project: required field is not set
        
          on /var/folders/b1/dthn83bs2qbcrg38qszm22440000gn/T/tf-test231287532/main.tf line 2:
          (source code not available)
        
        
=== RUN   TestAccAccessContextManager/access_policy
    TestAccAccessContextManager/access_policy: testing.go:674: Step 0 error: errors during apply:
        
        Error: billing_project: required field is not set
        
          on /var/folders/b1/dthn83bs2qbcrg38qszm22440000gn/T/tf-test844553982/main.tf line 2:
          (source code not available)
        
        
=== RUN   TestAccAccessContextManager/service_perimeter
    TestAccAccessContextManager/service_perimeter: testing.go:674: Step 0 error: errors during apply:
        
        Error: billing_project: required field is not set
        
          on /var/folders/b1/dthn83bs2qbcrg38qszm22440000gn/T/tf-test116288736/main.tf line 2:
          (source code not available)
        
        
=== RUN   TestAccAccessContextManager/service_perimeter_update
    TestAccAccessContextManager/service_perimeter_update: testing.go:674: Step 0 error: errors during apply:
        
        Error: billing_project: required field is not set
        
          on /var/folders/b1/dthn83bs2qbcrg38qszm22440000gn/T/tf-test462741266/main.tf line 2:
          (source code not available)
        
        
=== RUN   TestAccAccessContextManager/service_perimeter_resource
    TestAccAccessContextManager/service_perimeter_resource: bootstrap_utils_test.go:334: Bootstrapping failed. Unable to load test config: unable to parse credentials from 'true': json: cannot unmarshal bool into Go value of type google.credentialsFile
=== RUN   TestAccAccessContextManager/access_level
    TestAccAccessContextManager/access_level: testing.go:674: Step 0 error: errors during apply:
        
        Error: billing_project: required field is not set
        
          on /var/folders/b1/dthn83bs2qbcrg38qszm22440000gn/T/tf-test825725972/main.tf line 2:
          (source code not available)
        
        
--- FAIL: TestAccAccessContextManager (1.39s)
    --- FAIL: TestAccAccessContextManager/access_level_full (0.25s)
    --- FAIL: TestAccAccessContextManager/access_level_custom (0.25s)
    --- FAIL: TestAccAccessContextManager/access_policy (0.22s)
    --- FAIL: TestAccAccessContextManager/service_perimeter (0.22s)
    --- FAIL: TestAccAccessContextManager/service_perimeter_update (0.22s)
    --- FAIL: TestAccAccessContextManager/service_perimeter_resource (0.00s)
    --- FAIL: TestAccAccessContextManager/access_level (0.22s)
FAIL
FAIL    github.com/hashicorp/terraform-provider-google-beta/google-beta 2.473s
FAIL
make: *** [testacc] Error 1
 maha4472  MCW0CDP3YY  ~  go  …  github.com  hashicorp  terraform-provider-google-beta   master  41⬇  23✎  USAGE  $    env | grep GOOGLE
GOOGLE_REGION=us-central1
GOOGLE_BILLING_PROJECT=REDACTED
GOOGLE_ORG=684510004702
GOOGLE_ZONE=us-central1-a
GOOGLE_PROJECT=REDACTED
GOOGLE_USE_DEFAULT_CREDENTIALS=true
 REDACTED  MCW0CDP3YY  ~  go  …  github.com  hashicorp  terraform-provider-google-beta   master  41⬇  23✎  $ 

modular-magician · 2020-08-19T15:27:12Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 18 files changed, 458 insertions(+), 15 deletions(-))
Terraform Beta: Diff ( 18 files changed, 458 insertions(+), 15 deletions(-))
TF Conversion: Diff ( 3 files changed, 19 insertions(+))

rileykarson

This seems like a reasonable approach! Right now it's presented as an alternative sourcing mechanism for supports_indirect_user_project_override, but I think we can use the provider-level billing_project resource across all resources regardless of whether they support a direct/indirect project or no project (like ACM).

I'll provide some more specific comments inline, but broadly:

This should take priority over direct/indirect projects, since it's a dedicated field. If a user specified it, we should prefer it.
All MM-generated resources should be able to use it. We don't need to lock it behind supports_indirect_user_project_override.

products/accesscontextmanager/terraform.yaml

templates/terraform/resource.erb

third_party/terraform/utils/utils.go.erb

third_party/terraform/utils/provider.go.erb

templates/terraform/resource.erb

third_party/terraform/utils/field_helpers.go

modular-magician · 2020-08-20T00:31:18Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 160 files changed, 4622 insertions(+), 665 deletions(-))
Terraform Beta: Diff ( 191 files changed, 5530 insertions(+), 784 deletions(-))
TF Conversion: Diff ( 3 files changed, 19 insertions(+))

rileykarson

Broadly looks good in resource.erb! Left a couple more minor comments.

templates/terraform/resource.erb

rileykarson

Oh, whoops, I didn't reply to the VCR comment. Unfortunately we're unable to run those locally. Since they contain secrets, we're also unable to run them on community contributions or to share logs. The way you ran the tests in #3886 (comment) should be sufficient.

You're passing our Cloud Build CI right now (this one doesn't run integration tests, though) other than some of the testing-related code that isn't being used anywhere triggering a linter error. We can get to that on my next review pass- I need to double check with a teammate how well our CI can test this stuff. It generally runs as a service account, but there may be a way to run an interactive account too that I'm not aware of.

upodroid · 2020-08-20T18:24:58Z

I was looking for the debug logs but I figured how to get them by prepending TF_LOG=debug before the make command.

It is not setting the header. I'll investigate and update the PR if I can't figure it out.

upodroid · 2020-08-20T19:50:59Z

I think we are good to go.

 REDACTED  MCW0CDP3YY  ~  go  …  github.com  hashicorp  terraform-provider-google-beta   master  49⬇  188✎  $    make testacc TEST=./google-beta TESTARGS='-run=TestAccAccessContextManager'
==> Checking source code against gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google-beta -v -run=TestAccAccessContextManager -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google-beta/version.ProviderVersion=acc"
=== RUN   TestAccAccessContextManager
=== RUN   TestAccAccessContextManager/access_level
=== RUN   TestAccAccessContextManager/access_level_full
=== RUN   TestAccAccessContextManager/access_level_custom
=== RUN   TestAccAccessContextManager/access_policy
=== RUN   TestAccAccessContextManager/service_perimeter
=== RUN   TestAccAccessContextManager/service_perimeter_update
=== RUN   TestAccAccessContextManager/service_perimeter_resource
    TestAccAccessContextManager/service_perimeter_resource: provider_test.go:911: VCR enabled, skipping test: TestAccAccessContextManager/service_perimeter_resource
--- PASS: TestAccAccessContextManager (170.16s)
    --- PASS: TestAccAccessContextManager/access_level (30.09s)
    --- PASS: TestAccAccessContextManager/access_level_full (19.22s)
    --- PASS: TestAccAccessContextManager/access_level_custom (23.58s)
    --- PASS: TestAccAccessContextManager/access_policy (16.08s)
    --- PASS: TestAccAccessContextManager/service_perimeter (23.51s)
    --- PASS: TestAccAccessContextManager/service_perimeter_update (57.68s)
    --- SKIP: TestAccAccessContextManager/service_perimeter_resource (0.00s)
PASS
ok      github.com/hashicorp/terraform-provider-google-beta/google-beta 171.102s
 REDACTED  MCW0CDP3YY  ~  go  …  github.com  hashicorp  terraform-provider-google-beta   master  49⬇  188✎  USAGE  $    make testacc TEST=./google-beta TESTARGS='-run=TestAccCloudAssetFolderFeed_cloudAssetFolderFeedExample'
==> Checking source code against gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google-beta -v -run=TestAccCloudAssetFolderFeed_cloudAssetFolderFeedExample -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google-beta/version.ProviderVersion=acc"
=== RUN   TestAccCloudAssetFolderFeed_cloudAssetFolderFeedExample
=== PAUSE TestAccCloudAssetFolderFeed_cloudAssetFolderFeedExample
=== CONT  TestAccCloudAssetFolderFeed_cloudAssetFolderFeedExample
--- PASS: TestAccCloudAssetFolderFeed_cloudAssetFolderFeedExample (30.85s)
PASS
ok      github.com/hashicorp/terraform-provider-google-beta/google-beta 31.807s

modular-magician · 2020-08-20T19:58:06Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 161 files changed, 4634 insertions(+), 665 deletions(-))
Terraform Beta: Diff ( 192 files changed, 5542 insertions(+), 784 deletions(-))
TF Conversion: Diff ( 3 files changed, 19 insertions(+))

modular-magician · 2020-08-20T21:38:51Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 161 files changed, 3751 insertions(+), 665 deletions(-))
Terraform Beta: Diff ( 192 files changed, 4460 insertions(+), 784 deletions(-))
TF Conversion: Diff ( 3 files changed, 19 insertions(+))

templates/terraform/resource.erb

api/resource.rb

third_party/terraform/website/docs/guides/provider_reference.html.markdown

third_party/terraform/utils/provider_test.go.erb

modular-magician · 2020-08-21T20:37:44Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 160 files changed, 3872 insertions(+), 673 deletions(-))
Terraform Beta: Diff ( 191 files changed, 4619 insertions(+), 792 deletions(-))
TF Conversion: Diff ( 3 files changed, 18 insertions(+))

modular-magician · 2020-08-21T20:40:49Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 160 files changed, 3868 insertions(+), 666 deletions(-))
Terraform Beta: Diff ( 191 files changed, 4615 insertions(+), 785 deletions(-))
TF Conversion: Diff ( 3 files changed, 18 insertions(+))

modular-magician · 2020-08-24T23:24:41Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 164 files changed, 3892 insertions(+), 666 deletions(-))
Terraform Beta: Diff ( 195 files changed, 4639 insertions(+), 785 deletions(-))
TF Conversion: Diff ( 3 files changed, 18 insertions(+))

rileykarson

Sorry for the delay in getting back, I had a busy end of last week! Just one last thing about how we're writing the billingProject value, I think.

templates/terraform/resource.erb

products/accesscontextmanager/terraform.yaml

upodroid · 2020-08-25T19:46:12Z

Lint and acceptance tests are passing 😃

modular-magician · 2020-08-25T19:51:30Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 164 files changed, 5670 insertions(+), 663 deletions(-))
Terraform Beta: Diff ( 195 files changed, 6775 insertions(+), 782 deletions(-))
TF Conversion: Diff ( 3 files changed, 18 insertions(+))

rileykarson

LGTM. Thanks for your collaboration when making changes to how this PR ended up- the billing_project field works consistently across the majority of the provider's resources, and simplified the custom logic in google_cloud_asset_feed!

upodroid · 2020-08-26T19:51:33Z

Yeah, we solved it for all future APIs that don't play nice with User ADCs

upodroid added 3 commits August 3, 2020 12:29

Merge pull request #1 from GoogleCloudPlatform/master

71c3aeb

Fork Update

Merge branch 'master' of github.com:GoogleCloudPlatform/magic-modules

e1a9488

add billing_project parameter to provider

0db07fb

google-cla bot added the cla: yes label Aug 19, 2020

modular-magician requested a review from rileykarson August 19, 2020 15:20

rileykarson reviewed Aug 19, 2020

View reviewed changes

rework the billing project order

e4be0a3

rileykarson reviewed Aug 20, 2020

View reviewed changes

templates/terraform/resource.erb Outdated Show resolved Hide resolved

templates/terraform/resource.erb Outdated Show resolved Hide resolved

rileykarson reviewed Aug 20, 2020

View reviewed changes

add new field to config

c110754

introduce product level override for header

3173d6e

rileykarson reviewed Aug 21, 2020

View reviewed changes

upodroid added 2 commits August 21, 2020 21:29

reorder billing project override

55512f8

update docs

b7902c7

add warning note on autogen documentation

61ade65

rileykarson reviewed Aug 25, 2020

View reviewed changes

templates/terraform/resource.erb Outdated Show resolved Hide resolved

rileykarson reviewed Aug 25, 2020

View reviewed changes

products/accesscontextmanager/terraform.yaml Outdated Show resolved Hide resolved

update docs and reorder billing project assignment

85f117f

rileykarson approved these changes Aug 26, 2020

View reviewed changes

rileykarson changed the title ~~Enable User Project Header for Access Context Manager~~ Add billing_project configuration to the provider Aug 26, 2020

rileykarson merged commit 50d373e into GoogleCloudPlatform:master Aug 26, 2020

upodroid mentioned this pull request Oct 12, 2020

Cloud Identity Group Resources don't work with Application Default Credentials hashicorp/terraform-provider-google#7477

Closed

upodroid mentioned this pull request May 20, 2021

DCL Clients don't set X-Goog-User-Project header in API Calls hashicorp/terraform-provider-google#9204

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `billing_project` configuration to the provider #3886

Add `billing_project` configuration to the provider #3886

upodroid commented Aug 19, 2020 •

edited by rileykarson

Loading

modular-magician commented Aug 19, 2020

upodroid commented Aug 19, 2020

modular-magician commented Aug 19, 2020

rileykarson left a comment

modular-magician commented Aug 20, 2020

rileykarson left a comment •

edited

Loading

rileykarson left a comment •

edited

Loading

upodroid commented Aug 20, 2020

upodroid commented Aug 20, 2020

modular-magician commented Aug 20, 2020

modular-magician commented Aug 20, 2020

modular-magician commented Aug 21, 2020

modular-magician commented Aug 21, 2020

modular-magician commented Aug 24, 2020

rileykarson left a comment

upodroid commented Aug 25, 2020

modular-magician commented Aug 25, 2020

rileykarson left a comment

upodroid commented Aug 26, 2020

Add billing_project configuration to the provider #3886

Add billing_project configuration to the provider #3886

Conversation

upodroid commented Aug 19, 2020 • edited by rileykarson Loading

modular-magician commented Aug 19, 2020

upodroid commented Aug 19, 2020

modular-magician commented Aug 19, 2020

Diff report:

rileykarson left a comment

Choose a reason for hiding this comment

modular-magician commented Aug 20, 2020

Diff report:

rileykarson left a comment • edited Loading

Choose a reason for hiding this comment

rileykarson left a comment • edited Loading

Choose a reason for hiding this comment

upodroid commented Aug 20, 2020

upodroid commented Aug 20, 2020

modular-magician commented Aug 20, 2020

Diff report:

modular-magician commented Aug 20, 2020

Diff report:

modular-magician commented Aug 21, 2020

Diff report:

modular-magician commented Aug 21, 2020

Diff report:

modular-magician commented Aug 24, 2020

Diff report:

rileykarson left a comment

Choose a reason for hiding this comment

upodroid commented Aug 25, 2020

modular-magician commented Aug 25, 2020

Diff report:

rileykarson left a comment

Choose a reason for hiding this comment

upodroid commented Aug 26, 2020

Add `billing_project` configuration to the provider #3886

Add `billing_project` configuration to the provider #3886

upodroid commented Aug 19, 2020 •

edited by rileykarson

Loading

rileykarson left a comment •

edited

Loading

rileykarson left a comment •

edited

Loading