[Terraform] autogenerate Pubsub Topics/Subscriptions (+labels)

[drebes]

---

[all]

Adds labels to Pubsub Topics & Subscriptions.

[terraform]

Autogenerate Pubsub Topics & Subscriptions, including labels support. Fixes hashicorp/terraform-provider-google#1500

[terraform-beta]

[ansible]

[inspec]

[modular-magician]

Hello! I am a robot who works on Magic Modules PRs.

I have detected that you are a community contributor, so your PR will be assigned to someone with a commit-bit on this repo for initial review. They will authorize it to run through our CI pipeline, which will generate downstream PRs.

Thanks for your contribution! A human will be with you soon.

[rambleraptor]

Hello! Thanks so much for submitting this PR. Sorry we didn't notice it.

@rileykarson is going to help you out. If you could rebase the PR, we'll run it through our CI system and get this review going + merged as quickly as possible.

[drebes]

Rebased.

[rileykarson]

Thanks! Running the Magician now. At first glance the change seems good, I'll be able to do a full review with downstreams generated.

[rileykarson]

Seeing this error, I think that the variable was removed recently so you can likely blow it away:

RuntimeError: Extraneous variable 'exports' in TopicApi::Resource

edit: yep, see https://github.com/GoogleCloudPlatform/magic-modules/pull/1345/files#diff-293147022553799829844cb248360ba3L30

[drebes]

Sorry, should be good now.

[rileykarson]

Ah, Ansible requires fields to be separately versioned. You'll need to add versions for the field similar to this:

magic-modules/products/storage/ansible.yaml

Line 42 in 5c81583

version_added: '2.7'

ERROR: Found 2 validate-modules issue(s) which need to be resolved:
ERROR: lib/ansible/modules/cloud/google/gcp_pubsub_subscription.py:0:0: E309 version_added for new option (labels) should be 2.8. Currently 0.0
ERROR: lib/ansible/modules/cloud/google/gcp_pubsub_topic.py:0:0: E309 version_added for new option (labels) should be 2.8. Currently 0.0

[drebes]

Sorry, missed that. Should be good now.

[modular-magician]

I am a robot that works on MagicModules PRs!

I built this PR into one or more PRs on other repositories, and when those are closed, this PR will also be merged and closed.
depends: hashicorp/terraform-provider-google-beta#432
depends: hashicorp/terraform-provider-google#3043
depends: modular-magician/ansible#191
depends: modular-magician/inspec-gcp#109

[rileykarson]

There we go! Reviewing Terraform now.

@rambleraptor @slevenick can you review your downstreams?

Edit: ah @rambleraptor beat me to it inside the downstream PR

[drebes]

Actually, the PATCH from gcloud works, and contains the modified fields inside the subscription object:

==== request start ====
uri: https://pubsub.googleapis.com/v1/projects/drebes-playground-alpha-5ae4/subscriptions/a-sub?alt=json
method: PATCH
== headers start ==
Authorization: --- Token Redacted ---
accept: application/json
accept-encoding: gzip, deflate
content-length: 149
content-type: application/json
user-agent: google-cloud-sdk x_Tw5K8nnjoRAqULM9PFAC2b gcloud/234.0.0 command/gcloud.pubsub.subscriptions.update invocation-id/1547efab192a4ca4a09cba70a36e36d0 environment/None environment-version/None interactive/True from-script/False python/2.7.15 (Macintosh; Intel Mac OS X 18.2.0)
== headers end ==
== body start ==
{"subscription": {"ackDeadlineSeconds": 30, "name": "projects/drebes-playground-alpha-5ae4/subscriptions/a-sub"}, "updateMask": "ackDeadlineSeconds"}
== body end ==
==== request end ====
---- response start ----
-- headers start --
-content-encoding: gzip
alt-svc: quic=":443"; ma=2592000; v="44,43,39"
cache-control: private
content-length: 339
content-type: application/json; charset=UTF-8
date: Sun, 17 Feb 2019 19:19:48 GMT
server: ESF
status: 200
transfer-encoding: chunked
vary: Origin, X-Origin, Referer
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
-- headers end --
-- body start --
{
  "name": "projects/drebes-playground-alpha-5ae4/subscriptions/a-sub",
  "topic": "projects/drebes-playground-alpha-5ae4/topics/a-topic",
  "pushConfig": {
    "attributes": {
      "x-goog-version": "v1"
    }
  },
  "ackDeadlineSeconds": 30,
  "messageRetentionDuration": "604800s",
  "expirationPolicy": {
    "ttl": "2678400s"
  }
}

-- body end --
total round trip time (request+response): 1.489 secs
---- response end ----

How can I force it to update the modified fields inside the subscription object (and updateMask as field as well, not parameter)?

[drebes]

I added an update_encoder and now it all looks fine. 👍

[rileykarson]

Yep, an update encoder is the right tool for a case like that 👍

Generating downstreams and making another pass once they're done.

[modular-magician]

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, d99ef9b.

Pull request statuses

New Pull Requests

I built this PR into one or more new PRs on other repositories, and when those are closed, this PR will also be merged and closed.
depends: hashicorp/terraform-provider-google-beta#450
depends: hashicorp/terraform-provider-google#3080
depends: modular-magician/ansible#193
depends: modular-magician/inspec-gcp#111

[rileykarson]

Looks like a Magician update changed the branch we PR off of, these new downstreams will supersede the old ones.

[rileykarson]

Since the original downstreams are still getting updated, I commented on hashicorp/terraform-provider-google-beta#432 again.

While adding resource/product specific reference (self_link/name) validation is a good idea eventually, I don't think that handwriting it is the right balance of maintenance vs benefit. We can continue to reuse generic functions for handling short names / partial URIs / reference links & keep consistent with most existing resources; that'll make an eventual migration easier.

My apologies again for the fiddlyness of name-style references in MM, especially when converting handwritten resources. I'm passively thinking about how to improve the situation (https://github.com/GoogleCloudPlatform/magic-modules/issues/937), the problem is that many APIs have adopted slightly different patterns vs the relative consistency of self links in the past.

[drebes]

I've pushed my changes. If you can regenerate the downstream PRs and give it a new look...

[modular-magician]

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, c264f4e.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-provider-google already has an open PR.
Ansible already has an open PR.
Ansible already has an open PR.

New Pull Requests

I didn't open any new pull requests because of this PR.

[rileykarson]

Thanks for the great work @drebes!

Fiddling with migrations of name-style resources to MM can be a little trying, so thanks again for your patience in working out how we continue to support the existing behaviour while working within the confines of MM.