support GitHub repositories with google_cloudbuild_trigger

[bbarnes52-zz]

Google's cloud build service allows a user to run CI workloads for code hosted on GitHub, Bitbucket, and Google Cloud Source Repositories (CSRs). Currently, google_cloudbuild_trigger resource only supports CSRs.

This feature request is to support GitHub specifically, though I imagine Bitbucket support would also be valuable to end users.

[skarj]

It seems that this functionality is not implemented even in the GCP CloudBuild API
https://godoc.org/google.golang.org/api/cloudbuild/v1#RepoSource

[hden]

You'll need to manually create a source repository that mirrors the GitHub repo first.

[eveld]

Adding a different resource that would create the mirror between the GitHub and CSR repository would also solve this. But I can imagine the difficulty being that there is now the authentication flow in that action (when performed in the Console).

Edit:
Having googled a bit I found: https://godoc.org/google.golang.org/api/sourcerepo/v1#MirrorConfig
Perhaps this can be leveraged together with the tls_private_key, github_user_ssh_key and google_cloudbuild_trigger to get the required result?

[MPV]

I guess the MirrorConfig in the golang API represents this in the REST API?
https://cloud.google.com/source-repositories/docs/reference/rest/v1/projects.repos#mirrorconfig

...which is mentioned as read-only:

How this repository mirrors a repository managed by another service.
Read-only field.

Until that changes, I assume manually creating the mirrored source repository is what's required to allow cloudbuild triggers from a mirrored repo (?).

[naseemkullah]

There is a way of triggering builds from GitHub via https://github.com/marketplace/google-cloud-build
where the former way has a Connection type of GitHub - mirrored
the new way has a Connection type of Cloud Build GitHub App

I'm not sure if the current google_cloudbuild_trigger resource supports the second way though?

[danawillow]

https://cloud.google.com/cloud-build/docs/api/reference/rest/v1/projects.triggers#GitHubEventsConfig is promising, though no promises on any sort of timeline for TF integration. I'll at least remove the upstream label since it appears to no longer be blocked.

[ocervell]

@danawillow any update on this ?

[danawillow]

Since the API documentation says that the field is experimental, I'd prefer not to add TF support for it at the moment. You might be able to help us out by pushing on that team to get something stable ready that we can use :)

[carlpett]

@danawillow What is the best way to do so? Via our GCP account contact, or is there some other path we can use? Thanks!

[naseemkullah]

I got this response from a cloudbuild engineer a few weeks ago:

Hi!

We're aiming to bring the GCB Triggers API to GA sometime later this year. Once that is done the experimental label will be dropped. Technically the entire BuildTriggers API (even the existing mirrored triggers) are documented as experimental (see https://cloud.google.com/cloud-build/docs/api/reference/rest/v1/projects.triggers/create).

That said, GitHubEventsConfig field will be made available to all users as Beta soon, at which point it should unblock Terraform support. =]

[naseemkullah]

This just in:

The Alpha features GitHub App triggers, GitHub App triggers API, and Settings IAM UI have all launched to Beta and are now publicly available for all GCP projects.

[nat-henderson]

Hey, that's great - let me take a look at what this will take. :)

[nat-henderson]

That's GoogleCloudPlatform/magic-modules#2416.

[orinokai]

It's fantastic that support for this has landed in beta 👍 Is it sensible to suggest that the owner property could optionally be sourced from the GITHUB_ORGANIZATION environment variable? We use this env var in the Github provider and it would avoid having to hardcode the owner in the terraform file or pass as a variable.

[denismaggior8]

That's GoogleCloudPlatform/magic-modules#2416.

Hi @ndmckinley ,

It's nice to see that cloudbuild-github branch has been merged into master, so probably this feature will be released soon.

BTW, I was trying to get a preview of this feature so I've cloned the magic-modules repo, checkout the master branch and applied all the instructions reported here here. I succeeded to compile both google and google-beta providers coming from the master branch (all the tests have passed as well).
I've linked my compiled version of the providers into ~/.terraform.d/plugins/ so now when I type terraform init I'm using my version of both.
Unfortunately, setting this block within a google_cloudbuild_trigger

github {
    owner = "${var.WORLD_REPO_OWNER}"
    name = "${var.WORLD_REPO_NAME}"
    push {
      tag = ".*"
    }
  }

I still get the error:

Error: Unsupported block type

  on triggers.tf line 8, in resource "google_cloudbuild_trigger" "world-trigger":
   8:   github {

Blocks of type "github" are not expected here.

Why I'm not hitting the right version of the GCP provider I've compiled myself (and the one I expect to have this new feature)? Is there something that I'm missing?

I really appreciate your help since I'm stuck with this error.

Best regards,
Denis

[nat-henderson]

It's fantastic that support for this has landed in beta Is it sensible to suggest that the owner property could optionally be sourced from the GITHUB_ORGANIZATION environment variable? We use this env var in the Github provider and it would avoid having to hardcode the owner in the terraform file or pass as a variable.

Would you mind posting a new issue? We can determine if this is a good idea by 👍s on that issue. We try to keep the number of configuration environment variables down, but if a good number of people want this, then we'd be willing to do it.

Why I'm not hitting the right version of the GCP provider I've compiled myself (and the one I expect to have this new feature)? Is there something that I'm missing?

Can you post the output of terraform version? There's a chance there could be an error in the provider. It's also possible that you're using the google provider instead of google-beta?

[orinokai]

@ndmckinley okay no problem, that's done in #4629.

[denismaggior8]

It's fantastic that support for this has landed in beta Is it sensible to suggest that the owner property could optionally be sourced from the GITHUB_ORGANIZATION environment variable? We use this env var in the Github provider and it would avoid having to hardcode the owner in the terraform file or pass as a variable.

Would you mind posting a new issue? We can determine if this is a good idea by 👍s on that issue. We try to keep the number of configuration environment variables down, but if a good number of people want this, then we'd be willing to do it.

Why I'm not hitting the right version of the GCP provider I've compiled myself (and the one I expect to have this new feature)? Is there something that I'm missing?

Can you post the output of terraform version? There's a chance there could be an error in the provider. It's also possible that you're using the google provider instead of google-beta?

HI Nathan and thanks for your quick response.
Here are the infos you requested:

$ terraform version
Terraform v0.12.9

Your version of Terraform is out of date! The latest version
is 0.12.10. You can update by downloading from www.terraform.io/downloads.html

In the providers.tf file I have only declared the google-beta as follows

provider "google-beta" {
 credentials = "${file("${var.CREDENTIAL_FILE}")}"
 project     = "${var.PROJECT_ID}"
 region      = "${var.REGION}"
}

Any clue?

[nat-henderson]

If you only provide google-beta, you still need to add provider = "google-beta" in each block in your config.

If the provider field is omitted, Terraform will implicitly use the google provider by default even if you have only defined a google-beta provider block.

https://www.terraform.io/docs/providers/google/provider_versions.html

Are you doing that? It's not in the config fragment you posted.

[boredland]

Is there a way to automate the creation of a respository mapping? Currently I'd get

Error: Error creating Trigger: googleapi: Error 400: Repository mapping does not exist. Please visit https://console.cloud.google.com/cloud-build/triggers/connect?project=xxx to connect a repository to your project

but I guess that isn't really an automated way...

[nat-henderson]

To the best of my knowledge there isn't one, no. If there is one, it's not in the REST spec - if you find a flow we can sensibly automate we'll seriously consider it.

[orinokai]

To the best of my knowledge there isn't one, no. If there is one, it's not in the REST spec

Ah, that's a shame because the Google Cloud Build GitHub app is authorised for all repositories in the organisation, so it's not like authorisation is the blocker.

I was hoping the github block would be a way to automate creation of Cloudbuild Triggers from start to finish, but it's currently less useful than the trigger_template block because it errors when there is no repository mapping, whereas a trigger_template can be created before the mirrored Google Source Repository is set up.

Is there anywhere we can make a feature request for GitHub repository connection in the GCP API?

[orinokai]

Ah, I found where to make a feature request and have done so. If anyone is interested in this feature, please 'star' it.

https://issuetracker.google.com/issues/142550612

[danawillow]

@ndmckinley, what's remaining here in order to close this?

[nat-henderson]

Ah, this should have been closed back in early October, sorry about that!

[Tenzer]

Has support for this arrived in the non-beta provider? According to the documentation the github block is only supported in the google-beta provider:
https://www.terraform.io/docs/providers/google/r/cloudbuild_trigger.html#github

[danawillow]

That's correct- the beta provider tracks GCP beta features, so it'll stay in the beta provider as long as the GCP functionality is still in beta (see https://cloud.google.com/cloud-build/docs/create-github-app-triggers and https://www.terraform.io/docs/providers/google/guides/provider_versions.html)

Once it goes GA in GCP, feel free to open an issue to have it moved to the GA provider.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!