Terraform / MagicModules integration, part I (#1)

This sets up the CI system by which changes can flow from MagicModules down to Terraform. In its current form it only does Terraform, but a major goal is to extend it to all generated repositories.

The code here refers to a handful of docker containers I had to create - one of them has its Dockerfile checked in here, but the rest are built from other, existing repositories. Still working to get my changes to those upstream repos merged, but in the meantime I've built my own versions and am using those.

This isn't all the way done: I've commented out "trigger: true" in a few places here because I don't think this is ready to be fully automated yet. It also currently does all the work in a user's own forks rather than the main repos. I want to review it as-is (since it's approximately the minimum viable proof-of-concept), and I'll try to make smaller changes from here on in.

README.md

@@ -1,9 +1,60 @@
-
-Terraform Provider for GCP CI-Related tools
+Concourse CI tools for MagicModules and Google Providers
 ===
 
-Small collection of tools related to CI and terraform-provider-google. Right now
-this is experimental and not supported in any way.
+These tools manage the downstream repositories of [magic-modules](https://github.com/GoogleCloudPlatform/magic-modules).
+
+# Jobs
+The concourse pipeline defined here runs through four stages when a Github pull request is opened against `magic-modules`.
+## `mm-generate`
+The first job's goal is to generate the downstream repositories of MagicModules.  This is done almost entirely by the `generate` task.
+
+### `generate`
+`generate` takes two inputs:
+* The magic-modules repository, with the pull request's `head` checked out and the repo in "detached `HEAD`" state.
+* This CI repository.
+
+It then runs `magic-modules/generate.yml`, which specifies one output:
+
+* The magic-modules repository, after generation has been accomplished.
+
+Since Concourse doesn't support input/output directories, the pre and post directories have different names.  The input is `magic-modules` and the output is `mm-output`.
+
+After that, the generated repositories are uploaded to GitHub, in the concourse process runner's forks.  The pull request is updated to point to those forks.
+
+## Individual repo tests
+
+The second stage's goal is to confirm that the individual repos still pass tests.  It runs only after the first stage finishes.
+
+### `terraform-test`
+
+`terraform-test` takes two inputs:
+* The updated magic-modules repo, after the robot adds its commit to point the submodules to the new generated version.
+* This CI repository.
+
+It then runs `unit-tests/task.yml`, which has no outputs because it makes no changes to any code.  It just runs and succeeds if the unit tests pass, and fails if the unit tests don't pass.  The failure and detailed logs are available in concourse.
+
+## `create-prs`
+
+The third stage's goal is to create the downstream PRs.
+
+TODO(ndmckinley): fill in this section - downstream PR creation changes dramatically in a followup PR.
+
+## `merge-prs`
+
+The fourth stage's goal is to merge `magic-modules` PRs after they have been approved and all the downstream PRs have been merged.
+
+### `merge-and-update`
+
+`merge-and-update` takes two inputs:
+* The approved PR, after all downstream PRs have been merged.
+* This CI repository.
+
+It then runs `magic-modules/merge.yml`, which declares one output:
+
+* `mm-output`: the `magic-modules` PR repo after it has been updated to be ready to merge.
+
+This job sets the submodules back to tracking their downstream repositories on `master`.  It updates the submodules to point to the most recent commit on `master`.  This may not be ideal - other commits may have been made to `master` since the downstream PR was merged - however, this is the best way to ensure that we do not go backwards.  Imagine the following situation:
+
+`magic-modules` PR #1 is created.  `downstream-repo-a` PR #7 and `downstream-repo-b` PR #8 are created from that PR.  `magic-modules` PR #2 is created.  `downstream-repo-a` PR #9 is created from that PR.  PR #7 is merged, then PR #9.  Since all of PR #2's downstream PRs are merged, PR #2 is merged, and it includes the changes from PR #1.  PR #8 is finally merged, and so PR #1 is ready to be merged.
 
-This repo is not intended to be long-lived; rather it should eventually be
-merged upstream.
+When merging PR #1, if we update `downstream-repo-a` to the merge commit for PR #7, we will go backwards, erasing PR #2.  If we update to `master`, we will definitely include both the changes of both PR #1 and PR #2.

ci.yml

@@ -0,0 +1,167 @@
+# These resource types are here until the PRs get merged in upstream.  :)
+resource_types:
+    - name: git-branch
+      type: docker-image
+      source:
+          repository: nmckinley/concourse-git-resource
+          tag: v0.1.7
+
+    - name: github-pull-request
+      type: docker-image
+      source:
+          repository: nmckinley/concourse-github-pr-resource
+          tag: v0.1.1
+
+resources:
+    - name: magic-modules
+      type: git-branch
+      source:
+          uri: [email protected]:((username))/magic-modules.git
+          private_key: ((repo_key))
+
+    - name: magic-modules-new-prs
+      type: github-pull-request
+      source:
+          repo: ((username))/magic-modules
+          private_key: ((repo_key))
+          access_token: ((repo_access))
+          only_mergeable: true
+          authorship_restriction: true
+
+    - name: terraform-intermediate
+      type: git-branch
+      source:
+          uri: [email protected]:((username))/terraform-provider-google.git
+          private_key: ((repo_key))
+
+    - name: ci
+      type: git
+      source:
+          uri: [email protected]:((username))/terraform-provider-google-ci.git
+          branch: git-pull-resource
+          private_key: ((repo_key))
+
+    - name: mm-approved-prs
+      type: github-pull-request
+      source:
+          repo: ((username))/magic-modules
+          private_key: ((repo_key))
+          access_token: ((repo_access))
+          only_mergeable: true
+          require_review_approval: true
+
+    - name: mm-prs-out
+      type: github-pull-request
+      source:
+          repo: ((username))/magic-modules
+          private_key: ((repo_key))
+          access_token: ((repo_access))
+
+    - name: terraform-prs-out
+      type: github-pull-request
+      source:
+          repo: ((username))/terraform-provider-google
+          private_key: ((repo_key))
+          access_token: ((repo_access))
+
+jobs:
+    - name: mm-generate
+      plan:
+          - aggregate:
+              - get: magic-modules
+                resource: magic-modules-new-prs
+                version: every
+                # trigger: true
+                params:
+                    submodules: [build/terraform]
+              - get: ci
+            # TODO(@ndmckinley): This task is too monolithic and should be split up.
+          - task: generate
+            file: ci/magic-modules/generate.yml
+            params:
+                GH_USERNAME: ((username))
+          - put: terraform-intermediate
+            params:
+                repository: mm-output/build/terraform
+                branch_file: mm-output/branchname
+                only_if_diff: true
+
+          # This needs to go below all the submodules because it has a commit which
+          # points to those submodules.
+          - put: magic-modules
+            params:
+                repository: mm-output
+                branch_file: mm-output/branchname
+                only_if_diff: true
+
+    - name: terraform-test
+      plan:
+          - aggregate:
+              - get: ci
+              - get: magic-modules
+                version: every
+                # trigger: true
+                params:
+                    submodules: [build/terraform]
+                passed: [mm-generate]
+          - task: test
+            file: ci/unit-tests/task.yml
+
+    - name: create-prs
+      plan:
+          - get: ci
+          - get: magic-modules
+            version: every
+            # trigger: true
+            params:
+                submodules: [build/terraform]
+            passed: [terraform-test, mm-generate]
+            # TODO(@ndmckinley): This get/task/put/task/put flow is confusing and should be cleaned up.
+          - get: mm-initial-pr
+            resource: magic-modules-new-prs
+            passed: [mm-generate]
+            version: every
+          - task: make-pr
+            file: ci/magic-modules/make-pr.yml
+          - put: magic-modules
+            params:
+                branch_file: mm-initial-pr/.git/branch
+                repository: magic-modules-out
+                only_if_diff: true
+          - put: terraform-prs-out
+            params:
+                path: magic-modules-out/build/terraform
+                status: success
+          - task: make-comment
+            file: ci/magic-modules/make-comment.yml
+            params:
+                GH_USERNAME: ((username))
+          - put: mm-prs-out
+            params:
+                status: success
+                path: magic-modules-comment
+                comment: magic-modules-comment/pr_comment
+
+    - name: merge-prs
+      plan:
+          - get: mm-approved-prs
+          - get: ci
+          - task: merge-and-update
+            file: ci/magic-modules/merge.yml
+            params:
+                CREDS: ((repo_key))
+          - put: magic-modules
+            params:
+                repository: mm-output
+                branch_file: mm-approved-prs/.git/branch
+                only_if_diff: true
+          - put: mm-approved-prs
+            params:
+                path: mm-output
+                status: success
+                merge:
+                    method: squash
+                    commit_msg: mm-output/commit_message
+
+# TODO(@ndmckinley): System for confirming no downstream conflicts.
+# TODO(@ndmckinley): System for building the Docker containers this depends on.

containers/go-ruby/Dockerfile

@@ -0,0 +1,29 @@
+from golang:1.9
+
+RUN ssh-keyscan github.com >> /known_hosts
+RUN echo "UserKnownHostsFile /known_hosts" >> /etc/ssh/ssh_config
+RUN apt-get update
+RUN apt-get install -y bzip2 libssl-dev libreadline-dev zlib1g-dev
+
+ENV RUBY_VERSION 2.5.0
+ENV RUBYGEMS_VERSION 2.7.4
+ENV BUNDLER_VERSION 1.16.1
+
+RUN git clone https://github.com/rbenv/ruby-build.git
+RUN PREFIX=/usr/local ./ruby-build/install.sh
+RUN ruby-build "$RUBY_VERSION" /usr/
+
+RUN gem update --system "$RUBYGEMS_VERSION"
+RUN gem install bundler --version "$BUNDLER_VERSION" --force
+
+# install things globally, for great justice
+# and don't create ".bundle" in all our apps
+ENV GEM_HOME /usr/local/bundle
+ENV BUNDLE_PATH="$GEM_HOME" \
+	BUNDLE_BIN="$GEM_HOME/bin" \
+	BUNDLE_SILENCE_ROOT_WARNING=1 \
+	BUNDLE_APP_CONFIG="$GEM_HOME"
+ENV PATH $BUNDLE_BIN:$PATH
+RUN mkdir -p "$GEM_HOME" "$BUNDLE_BIN" \
+	&& chmod 777 "$GEM_HOME" "$BUNDLE_BIN"
+

magic-modules/create-pr.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+# This script configures the git submodule under magic-modules so that it is
+# ready to create a new pull request.  It is cloned in a detached-head state,
+# but its branch is relevant to the PR creation process, so we want to make
+# sure that it's on a branch, and most importantly that that branch tracks
+# a branch upstream.
+
+set -e
+set -x
+
+shopt -s dotglob
+cp -r magic-modules/* magic-modules-out
+
+cd magic-modules-out
+
+# This says "check out the branch which contains HEAD, and set it up to track its upstream."
+git checkout -t "$(git branch -a --contains "$(git rev-parse HEAD)" | grep -v "detached")"
+
+cd build/terraform
+
+git checkout -t "$(git branch -a --contains "$(git rev-parse HEAD)" | grep -v "detached")"
+# This special string 'new' tells the PR resource to create a new PR.
+git config pullrequest.id new

magic-modules/generate.yml

@@ -0,0 +1,24 @@
+---
+# This file takes two inputs: magic-modules in detached-HEAD state, and the CI repo.
+# It spits out "mm-output", a magic-modules repo on a new branch (named after the
+# HEAD commit on the PR), with submodules updated to be on a branch with the same name.
+platform: linux
+
+image_resource:
+    type: docker-image
+    source:
+        repository: nmckinley/go-ruby
+        tag: '1.9-2.5'
+
+inputs:
+    - name: magic-modules
+    - name: ci
+
+outputs:
+    - name: mm-output
+
+run:
+    path: ci/magic-modules/generate_terraform.sh
+
+params:
+    GH_USERNAME: ""

magic-modules/generate_terraform.sh

@@ -0,0 +1,60 @@
+#!/bin/bash
+
+# This script takes in 'magic-modules', a git repo tracking the head of a PR against magic-modules.
+# It needs to output the same git repo, but with the code generation done, at 'mm-output'.
+
+# Setup GOPATH
+export GOPATH="${PWD}/go"
+WORKDIR="${PWD}"
+
+set -x
+set -e
+
+# Create $GOPATH structure - in order to successfully run Terraform codegen, we need to run
+# it with a correctly-set-up $GOPATH.  It calls out to `goimports`, which means that
+# we need to have all the dependencies correctly downloaded.
+mkdir -p "${GOPATH}/src/github.com/terraform-providers"
+
+pushd magic-modules
+git submodule update --init build/terraform
+ln -s "${PWD}/build/terraform/" "${GOPATH}/src/github.com/terraform-providers/terraform-provider-google"
+popd
+
+pushd "${GOPATH}/src/github.com/terraform-providers/terraform-provider-google"
+
+go get
+
+popd
+
+pushd magic-modules
+# We're going to use the short commit sha of the git repo's head as the branch name for the generated code.
+BRANCH=$(git rev-parse --short HEAD)
+
+bundle install
+bundle exec compiler -p products/compute -e terraform -o build/terraform
+
+pushd "build/terraform"
+git add -A
+git config --global user.email "[email protected]"
+git config --global user.name "Modular Magician"
+git commit -m "magic modules change happened here" || true  # don't crash if no changes
+# TODO(@ndmckinley): A better message that comes from the body of the magic-modules PR.
+git checkout -B "$BRANCH"
+popd
+
+git config -f .gitmodules submodule.build/terraform.branch "$BRANCH"
+git config -f .gitmodules submodule.build/terraform.url "[email protected]:$GH_USERNAME/terraform-provider-google.git"
+git submodule sync build/terraform
+
+# ./branchname is intentionally not committed - but run *before* the commit, because it should contain the hash of
+# the commit which kicked off this process, *not* the resulting commit.
+echo "$BRANCH" > ./branchname
+
+git add build/terraform
+git add .gitmodules
+git commit -m "update terraform." || true  # don't crash if no changes
+# TODO(@ndmckinley): A better message that comes from the body of the magic-modules PR.
+git checkout -B "$BRANCH"
+
+cp -r ./ "${WORKDIR}/mm-output/"
+popd