A Harness CI pipeline that allows bootstrapping Argo CD once terraform completes the provisioning, in this demo we will provision a GKE cluster using terraform.
The demo uses the following git repositories a sources,
- vanilla-gke - the terraform source repository that will be used with terraform cloud to provision the GKE.
- bootstrap-argocd - the repository that holds kubernetes manifests to bootstrap argo CD on to the GKE cluster
Please make sure you fork those repositories to your Git accounts. For reference sake we will refer the repositories as $TFC_GKE_REPO and $ARGOCD_BOOTSTRAP_REPO.
To make fork and clone easier we will use gh CLI
Navigate to the directory where you want to clone the sources typically it will be $HOME/git.
Clone and fork vanilla-gke repo,
gh repo clone harness-apps/vanilla-gke
cd vanilla-gke
gh repo fork
export TFC_GKE_REPO="$PWD"Clone and fork bootstrap-argocd repo,
cd ..
gh repo clone harness-apps/bootstrap-argocd
cd bootstrap-argocd
gh repo fork
export ARGOCD_BOOTSTRAP_REPO="$PWD"If you don't have Harness Free Tier account, please do sign up for one via Harness Free Tier Sign Up.
We will be using a project named terraform_integration_demos, let us create the project using Harness Web Console,
Update the details as shown,
Follow the wizard leaving rest to defaults and on the last screen choose Continuous Integration,
Click Go to Module to go to project home page.
Click Pipelines to define a new pipeline,
For this demo will be doing manual clone, hence disable the clone,
Click on Pipelines and delete the default Build pipeline,
As part of pipelines we will be pulling image from DockerHub. Let us configure an harnesImage connector as described in https://developer.harness.io/docs/platform/connectors/connect-to-harness-container-image-registry-using-docker-connector/. The pipelines we create as part of the later section will use this connector.
Create a GitHub PAT for the account where you have have forked the repositories $TFC_GKE_REPO and $ARGOCD_BOOTSTRAP_REPO. We will refer to the token as $GITHUB_PAT.
From the Project Setup click Secrets,
Update the encrypted text secret details as shown,
Click Save to save the secret,
As we need to clone the sources from GitHub, we need to define a GitHub Connector, from the Project Setup click Connectors,
From connector list select GitHub,
Enter the name as GitHub,
Click Continue to enter the connector details,
Click Continue and update the GitHub Connector credentials,
When selecting the Personal Access Token make sure you select the GitHub PAT that we defined in previous section,
Click Continue and use select Connect through Harness Platform,
Click Save and Continue to run the connection test, if all went well the connection should successful,
We need Google Service Account(GSA) credentials(JSON Key) to query the GKE cluster details and create resources on it.
export GCP_PROJECT="the Google Cloud Project where Kubernetes Cluster is created"
export GSA_KEY_FILE="path where to store the key file"gcloud iam service-accounts create gke-user \
--description "GKE User" \
--display-name "gke-user"Add permissions to the user to be able to provision kubernetes resources,
gcloud projects add-iam-policy-binding $GCP_PROJECT \
--member="serviceAccount:$GSA_NAME@$GCP_PROJECT.iam.gserviceaccount.com" \
--role="roles/container.admin"IMPORTANT: Only security admins can create the JSON keys. Ensure the Google Cloud user you are using has Security Admin role.
gcloud iam service-accounts keys create "${GSA_KEY_FILE}" \
--iam-account="gke-user@${GCP_PROJECT}.iam.gserviceaccount.com"Get back to the Project Setup click Secrets,
Add the GSA secret details as shown,
IMPORTANT: When you browse and select make sure you select the
$GSA_KEY_FILEas the file for the secret.
Click Save to save the secret,
On your terraform cloud account create a new workspace called vanilla-gke. Update the workspace settings to use Version Control and make it point to $TFC_GKE_REPO.
Configure the workspace with following variables,
For more details on available variables, check Terraform Inputs.
IMPORTANT: The
GOOGLE_CREDENTIALSis Google Service Account JSON Key with permissions to create GKE cluster. Please check the https://github.com/harness-apps/vanilla-gke#pre-requisites for the required roles and permissions. This key will be used by Terraform to create the GKE cluster. When you add the key to terraform variables, you need to make it as base64 encoded e.g.cat YOUR_GOOGLE_CREDENTIALS_KEY_FILE | tr -d \\n
Going forward we will refer to the Terraform Workspace as $TF_WORKSPACE
Your terraform organization as $TF_CLOUD_ORGANIZATION,
Since we need to pull the outputs of terraform run in our CI pipeline we need Terraform API Token.
From your terraform user settings Create an API token,
Make sure to copy the value and save it, we will refer to this token value as $TF_TOKEN_app_terraform_io as part of the Harness CI pipeline.
Getting back to Harness web console, navigate to your project terraform_integration_demos, click Pipelines and Create a Pipeline --> Import From Git,
Update the pipeline details as shown,
IMPORTANT: Make sure the Name of the pipeline is
bootstrap argocd pipelineto make the import succeed with defaults.
Click the bootstrap argocd pipeline from the list to open the Pipeline Studio and click on the stage Bootstrap Argo CD to bring up the pipeline steps,
You can click on each step to see the details.
The Pipeline uses the following secrets,
google_application_credentials- the GSA credentials to manipulate GKEterraform_cloud_api_token- the value of$TF_TOKEN_app_terraform_ioterraform_workspace- the value$TF_WORKSPACEterraform_cloud_organization- the value$TF_CLOUD_ORGANIZATION
We already added google_application_credentials secret as part of the earlier section. Following the similar pattern let us add the terraform_cloud_api_token, terraform_workspace and terraform_cloud_organization as text secrets.
HINT: From the Project Setup click Secrets,
TIP: You can also skip adding
terraform_workspaceandterraform_cloud_organizationas we can extract the values from the webhook payload.
For the Harness CI pipelines to listen to Terraform Cloud Events we need to define a Trigger, navigate back to pipelines and select the bootstrap argocd pipeline --> Triggers,
Click Add New Trigger to add a new webhook trigger(Type: Custom),
On the Configuration page enter the name of the trigger to be tfc notification,
Leave rest of the fields to defaults and click Continue, leave the Conditions to defaults and click Continue.
On the Pipeline Input update the Pipeline Reference Branch to be set to main
NOTE: The Pipeline Reference Branch does not have any implication with this demo as we do manual clone of resources.
Click Create Trigger to create and save the trigger.
Copy the WebHook url,
Let us refer to this value as $TRIGGER_WEBHOOK_URL.
On your terraform cloud console navigate to the workspace Settings --> Notifications,
Click Create Notification and select Webhook as the Destination,
Update the notification details as shown,
Since we need to bootstrap argo CD only on create events we set the triggers to happen only on Completed,
Click Create Notification to finish the creation of notification.
NOTE: The creation would have fired a notification, if the cluster is not ready yet the pipeline would have failed.
Congratulations!!. With this setup any new or updates thats done to the $TFC_GKE_REPO will trigger a plan and apply on Terraform Cloud. A Completed plan will trigger the bootstrap argocd pipline to run and apply the manifests from $BOOTSTRAP_ARGOCD_REPO on the GKE cluster.
An example of successful pipeline run
