Skip to content

Commit

Permalink
Add a task for creating a GKE cluster for e2e tests 🧪
Browse files Browse the repository at this point in the history
As part of tektoncd#373 I'm continuing to create Tasks that we can use to run
tests for items in the catalog. This Task creates a cluster within a GKE
project and is based on the way we create clusters for the existing end
to end tests we have in Tekton, which are themselves based on Knative,
when are THEMselves based on k8s. These tests all use a tool called
kubetest, which is responsible for invoking boskos (see tektoncd#408) and
creating a GKE cluster
(https://github.com/kubernetes/test-infra/tree/master/kubetest).

This Task attempts to create a cluster in the same way, which based on the
output in the end to end logs seems to consist of creating the cluster
and setting up a firewall for it.

I'm not sure if this belongs in the catalog itself since it is specific
to our own end to end tests and isn't as generic as other catalog tests,
so if we want to move this into the test directory that seems fine, but
I also think it could be an okay addition to the catalog on its own (esp
if other k8s based projects that test against GKE want to use it.)
  • Loading branch information
bobcatfish committed Jul 22, 2020
1 parent 7a481c5 commit 22829be
Show file tree
Hide file tree
Showing 4 changed files with 189 additions and 0 deletions.
43 changes: 43 additions & 0 deletions gke-e2e-cluster-create/0.1/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# GKE End to End Cluster Create

The Task `gke-e2e-cluster-create` can be used to create a GKE cluster in a GCP
project and fetch a kubeconfig that can be used (in a context with both kubectl and gcloud
available) to make requests to the cluster.

The cluster created will have a firewall applied such that the only traffic allowed to instances
in the cluster will be SSH, TCP services running on 80 or 8080, and services exposed via the
NodePort default range (https://kubernetes.io/docs/concepts/services-networking/service/#nodeport).

## Parameters

* **project-name**: The name of the GCP project in which to create the GKE cluster. (_required_)
* **private-key-path**: The path to the private key within the gcp-service-account workspace. (_required_)
* **identifier**: A string which identifies the purpose for which this cluster is being created. Used to name other resources created. (_required_)
* **min-nodes**: The minimum number of nodes in the cluster. (_default_:1)
* **max-nodes**: The maximum number of nodes in the cluster. (_default_:3)
* **region**: The region to create the cluster in. (_default_:us-central1)
* **machine-type**: The machine type to create, from
https://cloud.google.com/compute/docs/machine-types. (_default_:n1-standard-4)
* **image-type**: The type of image to create the nodes, from
https://cloud.google.com/kubernetes-engine/docs/concepts/node-images. (_default_:cos)
* **cluster-version**: The GKE version to install, in a format that can be used as the
`--cluster-version` argument to https://cloud.google.com/sdk/gcloud/reference/beta/container/clusters/create
(_default_: latest)

## Workspaces

* **gcp-service-account**: A Secret or volume containing the private key of a GCP service account
that can create GKE clusters in the project
* **kubeconfig**: A workspace into which a kubeconfig file called `kubeconfig` will be written that
will contain the information required to access the cluster. The `kubeconfig` will expect to use
gcloud to authenticate, so in order for it to be used it must be run in a container which contains
both kubectl and gcloud.

## Results

* **cluster-name** The name of the cluster that was created.

## Usage

See [samples/create-gke-cluster.yaml](samples/create-gke-cluster.yaml) for an example of a TaskRun
that creates a GKE cluster and writes the kubeconfig to a PVC.
105 changes: 105 additions & 0 deletions gke-e2e-cluster-create/0.1/gke-e2e-cluster-create.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: gke-e2e-cluster-create
labels:
app.kubernetes.io/version: "0.1"
annotations:
tekton.dev/pipelines.minVersion: "0.12.1"
tekton.dev/tags: "gke,test"
tekton.dev/displayName: "GKE End to End Cluster Create"
spec:
description: |
This Task can be used to create a GKE cluster in a GCP project and fetch a kubeconfig that
can be used (in a context with both kubectl and gcloud available) to make requests to the cluster.
params:
- name: project-name
description: The name of the GCP project in which to create the GKE cluster.
- name: private-key-path
description: The path to the private key within the gcp-service-account workspace.
- name: identifier
description: A string which identifies the purpose for which this cluster is being created. Used to name other resources created.
- name: min-nodes
description: The minimum number of nodes in the cluster.
default: "1"
- name: max-nodes
description: The maximum number of nodes in the cluster.
default: "3"
- name: region
description: The region to create the cluster in.
default: us-central1
- name: machine-type
description: The machine type to create, from https://cloud.google.com/compute/docs/machine-types.
default: n1-standard-4
- name: image-type
description: The type of image to create the nodes, from https://cloud.google.com/kubernetes-engine/docs/concepts/node-images.
default: cos
- name: cluster-version
description: |
The GKE version to install, in a format that can be used as the `--cluster-version` argument to
https://cloud.google.com/sdk/gcloud/reference/beta/container/clusters/create
default: latest
workspaces:
- name: gcp-service-account
description: A Secret or volume containing the private key of a GCP service account that can create GKE clusters in the project
- name: kubeconfig
description: |
A workspace into which a kubeconfig file called `kubeconfig` will be written that will contain the information
required to access the cluster. The `kubeconfig` will expect to use gcloud to authenticate, so in order for it to
be used it must be run in a container which contains both kubectl and gcloud.
results:
- name: cluster-name
description: The name of the cluster that was created.
steps:
- name: gcloud
image: google/cloud-sdk:slim@sha256:27b2c22bf259d9bc1a291e99c63791ba0c27a04d2db0a43241ba0f1f20f4067f
script: |
UNIQUE_STR=$(head /dev/urandom | tr -dc a-z0-9 | head -c 10 ; echo '')
UNIQUE_NAME=$(params.identifier)-$UNIQUE_STR
# Configure gcloud to use the provided project and service account
gcloud auth activate-service-account --key-file=$(workspaces.gcp-service-account.path)/$(params.private-key-path)
gcloud config set project $(params.project-name)
# Create a network and a new cluster
gcloud compute networks create $UNIQUE_NAME --project $(params.project-name) --subnet-mode=auto
gcloud beta container clusters create \
--quiet \
--enable-autoscaling \
--scopes=cloud-platform \
--enable-basic-auth \
--no-issue-client-certificate \
--project=$(params.project-name) \
--cluster-version=$(params.cluster-version) \
--min-nodes=$(params.min-nodes) \
--max-nodes=$(params.max-nodes) \
--region=$(params.region) \
--machine-type=$(params.machine-type) \
--image-type=$(params.image-type) \
--num-nodes=1 \
--network=$UNIQUE_NAME\
$UNIQUE_NAME
# Write the kubeconfig for connecting to the new cluster to the provided workspace
KUBECONFIG=$(workspaces.kubeconfig.path)/kubeconfig gcloud beta container clusters get-credentials \
--project=$(params.project-name) \
--region=$(params.region) \
$UNIQUE_NAME
# Get the tag used for the instances created and use that to apply firewall rules to them
INSTANCE_TAG=$(gcloud compute instances list \
--project=$(params.project-name) \
--filter=metadata.cluster-name=$UNIQUE_NAME \
--limit=1 \
--format=get\(tags.items\) | tr -d '\n')
# This firewall rule allows the cluster to expose SSH, TCP services running on 80 or 8080,
# and services exposed via the NodePort default range (https://kubernetes.io/docs/concepts/services-networking/service/#nodeport)
gcloud compute firewall-rules create e2e-ports-$UNIQUE_STR \
--project=$(params.project-name) \
--network=$UNIQUE_NAME \
--allow=tcp:22,tcp:80,tcp:8080,tcp:30000-32767,udp:30000-32767 \
--target-tags=$INSTANCE_TAG
printf $UNIQUE_NAME > /tekton/results/cluster-name
36 changes: 36 additions & 0 deletions gke-e2e-cluster-create/0.1/samples/create-gke-cluster.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: kubeconfig-pvc
spec:
resources:
requests:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
---
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
generateName: create-gke-cluster-
spec:
taskRef:
name: gke-e2e-cluster-create
params:
- name: project-name
value: tekton-prow-14
- name: private-key-path
value: service-account.json
- name: identifier
value: try-it-out
workspaces:
- name: gcp-service-account
secret:
# This secret exists in the tekton-releases prow cluster
secretName: test-account
- name: kubeconfig
persistentVolumeClaim:
claimName: kubeconfig-pvc


5 changes: 5 additions & 0 deletions gke-e2e-cluster-create/OWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
approvers:
- bobcatfish

reviewers:
- bobcatfish

0 comments on commit 22829be

Please sign in to comment.