Initial cluster Terraform configuration #306

cblecker · 2019-07-11T00:30:22Z

Prerequisites:

Have GCP access to k8s-infra-dev-cluster-turnup
Have access to https://app.terraform.io/app/k8s-infra/workspaces

Instructions:

Go to https://app.terraform.io/app/settings/tokens and create a user token
Ensure there is a .terraformrc file in your home directory with the following format:

credentials "app.terraform.io" {
  token = "XXXX"
}

Clone branch, cd into k8s-tf-cluster/ folder
terraform plan will print changes needed to create cluster
terraform apply will apply them
terraform destroy will destroy and clean up all created resources

listx · 2019-07-11T01:40:07Z

k8s-tf-cluster/image/Dockerfile

@@ -0,0 +1,47 @@
+FROM alpine:3.9 as installer


A big part of me wishes that we build deterministic Docker images with Bazel, since all we're doing is extracting a known zip and just setting some ENV vars.

But I'm fine with a Dockerfiles for now.

ameukam · 2019-07-17T19:58:02Z

k8s-tf-cluster/01-datasources.tf

+}
+
+data "google_container_engine_versions" "us-central1" {
+  project        = data.google_project.project.id


Based on the doc, there is no attribute id. Is terraform trying to extract this information from the GCP API ?

Docs are inaccurate. Using a data resource for this confirms that A) the project exists and B) that you have access to that project.

This value is computed to the GCP project ID (e.g. k8s-infra-dev-cluster-turnup).

scottilee · 2019-07-20T04:57:26Z

k8s-tf-cluster/00-inputs.tf

+
+variable "cluster_name" {
+  type        = string
+  description = <<EOT


Just curious, why are you using here docs for the descriptions?

Allows for multi-line descriptions. Only one variable uses it, but it keeps things consistent.

hh · 2019-07-24T15:51:37Z

@cblecker this might be of use for patterns: https://github.com/crosscloudci/cross-cloud

hh · 2019-07-24T15:53:02Z

/cc @denverwilliams
Could use some terraform eyes.

k8s-ci-robot · 2019-07-24T15:53:04Z

@hh: GitHub didn't allow me to request PR reviews from the following users: denverwilliams.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @denverwilliams
Could use some terraform eyes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hh · 2019-07-24T15:55:02Z

/assign

cblecker · 2019-07-24T16:00:08Z

@hh let me know if there's anything specific from the GKE module you'd like to bring in. Having a very quick look at it, it seems like a pretty basic template, but forces certain things like a hardcoded admin password (we just disable this local account), hard coded scopes, and forced use of a single node pool.

ameukam · 2019-07-24T23:04:25Z

k8s-tf-cluster/10-clusterconfiguration.tf

+resource "google_container_node_pool" "pool-1" {
+  provider = google-beta
+
+  name_prefix = "pool-1-"


I would suggest to put this as a variable.

ameukam · 2019-07-24T23:05:03Z

k8s-tf-cluster/10-clusterconfiguration.tf

+
+  // Set machine type, and enable all oauth scopes tied to the service account
+  node_config {
+    machine_type    = "n1-standard-4"


I would suggest to put this as a variable.

ameukam · 2019-07-24T23:18:19Z

k8s-tf-cluster/10-clusterconfiguration.tf

+}
+
+resource "google_bigquery_dataset" "usage_metering" {
+  dataset_id  = "usage_metering"


We should include the name of the cluster in the id so we don't have the same dataset for differents clusters.

ameukam · 2019-07-24T23:37:01Z

k8s-tf-cluster/10-clusterconfiguration.tf

+}
+
+// Create GKE cluster, but with no node pools. Node pools can be provisioned below
+resource "google_container_cluster" "cluster" {


There is no NetworkPolicy Provider deployed. Shouldn't it be deployed ?

ameukam · 2019-07-24T23:54:54Z

I was able to run this PR. Awesome!

google_container_cluster.cluster: Creation complete after 8m42s [id=k8s-services-cluster]
google_container_node_pool.pool-1: Creating...
google_container_node_pool.pool-1: Still creating... [10s elapsed]
google_container_node_pool.pool-1: Still creating... [20s elapsed]
google_container_node_pool.pool-1: Still creating... [30s elapsed]
google_container_node_pool.pool-1: Still creating... [40s elapsed]
google_container_node_pool.pool-1: Still creating... [50s elapsed]
google_container_node_pool.pool-1: Still creating... [1m0s elapsed]
google_container_node_pool.pool-1: Still creating... [1m10s elapsed]
google_container_node_pool.pool-1: Still creating... [1m20s elapsed]
google_container_node_pool.pool-1: Still creating... [1m30s elapsed]
google_container_node_pool.pool-1: Still creating... [1m40s elapsed]
google_container_node_pool.pool-1: Still creating... [1m50s elapsed]
google_container_node_pool.pool-1: Still creating... [2m0s elapsed]
google_container_node_pool.pool-1: Creation complete after 2m8s [id=us-central1/k8s-services-cluster/pool-1-20190724224037646600000001]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

It would be amazing to see the description of this PR in the README, also add the command that generate the application_default_credentials.json file.
Since i think this PR is a terraform version of #250, can you add the workload_metadata_config ?

Edit: Also, can you please specify in the README that the cluster is only accessible through gcloud shell ?

scottilee · 2019-07-25T08:52:53Z

k8s-tf-cluster/10-clusterconfiguration.tf

+
+  // Start with a single node
+  initial_node_count = 1
+


I believe @thockin also wanted to default max pods per node to 32 (default_max_pods_per_node = 32)

scottilee · 2019-07-25T09:01:08Z

Two things I saw on https://github.com/kubernetes/k8s.io/pull/250/files that aren't here:

discussion about whether or not to make this a private cluster, this can be done with:

private_cluster_config = {
  enable_private_nodes = true
}

@ameukam had done bigquery setup, is that still relevant here?

ameukam · 2019-07-25T10:01:10Z

Two things I saw on https://github.com/kubernetes/k8s.io/pull/250/files that aren't here:

AFAIR we didn't have consensus about this option. Enable this have some limitations and require additional configurations

discussion about whether or not to make this a private cluster, this can be done with:
private_cluster_config = {
  enable_private_nodes = true
}

scottilee · 2019-07-25T16:40:25Z

@cblecker Is there anything else I can do to help on this ticket? If it's just setting up a GKE cluster the majority of the work seems to be done. If there's other Terraform code you need written please let me know and I'll be happy to add it.

spiffxp · 2019-08-01T03:06:39Z

What needs to be done / where do we need to get this to for this to reach "not perfect but let's merge and iterate"?

spiffxp · 2019-08-07T19:52:40Z

@thockin to review this one last time, and we'll need to drop the [wip] before we can reach "merge and iterate" for this

clusters/image/Dockerfile

vincepri

Few non-blocking comments, this looks great!

vincepri · 2019-08-21T16:16:42Z

clusters/k8s-infra-dev-cluster-turnup/00-inputs.tf

+EOT
+}
+
+variable "region" {


I'd rework the folder structure to like clusters/<environment>/<region>/<name>/..., this has a few benefits like for example we can setup and generate a terraform.tfvars (https://www.terraform.io/docs/configuration/variables.html#variable-definitions-tfvars-files) from the structure and automatically populate the variables.

vincepri · 2019-08-21T16:17:58Z

clusters/k8s-infra-dev-cluster-turnup/README.md

+}
+```
+- Ensure you are logged into your GCP account with `gcloud auth application-default login`
+- `terraform plan` will print changes needed to create cluster


Prefer using -out= and then apply the generated plan using terraform apply <path>.

vincepri · 2019-08-21T16:19:54Z

clusters/k8s-infra-dev-cluster-turnup/terraform.tfvars

@@ -0,0 +1,2 @@
+project      = "k8s-infra-dev-cluster-turnup"


Is it fair to say that k8s-infra is a prefix, dev is the environment and cluster-turnup is what gives the unique name to the project? If so, I'd split these into multiple variables and document the naming convention for the project somewhere

it's a great suggestion! I think it's better to put this as an issue since we have no real hierarchy or convention and it may take some time to implement this.

clusters/k8s-infra-dev-cluster-turnup/11-pool-1-configuration.tf

Co-Authored-By: Arnaud M. <[email protected]>

ameukam · 2019-08-21T23:13:14Z

/lgtm

cblecker · 2019-08-21T23:37:26Z

/approve
/hold

@thockin Any objections to this as-is for the dev cluster? If you're good, feel free to remove the hold.

k8s-ci-robot · 2019-08-21T23:37:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cblecker

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [cblecker]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

cblecker · 2019-08-23T18:09:37Z

/hold cancel

Merging and we can iterate!

cblecker assigned dims, ameukam and thockin Jul 11, 2019

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 11, 2019

k8s-ci-robot requested review from dims and nikhita July 11, 2019 00:30

k8s-ci-robot added the wg/k8s-infra label Jul 11, 2019

cblecker mentioned this pull request Jul 11, 2019

Burn down and recreate k8s-infra cluster #243

Closed

listx reviewed Jul 11, 2019

View reviewed changes

ameukam reviewed Jul 17, 2019

View reviewed changes

scottilee reviewed Jul 20, 2019

View reviewed changes

k8s-ci-robot assigned hh Jul 24, 2019

ameukam reviewed Jul 24, 2019

View reviewed changes

scottilee reviewed Jul 25, 2019

View reviewed changes

Initial commit of cluster Terraform configurations

f595298

cblecker force-pushed the tf-wip branch from 0d0e796 to f595298 Compare August 21, 2019 05:00

cblecker changed the title ~~[WIP] Terraform GKE Config~~ Initial cluster Terraform configuration Aug 21, 2019

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 21, 2019

ameukam reviewed Aug 21, 2019

View reviewed changes

clusters/image/Dockerfile Outdated Show resolved Hide resolved

vincepri reviewed Aug 21, 2019

View reviewed changes

ameukam reviewed Aug 21, 2019

View reviewed changes

clusters/k8s-infra-dev-cluster-turnup/11-pool-1-configuration.tf Outdated Show resolved Hide resolved

cblecker force-pushed the tf-wip branch from 72df87f to 84af1f2 Compare August 21, 2019 20:00

Update TF version in image

5dd741f

Co-Authored-By: Arnaud M. <[email protected]>

cblecker force-pushed the tf-wip branch from 84af1f2 to 5dd741f Compare August 21, 2019 20:06

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 21, 2019

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 21, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 21, 2019

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 23, 2019

k8s-ci-robot merged commit 4dd90f2 into kubernetes:master Aug 23, 2019

cblecker deleted the tf-wip branch August 23, 2019 20:01

ameukam mentioned this pull request Sep 12, 2019

Enable GCS and multiple clusters for terraform #364

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial cluster Terraform configuration #306

Initial cluster Terraform configuration #306

cblecker commented Jul 11, 2019 •

edited

Loading

listx Jul 11, 2019

ameukam Jul 17, 2019

cblecker Jul 17, 2019

scottilee Jul 20, 2019

cblecker Jul 20, 2019

hh commented Jul 24, 2019

hh commented Jul 24, 2019

k8s-ci-robot commented Jul 24, 2019

hh commented Jul 24, 2019

cblecker commented Jul 24, 2019

ameukam Jul 24, 2019

ameukam Jul 24, 2019 •

edited

Loading

ameukam Jul 24, 2019

ameukam Jul 24, 2019 •

edited

Loading

ameukam commented Jul 24, 2019 •

edited

Loading

scottilee Jul 25, 2019

scottilee commented Jul 25, 2019

ameukam commented Jul 25, 2019 •

edited

Loading

scottilee commented Jul 25, 2019

spiffxp commented Aug 1, 2019

spiffxp commented Aug 7, 2019

vincepri left a comment

vincepri Aug 21, 2019

vincepri Aug 21, 2019

vincepri Aug 21, 2019

ameukam Aug 21, 2019

ameukam commented Aug 21, 2019

cblecker commented Aug 21, 2019

k8s-ci-robot commented Aug 21, 2019

cblecker commented Aug 23, 2019

Initial cluster Terraform configuration #306

Initial cluster Terraform configuration #306

Conversation

cblecker commented Jul 11, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hh commented Jul 24, 2019

hh commented Jul 24, 2019

k8s-ci-robot commented Jul 24, 2019

hh commented Jul 24, 2019

cblecker commented Jul 24, 2019

Choose a reason for hiding this comment

ameukam Jul 24, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ameukam Jul 24, 2019 • edited Loading

Choose a reason for hiding this comment

ameukam commented Jul 24, 2019 • edited Loading

Choose a reason for hiding this comment

scottilee commented Jul 25, 2019

ameukam commented Jul 25, 2019 • edited Loading

scottilee commented Jul 25, 2019

spiffxp commented Aug 1, 2019

spiffxp commented Aug 7, 2019

vincepri left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ameukam commented Aug 21, 2019

cblecker commented Aug 21, 2019

k8s-ci-robot commented Aug 21, 2019

cblecker commented Aug 23, 2019

cblecker commented Jul 11, 2019 •

edited

Loading

ameukam Jul 24, 2019 •

edited

Loading

ameukam Jul 24, 2019 •

edited

Loading

ameukam commented Jul 24, 2019 •

edited

Loading

ameukam commented Jul 25, 2019 •

edited

Loading