roachprod: spread instances out over AZs for a region #36400
Labels
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
no-issue-activity
X-stale
Comments
ajwerner
added a commit
to ajwerner/cockroach
that referenced
this issue
Apr 2, 2019
In order to run a roachprod cluster in AWS one needs to configure vms with resources which exist on a per-region or per-availability zone basis. Each region requires a VPC with security groups, AMIs and subnets which defines the availability zone within the region. Furthermore, each region's VPC must be configured with peering to each other VPC. Currently this is set up just for 3 regions (us-east-2, us-west-2, eu-west-2). Using additional regions requires setting up the required resources manually and then connecting them through a variety of less than ergonomic flags. In gcloud we have a different story; users can just specify a list of availability zones in which to place the instances. This could yet be made better by providing an easy way to distribute nodes within regions over availability zones (see cockroachdb#36400). This PR creates the machinery to set up AWS for convenient use with arbitrary regions. The basic architecture is that there's a collection of terraform files under `cmd/roachprod/vm/aws/terraform` which defines resources and modules to manage the above described required AWS resources. The `main.tf` for the terraform project is generated by a go program `terraformgen` which allows users to specify the account number, resource name prefix and the set of regions. The templating is especially important because of the need for pair-wise peerings between each region. The terraform project is set up to output the relevant IDs which, with the combination of the the `--json` flag will be used to produce an artifact which roachprod will consume. Release note: None
ajwerner
added a commit
to ajwerner/cockroach
that referenced
this issue
Apr 2, 2019
In order to run a roachprod cluster in AWS one needs to configure vms with resources which exist on a per-region or per-availability zone basis. Each region requires a VPC with security groups, AMIs and subnets which defines the availability zone within the region. Furthermore, each region's VPC must be configured with peering to each other VPC. Currently this is set up just for 3 regions (us-east-2, us-west-2, eu-west-2). Using additional regions requires setting up the required resources manually and then connecting them through a variety of less than ergonomic flags. In gcloud we have a different story; users can just specify a list of availability zones in which to place the instances. This could yet be made better by providing an easy way to distribute nodes within regions over availability zones (see cockroachdb#36400). This PR creates the machinery to set up AWS for convenient use with arbitrary regions. The basic architecture is that there's a collection of terraform files under `cmd/roachprod/vm/aws/terraform` which defines resources and modules to manage the above described required AWS resources. The `main.tf` for the terraform project is generated by a go program `terraformgen` which allows users to specify the account number, resource name prefix and the set of regions. The templating is especially important because of the need for pair-wise peerings between each region. The terraform project is set up to output the relevant IDs which, with the combination of the the `--json` flag will be used to produce an artifact which roachprod will consume. Release note: None
craig bot
pushed a commit
that referenced
this issue
Apr 3, 2019
36418: roachprod: create terraform-based API resource management for AWS r=ajwerner a=ajwerner In order to run a roachprod cluster in AWS one needs to configure vms with resources which exist on a per-region or per-availability zone basis. Each region requires a VPC with security groups, AMIs and subnets which defines the availability zone within the region. Furthermore, each region's VPC must be configured with peering to each other VPC. Currently this is set up just for 3 regions (us-east-2, us-west-2, eu-west-2). Using additional regions requires setting up the required resources manually and then connecting them through a variety of less than ergonomic flags. In gcloud we have a different story; users can just specify a list of availability zones in which to place the instances. This could yet be made better by providing an easy way to distribute nodes within regions over availability zones (see #36400). This PR creates the machinery to set up AWS for convenient use with arbitrary regions. The basic architecture is that there's a collection of terraform files under `cmd/roachprod/vm/aws/terraform` which defines resources and modules to manage the above described required AWS resources. The `main.tf` for the terraform project is generated by a go program `terraformgen` which allows users to specify the account number, resource name prefix and the set of regions. The templating is especially important because of the need for pair-wise peerings between each region. The terraform project is set up to output the relevant IDs which, with the combination of the the `--json` flag will be used to produce an artifact which roachprod will consume. The second commit adopts this information from the first by setting up `go:generate` directives to: 1. Generate the terraform `main.tf` which is used to generate `config.json`. 2. Generate the `embedded.go` bindata file to contain the default json value. The commit then defines the `awsConfig` struct to usefully expose the information produced by terraform to create instances. Users can override the embedded default configuration by providing similarly formatted json in a file passed by path to the '--aws-config` flag. Co-authored-by: Andrew Werner <[email protected]>
awoods187
added
the
C-enhancement
|
We have marked this issue as stale because it has been inactive for |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently on both AWS and GCP roachprod always places instances in the same AZ for a given region. This is likely not the desired behavior, we should add a flag to enable the current concentration and add support to spread nodes over AZs.
Note that supporting this on GCP will be more work than on AWS as GCP is currently configured to only know about a small number of availability zones whereas AWS knows about the AZs but chooses a random one.
The text was updated successfully, but these errors were encountered: