Github Multi-Runner deployment on ec2 instances

This Module is a wrapper of the original Philips Labs Multi-Runner module. All credits for the original implementation goes to philips-labs, the module in this repo has been created according to the MIT license.

The goal of this wrapper is to solve one simple problem: the deployment of a basic Github Runner setup. We aimed at hiding as much configuration as possible behind defaults, giving the user a minimal set of required variables for a fast, opinionated deployment of the original multi-runner module.

Ephemeral and persistent runner configurations are both possible as well as x64 and arm64. They are configurable with their respective parameters runner.ephemeral and runner.architecture.

By default, a single set of persistent, x64 runners with a minimum size of 1 is created.

Usage

Example deployment with the required variables:

This snippet will deploy two set of runners:

1. x64 with labels ["self-hosted", "linux", "x64", "team-red", "spot"]
2. arm64 with labels ["self-hosted", "linux", "arm64", "team-blue", "on-demand"]

each have a default maximum count of 15 runners and 1 idle/warm runner each during office hours (8am-7pm Zurich time)

module "example_multi_runner" {
  source                    = "github.com/tx-pts-dai/terraform-aws-ec2-actions-runners?ref=vX.X.X"
  unique_prefix             = "build-runners"
  github_app_multirunner_id = "123456"
  github_app_key_base64     = "myprivatekey"
  vpc_id                    = "vpc-01234567"
  subnet_ids                = ["subnet-0123456", "subnet-1234567"]
  runners = {
    team-red-x64 = {
      architecture       = "x64"
      instance_types     = ["c6i.large"]
      labels             = ["team-red"]
      use_spot_instances = true
    }
    team-blue-arm64 = {
      architecture       = "arm64"
      instance_types     = ["c7g.large"]
      labels             = ["team-blue"]
      ephemeral          = true
    }
  }
}

You can select the runners in a github workflow with:

runs-on: ["self-hosted", "linux", "x64", "team-red", "spot"]
# or for arm64 arch
runs-on: ["self-hosted", "linux", "arm64", "team-blue", "on-demand"]

Keep in mind that a subset of labels can be used too, for example you can use ["self-hosted", "linux"] to select either one or the other set indifferently.

The labels used by the runners are set as a Terraform output runner_labels. Our module adds the following labels additionally to the one you specify with runner.labels:

1. Architecture -> either x64 or arm64
2. OS -> either linux or windows
3. Capacity type -> either on-demand or spot
4. Self-hosted -> self-hosted

IMPORTANT: When destroying the resources created by this module, there could be some EC2 instances as leftovers. Since they are launched dynamically via Lambda function, Terraform doesn't have any knowledge about them, therefore you should terminate/refresh them manually.

Github Application (required)

Please follow the instruction on the original repo Setup Github Application

The webhook_endpoint can be obtained as output from the module via terraform output module.MY_MODULE_NAME.webhook_endpoint.

The webhook_secret can be obtained in two ways:

1. As output from the module via terraform output module.MY_MODULE_NAME.webhook_endpoint_secret. This requires a valid terraform initialization.
2. From SSM: aws ssm get-parameter --name /github-action-runners/MY_RUNNERS_UNIQUE_PREFIX/app/github_app_webhook_secret --with-decryption --output json, note that this is an ecrypted parameter, therefore you need the flag --with-decryption. This requires a valid access to aws.

The Github App private key is also stored encrypted in ssm, if needed it can be retrieved with the following command: aws ssm get-parameter --name /github-action-runners/MY_RUNNERS_UNIQUE_PREFIX/app/github_app_key_base64 --with-decryption

Contributing

This repo has a pre-commit configuration and a workflow that verify that all checks pass on each PR.

Pre-Commit

Installation: install pre-commit and execute pre-commit install. This will generate pre-commit hooks according to the config in .pre-commit-config.yaml

Before submitting a PR be sure to have used the pre-commit hooks or run: pre-commit run -a

The pre-commit command will run:

• Terraform fmt
• Terraform validate
• Terraform docs
• Terraform validate with tflint
• check for merge conflicts
• fix end of files

as described in the .pre-commit-config.yaml file

Update upstream

In order to update the upstream module version we need to:

1. Update versions in runners.tf and lambdas/runners_lambdas.tf.
2. Change directory into lambdas and run terraform init and terraform apply. This will download the latest .zip files needed for the different lambdas.
3. Commit all the changed files.

Requirements

Name	Version
terraform	>= 1.3.0
aws	>= 4.0
random	>= 3.0

Providers

Name	Version
random	>= 3.0

Modules

Name	Source	Version
multi_runner	philips-labs/github-runner/aws//modules/multi-runner	5.10.4

Resources

Name	Type
random_id.webhook_secret	resource

Inputs

Name	Description	Type	Default	Required
aws_region	aws zone where to host the github actions runners	string	"eu-central-1"	no
dockerhub_credentials	DockerHub username and password so that the runner is will automatically be logged in to DockerHub and have increased rate limits	object({ username = string password = string })	null	no
github_app_key_base64	Github app private key. Ensure this value is the entire base64-encoded .pem file (e.g. the output of base64 app.private-key.pem), not its content.	string	n/a	yes
github_app_multirunner_id	id of the github app	string	n/a	yes
github_org	Name of the Github organization, owning the runners. Required only if specified with ephemeral runners	string	null	no
instance_allocation_strategy	allocation strategy for spot instances	string	"price-capacity-optimized"	no
log_retention_in_days	Specifies the number of days you want to retain log events for the lambda log group. Possible values are: 0, 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, and 3653.	number	7	no
runner_group_name	github actions runner group to attach the agents to	string	"Infrastructure-Repository-Deployment"	no
runner_iam_role_policy_arns	Attach AWS or customer-managed IAM policies (by ARN) to the runner IAM role	list(string)	[]	no
runner_log_files	Replaces the original module default cloudwatch log config. See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html for details.	list(object( { log_group_name = string prefix_log_group = bool file_path = string log_stream_name = string } ))	[ { "file_path": "/var/log/syslog", "log_group_name": "syslog", "log_stream_name": "{instance_id}", "prefix_log_group": true }, { "file_path": "/var/log/user-data.log", "log_group_name": "user_data", "log_stream_name": "{instance_id}/user_data", "prefix_log_group": true }, { "file_path": "/home/runners/actions-runner/diag/Runner**.log", "log_group_name": "runner", "log_stream_name": "{instance_id}/runner", "prefix_log_group": true } ]	no
runners	runners = { architecture: Must be either "x64" or "arm64" labels: List of extra labels to attach to the runner. "self-hosted", os and architecture labels are attached by default. Make sure this field is unique among the runners you host. idle_config: List of objects specifying the schedule for keeping runners idle/warm maximum_count: Number of maximum concurrent runners that can be spawned ephemeral: Boolean for selecting the type of runner use_spot_instances: Boolean for using spot EC2 instances instead of on-demand os: linux or windows. Operating system }	map(object({ architecture = string # x64 / arm64 labels = list(string) instance_types = list(string) idle_config = optional(list(object({ cron = optional(string, "* * 8-18 ? * 1-5") # cron schedule parsed by CronParser (used to keep idle runners up) poolCron = optional(string, "* 6-16 ? * Mon-Fri ") # AWS eventbridge cron schedule (used to keep runners pool up) timeZone = optional(string, "Europe/Zurich") # Applied to 'cron' only, not 'poolCron'. idleCount = optional(number, 1) })), [{ cron = " * 8-18 ? * 1-5" # Important to specify also the seconds or this won't work poolCron = "* 6-16 ? * Mon-Fri *" timeZone = "Europe/Zurich" idleCount = 1 }]) maximum_count = optional(number, 15) ephemeral = optional(bool, false) use_spot_instances = optional(bool, false) os = optional(string, "linux") # linux / windows base_ami = optional(string, "al2023") # amazonlinux2 / ubuntu / al2023 disk = optional(object({ throughput_mbps = optional(number) # between 125 and 750 volume_type = optional(string, "gp3") }), {}) }))	{ "runner-1": { "architecture": "x64", "instance_types": [ "c7a.xlarge", "c7i.xlarge", "c6a.xlarge", "c6i.xlarge" ], "labels": [ "multi-runner" ] } }	no
subnet_ids	The set of subnets where to deploy the runners	list(string)	n/a	yes
tags	Map of tags to apply to all resources deployed from the module	map(string)	{}	no
unique_prefix	The unique prefix used for naming AWS resources.	string	n/a	yes
userdata_post_install	Script to be ran after the GitHub Actions runner is installed on the EC2 instances	string	""	no
userdata_pre_install	Script to be ran before the GitHub Actions runner is installed on the EC2 instances	string	""	no
volume_size	EBS volume size mounted to runner instance	number	40	no
vpc_id	The vpc id where to deploy the runners	string	n/a	yes

Outputs

Name	Description
runner_iam_roles	Map of the IAM Roles used by the created runners
runner_labels	Map of the runner labels you can use in your jobs to select the runners
ssm_parameters	Names and ARNs of the ssm parameters created by the multi_runner module
webhook_endpoint	API gateway endpoint that handles GitHub App webhook events
webhook_secret	Webhook secret used to validate requests from Github. Use this as 'webhook secret' in the Github app.

Authors

Module is maintained by Alfredo Gottardo, David Beauvererd, Davide Cammarata, Demetrio Carrara and Roland Bapst

License

Apache 2 Licensed. See LICENSE for full details.