Merge branch 'release/v0.7.0' into master

CHANGELOG.md

@@ -7,6 +7,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+
+## [0.7.0] - 2020-12-04
+### Changed
+- Small clarifications in the README #368 @lrytz
+
+### Added
+- Allow operator to pass in a list of managed IAM policy ARNs for the runner role #361 @jpalomaki
+- expand options for sourcing lambda to include S3 #292 @eky5006 
+
 ## [0.6.0] - 2020-10-10
 
 ### Added
@@ -30,6 +39,7 @@ terraform import module.runners.module.runners.aws_cloudwatch_log_group.scale_up
 terraform import module.runners.module.runners.aws_cloudwatch_log_group.scale_down "/aws/lambda/default-scale-down"
 terraform import module.runners.module.webhook.aws_cloudwatch_log_group.webhook "/aws/lambda/default-webhook"
 ```
+- feat: Expose ami-filters and user-data template file location to users to allow use of custom AMIs
 
 - feat: Added option to binaries syncer to upgrade to pre-releases, preventing any auto-updating on startup. Option `runner_allow_prerelease_binaries` is disabled by default. (#141, #165) @sjagoe
 
@@ -80,7 +90,8 @@ terraform import module.runners.module.webhook.aws_cloudwatch_log_group.webhook
 
 - First release.
 
-[unreleased]: https://github.com/philips-labs/terraform-aws-github-runner/compare/v0.6.0..HEAD
+[unreleased]: https://github.com/philips-labs/terraform-aws-github-runner/compare/v0.7.0..HEAD
+[0.7.0]: https://github.com/philips-labs/terraform-aws-github-runner/releases/tag/v0.6.0..v0.7.0
 [0.6.0]: https://github.com/philips-labs/terraform-aws-github-runner/releases/tag/v0.5.0..v0.6.0
 [0.5.0]: https://github.com/philips-labs/terraform-aws-github-runner/releases/tag/v0.4.0..v0.5.0
 [0.4.0]: https://github.com/philips-labs/terraform-aws-github-runner/releases/tag/v0.3.0..v0.4.0

CONTRIBUTING.md

@@ -52,6 +52,7 @@ If your issue appears to be a bug, and hasn't been reported, open a new issue. H
 **If you get help, help others. Good karma rulez!**
 
 ### Submitting a Merge Request
+
 Before you submit your merge request consider the following guidelines:
 
 * Make your changes in a new git branch:
@@ -63,6 +64,7 @@ Before you submit your merge request consider the following guidelines:
 * Create your patch, **including appropriate test cases**.
 * Run the test suite and ensure that all tests pass.
 * Add a line in the CHANGELOG.md under Unreleased. This will be used form generating the release notes.
+* Install [pre-commit hooks](https://pre-commit.com/). The hooks runs some basic checks and update the docs. The commit will run the hooks, you can invoke the hooks manually `pre-commit run --all-files` as well.
 * Commit your changes using a descriptive commit message.
 
     ```shell

examples/default/lambdas-download/main.tf

@@ -3,15 +3,15 @@ module "lambdas" {
   lambdas = [
     {
       name = "webhook"
-      tag  = "v0.6.0"
+      tag  = "v0.7.0"
     },
     {
       name = "runners"
-      tag  = "v0.6.0"
+      tag  = "v0.7.0"
     },
     {
       name = "runner-binaries-syncer"
-      tag  = "v0.6.0"
+      tag  = "v0.7.0"
     }
   ]
 }

examples/default/main.tf

@@ -45,4 +45,7 @@ module "runners" {
 
   # disable KMS and encryption
   # encrypt_secrets = false
+
+  # Let the module manage the service linked role
+  # create_service_linked_role_spot = true
 }

examples/permissions-boundary/lambdas-download/main.tf

@@ -3,15 +3,15 @@ module "lambdas" {
   lambdas = [
     {
       name = "webhook"
-      tag  = "v0.6.0"
+      tag  = "v0.7.0"
     },
     {
       name = "runners"
-      tag  = "v0.6.0"
+      tag  = "v0.7.0"
     },
     {
       name = "runner-binaries-syncer"
-      tag  = "v0.6.0"
+      tag  = "v0.7.0"
     }
   ]
 }

examples/ubuntu/README.md

@@ -0,0 +1,21 @@
+# Action runners deployment ubuntu example
+
+This modules shows how to create GitHub action runners using an Ubuntu AMI. Lambda release will be downloaded from GitHub.
+
+## Usages
+
+Steps for the full setup, such as creating a GitHub app can be found in the root module's [README](../../README.md). First download the Lambda releases from GitHub. Alternatively you can build the lambdas locally with Node or Docker, there is a simple build script in `<root>/.ci/build.sh`. In the `main.tf` you can simple remove the location of the lambda zip files, the default location will work in this case.
+
+```bash
+cd lambdas-download
+terraform init
+terraform apply
+cd ..
+```
+
+Before running Terraform, ensure the GitHub app is configured.
+
+```bash
+terraform init
+terraform apply
+```

examples/ubuntu/lambdas-download/main.tf

@@ -0,0 +1,21 @@
+module "lambdas" {
+  source = "../../../modules/download-lambda"
+  lambdas = [
+    {
+      name = "webhook"
+      tag  = "v0.5.0"
+    },
+    {
+      name = "runners"
+      tag  = "v0.5.0"
+    },
+    {
+      name = "runner-binaries-syncer"
+      tag  = "v0.5.0"
+    }
+  ]
+}
+
+output "files" {
+  value = module.lambdas.files
+}

examples/ubuntu/main.tf

@@ -0,0 +1,61 @@
+locals {
+  environment = "ubuntu"
+  aws_region  = "eu-west-1"
+}
+
+resource "random_password" "random" {
+  length = 28
+}
+
+module "runners" {
+  source = "../../"
+
+  aws_region = local.aws_region
+  vpc_id     = module.vpc.vpc_id
+  subnet_ids = module.vpc.private_subnets
+
+  environment = local.environment
+  tags = {
+    Project = "ProjectX"
+  }
+
+  github_app = {
+    key_base64     = var.github_app_key_base64
+    id             = var.github_app_id
+    client_id      = var.github_app_client_id
+    client_secret  = var.github_app_client_secret
+    webhook_secret = random_password.random.result
+  }
+
+  webhook_lambda_zip                = "lambdas-download/webhook.zip"
+  runner_binaries_syncer_lambda_zip = "lambdas-download/runner-binaries-syncer.zip"
+  runners_lambda_zip                = "lambdas-download/runners.zip"
+
+  enable_organization_runners = false
+  runner_extra_labels         = "ubuntu,example"
+
+  # enable access to the runners via SSM
+  enable_ssm_on_runners = true
+
+  userdata_template = "./templates/user-data.sh"
+  ami_owners        = ["099720109477"] # Canonical's Amazon account ID
+
+  ami_filter = {
+    name = ["ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"]
+  }
+
+  block_device_mappings = {
+    # Set the block device name for Ubuntu root device
+    device_name = "/dev/sda1"
+  }
+
+  # Uncommet idle config to have idle runners from 9 to 5 in time zone Amsterdam
+  # idle_config = [{
+  #   cron      = "* * 9-17 * * *"
+  #   timeZone  = "Europe/Amsterdam"
+  #   idleCount = 1
+  # }]
+
+  # disable KMS and encryption
+  # encrypt_secrets = false
+}

examples/ubuntu/outputs.tf

@@ -0,0 +1,12 @@
+output "runners" {
+  value = {
+    lambda_syncer_name = module.runners.binaries_syncer.lambda.function_name
+  }
+}
+
+output "webhook" {
+  value = {
+    secret   = random_password.random.result
+    endpoint = module.runners.webhook.endpoint
+  }
+}

examples/ubuntu/providers.tf

@@ -0,0 +1,4 @@
+provider "aws" {
+  region  = local.aws_region
+  version = "3.0"
+}

examples/ubuntu/templates/user-data.sh

@@ -0,0 +1,37 @@
+#!/bin/bash -e
+exec > >(tee /var/log/user-data.log | logger -t user-data -s 2>/dev/console) 2>&1
+
+# Install AWS CLI
+apt-get update
+DEBIAN_FRONTEND=noninteractive apt-get install -y awscli jq
+
+# Install runner
+cd /home/ubuntu
+mkdir actions-runner && cd actions-runner
+
+aws s3 cp ${s3_location_runner_distribution} actions-runner.tar.gz
+tar xzf ./actions-runner.tar.gz
+rm -rf actions-runner.tar.gz
+
+INSTANCE_ID=$(wget -q -O - http://169.254.169.254/latest/meta-data/instance-id)
+REGION=$(curl -s 169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region)
+
+echo wait for configuration
+while [[ $(aws ssm get-parameters --names ${environment}-$INSTANCE_ID --with-decryption --region $REGION | jq -r ".Parameters | .[0] | .Value") == null ]]; do
+    echo Waiting for configuration ...
+    sleep 1
+done
+CONFIG=$(aws ssm get-parameters --names ${environment}-$INSTANCE_ID --with-decryption --region $REGION | jq -r ".Parameters | .[0] | .Value")
+aws ssm delete-parameter --name ${environment}-$INSTANCE_ID --region $REGION
+
+export RUNNER_ALLOW_RUNASROOT=1
+
+sudo -u ubuntu mkdir /home/ubuntu/work
+
+./bin/installdependencies.sh
+./config.sh --unattended --name $INSTANCE_ID --work "/home/ubuntu/work" $CONFIG
+
+chown -R ubuntu:ubuntu .
+./svc.sh install ubuntu
+
+./svc.sh start

examples/ubuntu/variables.tf

@@ -0,0 +1,8 @@
+
+variable "github_app_key_base64" {}
+
+variable "github_app_id" {}
+
+variable "github_app_client_id" {}
+
+variable "github_app_client_secret" {}

examples/ubuntu/vpc.tf

@@ -0,0 +1,7 @@
+module "vpc" {
+  source = "git::https://github.com/philips-software/terraform-aws-vpc.git?ref=2.1.0"
+
+  environment                = local.environment
+  aws_region                 = local.aws_region
+  create_private_hosted_zone = false
+}

main.tf

@@ -5,6 +5,8 @@ locals {
 
   s3_action_runner_url = "s3://${module.runner_binaries.bucket.id}/${module.runner_binaries.runner_distribution_object_key}"
   runner_architecture  = substr(var.instance_type, 0, 2) == "a1" || substr(var.instance_type, 1, 2) == "6g" ? "arm64" : "x64"
+
+  ami_filter = length(var.ami_filter) > 0 ? var.ami_filter : local.runner_architecture == "arm64" ? { name = ["amzn2-ami-hvm-2*-arm64-gp2"] } : { name = ["amzn2-ami-hvm-2.*-x86_64-ebs"] }
 }
 
 resource "random_string" "random" {
@@ -38,9 +40,12 @@ module "webhook" {
   sqs_build_queue           = aws_sqs_queue.queued_builds
   github_app_webhook_secret = var.github_app.webhook_secret
 
-  lambda_zip                = var.webhook_lambda_zip
-  lambda_timeout            = var.webhook_lambda_timeout
-  logging_retention_in_days = var.logging_retention_in_days
+  lambda_s3_bucket                 = var.lambda_s3_bucket
+  webhook_lambda_s3_key            = var.webhook_lambda_s3_key
+  webhook_lambda_s3_object_version = var.webhook_lambda_s3_object_version
+  lambda_zip                       = var.webhook_lambda_zip
+  lambda_timeout                   = var.webhook_lambda_timeout
+  logging_retention_in_days        = var.logging_retention_in_days
 
   role_path                 = var.role_path
   role_permissions_boundary = var.role_permissions_boundary
@@ -62,10 +67,12 @@ module "runners" {
   s3_bucket_runner_binaries   = module.runner_binaries.bucket
   s3_location_runner_binaries = local.s3_action_runner_url
 
-  instance_type = var.instance_type
+  instance_type         = var.instance_type
+  block_device_mappings = var.block_device_mappings
 
   runner_architecture = local.runner_architecture
-  ami_filter          = local.runner_architecture == "arm64" ? { name = ["amzn2-ami-hvm-2*-arm64-gp2"] } : { name = ["amzn2-ami-hvm-2.*-x86_64-ebs"] }
+  ami_filter          = local.ami_filter
+  ami_owners          = var.ami_owners
 
   sqs_build_queue                 = aws_sqs_queue.queued_builds
   github_app                      = var.github_app
@@ -78,17 +85,25 @@ module "runners" {
   idle_config                     = var.idle_config
   enable_ssm_on_runners           = var.enable_ssm_on_runners
 
-  lambda_zip                = var.runners_lambda_zip
-  lambda_timeout_scale_up   = var.runners_scale_up_lambda_timeout
-  lambda_timeout_scale_down = var.runners_scale_down_lambda_timeout
-  logging_retention_in_days = var.logging_retention_in_days
+  lambda_s3_bucket                 = var.lambda_s3_bucket
+  runners_lambda_s3_key            = var.runners_lambda_s3_key
+  runners_lambda_s3_object_version = var.runners_lambda_s3_object_version
+  lambda_zip                       = var.runners_lambda_zip
+  lambda_timeout_scale_up          = var.runners_scale_up_lambda_timeout
+  lambda_timeout_scale_down        = var.runners_scale_down_lambda_timeout
+  logging_retention_in_days        = var.logging_retention_in_days
 
   instance_profile_path     = var.instance_profile_path
   role_path                 = var.role_path
   role_permissions_boundary = var.role_permissions_boundary
 
+  userdata_template     = var.userdata_template
   userdata_pre_install  = var.userdata_pre_install
   userdata_post_install = var.userdata_post_install
+
+  create_service_linked_role_spot = var.create_service_linked_role_spot
+
+  runner_iam_role_managed_policy_arns = var.runner_iam_role_managed_policy_arns
 }
 
 module "runner_binaries" {
@@ -103,9 +118,12 @@ module "runner_binaries" {
   runner_architecture              = substr(var.instance_type, 0, 2) == "a1" || substr(var.instance_type, 1, 2) == "6g" ? "arm64" : "x64"
   runner_allow_prerelease_binaries = var.runner_allow_prerelease_binaries
 
-  lambda_zip                = var.runner_binaries_syncer_lambda_zip
-  lambda_timeout            = var.runner_binaries_syncer_lambda_timeout
-  logging_retention_in_days = var.logging_retention_in_days
+  lambda_s3_bucket                = var.lambda_s3_bucket
+  syncer_lambda_s3_key            = var.syncer_lambda_s3_key
+  syncer_lambda_s3_object_version = var.syncer_lambda_s3_object_version
+  lambda_zip                      = var.runner_binaries_syncer_lambda_zip
+  lambda_timeout                  = var.runner_binaries_syncer_lambda_timeout
+  logging_retention_in_days       = var.logging_retention_in_days
 
   role_path                 = var.role_path
   role_permissions_boundary = var.role_permissions_boundary

modules/download-lambda/README.md

@@ -10,15 +10,15 @@ module "lambdas" {
   lambdas = [
     {
       name = "webhook"
-      tag  = "v0.6.0"
+      tag  = "v0.7.0"
     },
     {
       name = "runners"
-      tag  = "v0.6.0"
+      tag  = "v0.7.0"
     },
     {
       name = "runner-binaries-syncer"
-      tag  = "v0.6.0"
+      tag  = "v0.7.0"
     }
   ]
 }

modules/runner-binaries-syncer/README.md

@@ -51,6 +51,7 @@ No requirements.
 | aws\_region | AWS region. | `string` | n/a | yes |
 | distribution\_bucket\_name | Bucket for storing the action runner distribution. | `string` | n/a | yes |
 | environment | A name that identifies the environment, used as prefix and for tagging. | `string` | n/a | yes |
+| lambda\_s3\_bucket | S3 bucket from which to specify lambda functions. This is an alternative to providing local files directly. | `any` | `null` | no |
 | lambda\_schedule\_expression | Scheduler expression for action runner binary syncer. | `string` | `"cron(27 * * * ? *)"` | no |
 | lambda\_timeout | Time out of the lambda in seconds. | `number` | `300` | no |
 | lambda\_zip | File location of the lambda zip file. | `string` | `null` | no |
@@ -59,6 +60,8 @@ No requirements.
 | role\_permissions\_boundary | Permissions boundary that will be added to the created role for the lambda. | `string` | `null` | no |
 | runner\_allow\_prerelease\_binaries | Allow the runners to update to prerelease binaries. | `bool` | `false` | no |
 | runner\_architecture | The platform architecture for the runner instance (x64, arm64), defaults to 'x64' | `string` | `"x64"` | no |
+| syncer\_lambda\_s3\_key | S3 key for syncer lambda function. Required if using S3 bucket to specify lambdas. | `any` | `null` | no |
+| syncer\_lambda\_s3\_object\_version | S3 object version for syncer lambda function. Useful if S3 versioning is enabled on source bucket. | `any` | `null` | no |
 | tags | Map of tags that will be added to created resources. By default resources will be tagged with name and environment. | `map(string)` | `{}` | no |
 
 ## Outputs