ADF terraform extension

.gitignore

@@ -139,3 +139,15 @@ dmypy.json
 .pyre/
 
 megalinter-reports/
+
+### Terraform ###
+# Local .terraform directories
+**/.terraform/*
+
+# .tfstate files
+*.tfstate
+*.tfstate.*
+
+# Ignore CLI configuration files
+.terraformrc
+terraform.rc

docs/user-guide.md

@@ -26,6 +26,11 @@
     - [Deploying Serverless Applications with SAM](#deploying-serverless-applications-with-sam)
     - [Using YAML Anchors and Aliases](#using-yaml-anchors-and-aliases)
     - [One to many relationships](#one-to-many-relationships)
+  - [Terraform pipeline](#terraform-pipeline)
+    - [Prerequisites](#prerequisites)
+    - [Overview](#overview)
+    - [Parameters](#parameters)
+    - [Deployment procedure](#deployment-procedure)
 
 ## Deployment Map
 
@@ -1028,3 +1033,160 @@ By passing in the Repository name *(repository)* we are overriding the
 **name** property which normally is the name of our associated repository.
 This will tie both of these pipelines to the single *sample-vpc* repository on
 the `111111111111` AWS Account.
+
+### Terraform pipeline
+
+#### Prerequisites
+
+To enable ADF Terraform extension the following steps are required:
+- Enable ADF Terraform extension. Set the parameter
+  `extensions > terraform > enabled` to `True` in the `adfconfig.yml` file,
+  as shown in the `example-adfconfig.yml`, to deploy all the necessary
+  resources.
+- Rename file `example-global-iam.yml` to `global-iam.yml` in the following
+  path `aws-deployment-framework-bootstrap/adf-bootstrap/` and ensure the
+  CloudFormation resources `ADFTerraformRole` and `ADFTerraformPolicy` are
+  no longer commented out.
+- Rename file `example-global-iam.yml` to `global-iam.yml` in the following
+  path `aws-deployment-framework-bootstrap/adf-bootstrap/deployment`
+  **Please note:** the use of `deployment` at the end)
+  and ensure the CloudFormation resources `ADFTerraformRole` and
+  `ADFTerraformPolicy` are no longer commented out.
+
+**Important note**: `ADFTerraformPolicy` IAM policy is a sample.
+This policies should **NOT** be used for purposes other than testing.
+You should scope this policy depending on what you would like to deploy
+using Terraform within the selected Organizational Units.
+
+#### Overview
+
+ADF support the deployment of Terraform code to multiple accounts and
+regions through Terraform pipelines. The module consists of four build
+stages defined in the following CodeBuild build specification:
+
+- `buildspec.yml`: install the version of Terraform specified in the
+  pipeline configuration.
+- `tf_scan.yml`: (optional) scans for vulnerabilities in the Terraform
+  code using the [Terrascan](https://github.com/accurics/terrascan)
+  application. If vulnerabilities are found, it will fail and block
+  further execution in the pipeline. It is recommended to enable this
+  step in all ADF Terraform pipelines.
+- `tf_plan.yml`: get the list of accounts from the organization and
+  run a Terraform plan.
+- `tf_apply.yml`: get the list of accounts from the organization and
+  run a Terraform plan and apply.
+
+An optional approval step could be added between plan and apply as
+shown in the pipeline definition below.
+
+Please look into the [sample-terraform](../samples/sample-terraform)
+pipeline for more details in the setup and integration.
+
+#### Parameters
+
+- `TERRAFORM_VERSION`: the Terraform version used to deploy the
+  resource. This parameter must be defined in the `buildspec.yml`
+  file of the repository.
+- `TARGET_ACCOUNTS`: comma separated list of target accounts.
+- `TARGET_OUS`: comma separated list of target leaf OUs (parent
+  OUs are supported).
+- `REGIONS`: comma separated list of target regions. If this parameter
+  is empty, the main ADF region is used.
+- `MANAGEMENT_ACCOUNT_ID`: id of the AWS Organizations management account.
+
+#### Deployment procedure
+
+1. Add a sample-terraform pipeline in ADF `deployment-map.yml` as in the
+   example:
+
+```yaml
+- name: sample-terraform
+  default_providers:
+    source:
+      provider: codecommit
+      properties:
+        account_id: 111111111111  # Source account id
+    build:
+      provider: codebuild
+    deploy:
+      provider: codebuild
+      properties:
+        image: "STANDARD_5_0"
+        environment_variables:
+          TARGET_ACCOUNTS: 111111111111,222222222222  # Target accounts
+          TARGET_OUS: /core/infrastructure,/sandbox  # Target OUs
+          MANAGEMENT_ACCOUNT_ID: 333333333333  # Billing account
+          # Target regions, as a comma separated list is supported
+          # For example, "eu-west-1,us-east-1".
+          REGIONS: eu-west-1
+  targets:
+    - name: terraform-scan # optional
+      properties:
+        spec_filename: tf_scan.yml  # Terraform scan
+    - name: terraform-plan
+      properties:
+        spec_filename: tf_plan.yml  # Terraform plan
+    - approval # manual approval
+    - name: terraform-apply
+      properties:
+        spec_filename: tf_apply.yml  # Terraform apply
+```
+
+2. Add the project name in `params/global.yml` file.
+3. Add Terraform code to the `tf` folder. **Please note**: Do not make changes
+   to `backend.tf` file and `main.tf` in the root folder of the sample.
+   These contain the definition of the remote state file location and the
+   Terraform provider definition. Any change to these files could disrupt
+   the standard functionalities of this module.
+4. Add variable definition to `tf/variables.tf` file and variable values to
+   `tfvars/global.auto.tfvars`.
+
+   - Local variables (per account) can be configured using the following
+     naming convention
+
+     ```
+     tfvars <-- This folder contains the structure to define Terraform
+     │          variables
+     │
+     └───global.auto.tfvars <-- this file contains global variables applied to
+     │                          all the target accounts
+     │
+     └───111111111111 <-- this folders contains variable files related to
+     │   │                the account
+     │   └──────│   local.auto.tfvars <-- this file contains variables related
+     │          │                         to the account
+     │
+     └───222222222222
+         └──────│   local.auto.tfvars
+     ```
+
+5. Define the desired `TERRAFORM_VERSION` in the `buildspec.yml` file as shown
+   in the sample-terraform example. ADF supports Terraform version v0.13.0 and
+   later.
+6. Push to your Terraform ADF repository, for example the sample-terraform one.
+7. Pipeline contains a manual approval step between Terraform plan and
+   Terraform apply. Confirm to proceed.
+
+Terraform state files are stored in the regional S3 buckets in the deployment
+account. One state file per account/region/module is created.
+
+e.g.
+
+- Project name: sample-tf-module
+- Target accounts: 111111111111, 222222222222
+- Target regions: eu-west-1 (main ADF region), us-east-1
+
+The following state files are created:
+
+- 111111111111 main region (eu-west-1)
+  -> adf-global-base-deployment-pipelinebucketxyz/sample-tf-module/111111111111.tfstate
+- 111111111111 secondary region (us-east-1)
+  -> adf-regional-base-deploy-deploymentframeworkregio-jsm/sample-tf-module/111111111111.tfstate
+- 222222222222 main region (eu-west-1)
+  -> adf-global-base-deployment-pipelinebucketxyz/sample-tf-module/222222222222.tfstate
+- 222222222222 secondary region (us-east-1)
+  -> adf-regional-base-deploy-deploymentframeworkregio-jsm/sample-tf-module/222222222222.tfstate
+
+A DynamoDB table is created to manage the lock of the state file. It is
+deployed in every ADF regions named `adf_locktable`. **Please note**: usage
+of this locking table is charged on the deployment account.

samples/sample-terraform/README.md

@@ -1,19 +1,50 @@
-# Sample Terraform
+# Terraform template
 
-## Deployment Map example
+## Overview
+
+Please read the[user guide on ADF's support for
+Terraform](../docs/user-guide.md#terraform-pipeline) before you proceed.
+
+## Deployment procedure
+
+1. Add a sample-terraform pipeline in ADF `deployment-map.yml` as in the
+example:
 
 ```yaml
-  - name: my-terraform-example
-    default_providers:
-      source:
-        provider: codecommit
-        properties:
-          account_id: 1111111111111
-      deploy:
-        provider: codebuild
-        properties:
-          image: "STANDARD_5_0"
-    targets:
-      - properties:
-          spec_filename: my_test_spec.yml
+- name: sample-terraform
+  default_providers:
+    source:
+      provider: codecommit
+      properties:
+        account_id: 111111111111  # Source account id
+    build:
+      provider: codebuild
+    deploy:
+      provider: codebuild
+      properties:
+        image: "STANDARD_5_0"
+        environment_variables:
+          TARGET_ACCOUNTS: 111111111111,222222222222  # Target accounts
+          TARGET_OUS: /core/infrastructure,/sandbox  # Target OUs
+          MANAGEMENT_ACCOUNT_ID: 333333333333  # Billing account
+          # Regions in comma-separated list format, for example
+          # "eu-west-1,us-east-1"
+          REGIONS: eu-west-1
+  targets:
+    - name: terraform-scan # optional
+      properties:
+        spec_filename: tf_scan.yml  # Terraform scan
+    - name: terraform-plan
+      properties:
+        spec_filename: tf_plan.yml  # Terraform plan
+    - approval # manual approval
+    - name: terraform-apply
+      properties:
+        spec_filename: tf_apply.yml  # Terraform apply
 ```
+
+The sample uses the following configuration, please update accordingly:
+
+- Project name: sample-tf-module
+- Target accounts: 111111111111, 222222222222
+- Target regions: eu-west-1 (main ADF region), us-east-1

samples/sample-terraform/buildspec.yml

@@ -1,10 +1,19 @@
 version: 0.2
 
+env:
+  variables:
+    # Terraform version to use. ADF supports Terraform version v0.13.0 and later.
+    TERRAFORM_VERSION: "1.0.10"
 phases:
   install:
     commands:
+      - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
       - export PATH=$PATH:$(pwd)
-      - bash scripts/terraform/install_terraform.sh
+      - bash adf-build/helpers/terraform/install_terraform.sh
+      - pip install -r adf-build/requirements.txt -q
+  build:
+    commands:
+      - python adf-build/generate_params.py
 
 artifacts:
-  files: '**/*'
+  files: "**/*"

samples/sample-terraform/params/global.yml

@@ -0,0 +1,2 @@
+Parameters:
+  ProjectName: "sample-terraform" # $ADF_PROJECT_NAME

samples/sample-terraform/tf/main.tf

@@ -0,0 +1,16 @@
+data "aws_partition" "current" {}
+
+terraform {
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 3.0"
+    }
+  }
+  required_version = ">= 0.13.0"
+}
+provider "aws" {
+  assume_role {
+    role_arn     = "arn:${data.aws_partition.current}:iam::${var.TARGET_ACCOUNT_ID}:role/${var.TARGET_ACCOUNT_ROLE}"
+  }
+}

samples/sample-terraform/tf/s3.tf

@@ -0,0 +1,13 @@
+resource "aws_s3_bucket" "s3" {
+  bucket = "my-tf-test-bucket-${var.TARGET_REGION}-${var.TARGET_ACCOUNT_ID}"
+  acl    = "private"
+}
+
+resource "aws_s3_bucket_public_access_block" "s3-public-block" {
+  bucket = aws_s3_bucket.s3.id
+
+  block_public_acls   = true
+  block_public_policy = true
+  ignore_public_acls      = true
+  restrict_public_buckets = true
+}

samples/sample-terraform/variables.tf → samples/sample-terraform/tf/variables.tf

@@ -1,3 +1,4 @@
 variable "TARGET_ACCOUNT_ID" {}
 variable "TARGET_ACCOUNT_ROLE" {}
-variable my_bucket_name {}
+variable "TARGET_REGION" {}
+

samples/sample-terraform/tf_apply.yml

@@ -0,0 +1,18 @@
+version: 0.2
+
+env:
+  variables:
+    TF_VAR_TARGET_ACCOUNT_ROLE: adf-terraform-role # The IAM Role terraform will assume to deploy resources
+    TF_IN_AUTOMATION: true
+    TF_CLI_ARGS: "-no-color"
+    TF_STAGE: "apply"
+
+phases:
+  install:
+    runtime-versions:
+      python: 3.8
+  build:
+    commands:
+      - python adf-build/helpers/terraform/get_accounts.py
+      - bash adf-build/helpers/terraform/adf_terraform.sh
+