From 14e7a104a53a4b71030e30f1c4c35eb90e10a2dc Mon Sep 17 00:00:00 2001 From: Sergei Ivanov Date: Mon, 22 Feb 2021 21:38:24 +0000 Subject: [PATCH 1/2] feat: Add new hook for `terraform providers lock` operation Signed-off-by: Sergei Ivanov --- .pre-commit-hooks.yaml | 9 ++++ README.md | 75 +++++++++++++++++++++++++------ terraform_providers_lock.sh | 88 +++++++++++++++++++++++++++++++++++++ 3 files changed, 159 insertions(+), 13 deletions(-) create mode 100755 terraform_providers_lock.sh diff --git a/.pre-commit-hooks.yaml b/.pre-commit-hooks.yaml index d7b2d61d6..76339abcc 100644 --- a/.pre-commit-hooks.yaml +++ b/.pre-commit-hooks.yaml @@ -42,6 +42,15 @@ files: (\.tf|\.tfvars)$ exclude: \.terraform\/.*$ +- id: terraform_providers_lock + name: Lock terraform provider versions + description: Updates provider signatures in dependency lock files. + require_serial: true + entry: terraform_providers_lock.sh + language: script + files: (\.terraform\.lock\.hcl)$ + exclude: \.terraform\/.*$ + - id: terraform_tflint name: Terraform validate with tflint description: Validates all Terraform configuration files with TFLint. diff --git a/README.md b/README.md index 03fcf00a9..4881f249e 100644 --- a/README.md +++ b/README.md @@ -6,12 +6,12 @@ ### 1. Install dependencies -* [`pre-commit`](https://pre-commit.com/#install) -* [`terraform-docs`](https://github.com/terraform-docs/terraform-docs) required for `terraform_docs` hooks. `GNU awk` is required if using `terraform-docs` older than 0.8.0 with Terraform 0.12. -* [`TFLint`](https://github.com/terraform-linters/tflint) required for `terraform_tflint` hook. -* [`TFSec`](https://github.com/liamg/tfsec) required for `terraform_tfsec` hook. -* [`coreutils`](https://formulae.brew.sh/formula/coreutils) required for `terraform_validate` hook on macOS (due to use of `realpath`). -* [`checkov`](https://github.com/bridgecrewio/checkov) required for `checkov` hook. +* [`pre-commit`][1] +* [`terraform-docs`][2] required for `terraform_docs` hooks. `GNU awk` is required if using `terraform-docs` older than 0.8.0 with Terraform 0.12. +* [`TFLint`][3] required for `terraform_tflint` hook. +* [`TFSec`][4] required for `terraform_tfsec` hook. +* [`coreutils`][5] required for `terraform_validate` hook on macOS (due to use of `realpath`). +* [`checkov`][6] required for `checkov` hook. ##### MacOS @@ -70,20 +70,21 @@ There are several [pre-commit](https://pre-commit.com/) hooks to keep Terraform | ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- | | `terraform_fmt` | Rewrites all Terraform configuration files to a canonical format. | | `terraform_validate` | Validates all Terraform configuration files. | +| `terraform_providers_lock` | Updates provider signatures in [dependency lock files][7]. | | `terraform_docs` | Inserts input and output documentation into `README.md`. Recommended. | | `terraform_docs_without_aggregate_type_defaults` | Inserts input and output documentation into `README.md` without aggregate type defaults. | -| `terraform_docs_replace` | Runs `terraform-docs` and pipes the output directly to README.md (requires terraform-docs v0.10.0 or later) | -| `terraform_tflint` | Validates all Terraform configuration files with [TFLint](https://github.com/terraform-linters/tflint). | -| `terragrunt_fmt` | Rewrites all [Terragrunt](https://github.com/gruntwork-io/terragrunt) configuration files (`*.hcl`) to a canonical format. | -| `terragrunt_validate` | Validates all [Terragrunt](https://github.com/gruntwork-io/terragrunt) configuration files (`*.hcl`) | -| `terraform_tfsec` | [TFSec](https://github.com/liamg/tfsec) static analysis of terraform templates to spot potential security issues. | -| `checkov` | [checkov](https://github.com/bridgecrewio/checkov) static analysis of terraform templates to spot potential security issues. | +| `terraform_docs_replace` | Runs `terraform-docs` and pipes the output directly to README.md (requires [terraform-docs][2] v0.10.0 or later) | +| `terraform_tflint` | Validates all Terraform configuration files with [TFLint][3]. | +| `terragrunt_fmt` | Rewrites all [Terragrunt][8] configuration files (`*.hcl`) to a canonical format. | +| `terragrunt_validate` | Validates all [Terragrunt][8] configuration files (`*.hcl`) | +| `terraform_tfsec` | [TFSec][4] static analysis of terraform templates to spot potential security issues. | +| `checkov` | [checkov][6] static analysis of terraform templates to spot potential security issues. | Check the [source file](https://github.com/antonbabenko/pre-commit-terraform/blob/master/.pre-commit-hooks.yaml) to know arguments used for each hook. ## Notes about terraform_docs hooks -1. `terraform_docs` and `terraform_docs_without_aggregate_type_defaults` will insert/update documentation generated by [terraform-docs](https://github.com/terraform-docs/terraform-docs) framed by markers: +1. `terraform_docs` and `terraform_docs_without_aggregate_type_defaults` will insert/update documentation generated by [terraform-docs][2] framed by markers: ```txt @@ -203,6 +204,45 @@ if they are present in `README.md`. `terraform_validate` hook will try to reinitialize them before running `terraform validate` command. +## Notes about terraform_providers_lock hook + +1. The hook requires Terraform 0.14 or later. + +1. The hook invokes two operations that can be really slow: + `terraform init` (in case `.terraform` directory is not initialised) + and `terraform providers lock`. Both operations require downloading + data from remote Terraform registries, and not all of that + downloaded data or meta-data is currently being cached by Terraform. + +1. `terraform_providers_lock` supports custom arguments. + + Example: + + ```yaml + hooks: + - id: terraform_providers_lock + args: ['--args=-platform=windows_amd64'] + ``` + + In order to pass multiple args, try the following: + + ```yaml + - id: terraform_providers_lock + args: + - '--args=-platform=windows_amd64' + - '--args=-platform=darwin_amd64' + ``` + +1. It may happen that Terraform working directory (`.terraform`) already exists but is outdated + (e.g. not initialized modules, wrong version of Terraform, etc). + To solve this problem you can find and delete all `.terraform` directories in your repository using this command: + + ```shell + find . -type d -name .terraform -prune -print -exec rm -rf {} \; + ``` + + `terraform_providers_lock` hook will try to reinitialize them before running `terraform providers lock` command. + ## Notes for developers 1. Python hooks are supported now too. All you have to do is: @@ -218,3 +258,12 @@ This repository is managed by [Anton Babenko](https://github.com/antonbabenko) w ## License MIT licensed. See LICENSE for full details. + +[1]: https://pre-commit.com/#install +[2]: https://github.com/terraform-docs/terraform-docs +[3]: https://github.com/terraform-linters/tflint +[4]: https://github.com/liamg/tfsec +[5]: https://formulae.brew.sh/formula/coreutils +[6]: https://github.com/bridgecrewio/checkov +[7]: https://www.terraform.io/docs/cli/commands/providers/lock.html +[8]: https://github.com/gruntwork-io/terragrunt diff --git a/terraform_providers_lock.sh b/terraform_providers_lock.sh new file mode 100755 index 000000000..31ea63f78 --- /dev/null +++ b/terraform_providers_lock.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash + +set -eo pipefail + +main() { + initialize_ + parse_cmdline_ "$@" + terraform_providers_lock_ +} + +initialize_() { + # get directory containing this script + local dir + local source + source="${BASH_SOURCE[0]}" + while [[ -L $source ]]; do # resolve $source until the file is no longer a symlink + dir="$(cd -P "$(dirname "$source")" > /dev/null && pwd)" + source="$(readlink "$source")" + # if $source was a relative symlink, we need to resolve it relative to the path where the symlink file was located + [[ $source != /* ]] && source="$dir/$source" + done + _SCRIPT_DIR="$(dirname "$source")" + + # source getopt function + # shellcheck source=lib_getopt + . "$_SCRIPT_DIR/lib_getopt" +} + +parse_cmdline_() { + declare argv + argv=$(getopt -o a: --long args: -- "$@") || return + eval "set -- $argv" + + for argv; do + case $argv in + -a | --args) + shift + ARGS+=("$1") + shift + ;; + --) + shift + FILES=("$@") + break + ;; + esac + done +} + +terraform_providers_lock_() { + local -a paths + local index=0 + local file_with_path + + for file_with_path in "${FILES[@]}"; do + file_with_path="${file_with_path// /__REPLACED__SPACE__}" + + paths[index]=$(dirname "$file_with_path") + + ((index += 1)) + done + + local path_uniq + for path_uniq in $(echo "${paths[*]}" | tr ' ' '\n' | sort -u); do + path_uniq="${path_uniq//__REPLACED__SPACE__/ }" + + if [[ ! -d "${path_uniq}/.terraform" ]]; then + set +e + init_output=$(terraform -chdir="${path_uniq}" init -backend=false 2>&1) + init_code=$? + set -e + + if [[ $init_code != 0 ]]; then + echo "Init before validation failed: $path_uniq" + echo "$init_output" + exit 1 + fi + fi + + terraform -chdir="${path_uniq}" providers lock "${ARGS[@]}" + done +} + +# global arrays +declare -a ARGS +declare -a FILES + +[[ ${BASH_SOURCE[0]} != "$0" ]] || main "$@" From 25588b600d16283ddf4b06e3a7f550901a12c783 Mon Sep 17 00:00:00 2001 From: Sergey Ivanov Date: Mon, 4 Oct 2021 11:46:27 +0100 Subject: [PATCH 2/2] fix: resolve linting issues --- CHANGELOG.md | 2 +- README.md | 14 +++++++------- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 84166f8e3..2b0cd8b77 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -164,7 +164,7 @@ All notable changes to this project will be documented in this file. - fix: Change terraform_validate hook functionality for subdirectories with terraform files ([#100](https://github.com/antonbabenko/pre-commit-terraform/issues/100)) -### +### configuration for the appropriate working directory. diff --git a/README.md b/README.md index 4c7efdbb6..5a9d17dff 100644 --- a/README.md +++ b/README.md @@ -184,7 +184,7 @@ There are several [pre-commit](https://pre-commit.com/) hooks to keep Terraform | `terraform_docs_without_aggregate_type_defaults` | Inserts input and output documentation into `README.md` without aggregate type defaults. Hook notes same as for [terraform_docs](#terraform_docs) | | `terraform_docs` | Inserts input and output documentation into `README.md`. Recommended. [Hook notes](#terraform_docs) | | `terraform_fmt` | Rewrites all Terraform configuration files to a canonical format. [Hook notes](#terraform_docs) | -| `terraform_providers_lock` | Updates provider signatures in [dependency lock files](https://www.terraform.io/docs/cli/commands/providers/lock.html). [Hook notes](#terraform_providers_lock) +| `terraform_providers_lock` | Updates provider signatures in [dependency lock files](https://www.terraform.io/docs/cli/commands/providers/lock.html). [Hook notes](#terraform_providers_lock) | `terraform_tflint` | Validates all Terraform configuration files with [TFLint](https://github.com/terraform-linters/tflint). [Available TFLint rules](https://github.com/terraform-linters/tflint/tree/master/docs/rules#rules). [Hook notes](#terraform_tflint). | | `terraform_tfsec` | [TFSec](https://github.com/liamg/tfsec) static analysis of terraform templates to spot potential security issues. [Hook notes](#terraform_tfsec) | | `terraform_validate` | Validates all Terraform configuration files. [Hook notes](#terraform_validate) | @@ -330,7 +330,7 @@ Example: `terraform_validate` hook will try to reinitialize them before running `terraform validate` command. - **Warning:** If you use Terraform workspaces, DO NOT use this workaround ([details](https://github.com/antonbabenko/pre-commit-terraform/issues/203#issuecomment-918791847)). Wait to [`force-init`](https://github.com/antonbabenko/pre-commit-terraform/issues/224) option implementation + **Warning:** If you use Terraform workspaces, DO NOT use this workaround ([details](https://github.com/antonbabenko/pre-commit-terraform/issues/203#issuecomment-918791847)). Wait to [`force-init`](https://github.com/antonbabenko/pre-commit-terraform/issues/224) option implementation ### terraform_providers_lock @@ -341,11 +341,11 @@ Example: and `terraform providers lock`. Both operations require downloading data from remote Terraform registries, and not all of that downloaded data or meta-data is currently being cached by Terraform. - + 1. `terraform_providers_lock` supports custom arguments. Example: - + ```yaml hooks: - id: terraform_providers_lock @@ -353,14 +353,14 @@ Example: ``` In order to pass multiple args, try the following: - + ```yaml - id: terraform_providers_lock args: - '--args=-platform=windows_amd64' - '--args=-platform=darwin_amd64' - ``` - + ``` + 1. It may happen that Terraform working directory (`.terraform`) already exists but is outdated (e.g. not initialized modules, wrong version of Terraform, etc). To solve this problem you can find and delete all `.terraform` directories in your repository using this command: