diff --git a/README.md b/README.md
index 82c04dad6..8aad4b9c2 100644
--- a/README.md
+++ b/README.md
@@ -39,16 +39,16 @@ If you are using `pre-commit-terraform` already or want to support its developme
* [All hooks: Usage of environment variables in `--args`](#all-hooks-usage-of-environment-variables-in---args)
* [All hooks: Set env vars inside hook at runtime](#all-hooks-set-env-vars-inside-hook-at-runtime)
* [All hooks: Disable color output](#all-hooks-disable-color-output)
- * [checkov (deprecated) and terraform_checkov](#checkov-deprecated-and-terraform_checkov)
- * [infracost_breakdown](#infracost_breakdown)
- * [terraform_docs](#terraform_docs)
- * [terraform_docs_replace (deprecated)](#terraform_docs_replace-deprecated)
- * [terraform_fmt](#terraform_fmt)
- * [terraform_providers_lock](#terraform_providers_lock)
- * [terraform_tflint](#terraform_tflint)
- * [terraform_tfsec](#terraform_tfsec)
- * [terraform_validate](#terraform_validate)
- * [terraform_wrapper_module_for_each](#terraform_wrapper_module_for_each)
+ * [checkov (deprecated) and terraform\_checkov](#checkov-deprecated-and-terraform_checkov)
+ * [infracost\_breakdown](#infracost_breakdown)
+ * [terraform\_docs](#terraform_docs)
+ * [terraform\_docs\_replace (deprecated)](#terraform_docs_replace-deprecated)
+ * [terraform\_fmt](#terraform_fmt)
+ * [terraform\_providers\_lock](#terraform_providers_lock)
+ * [terraform\_tflint](#terraform_tflint)
+ * [terraform\_tfsec](#terraform_tfsec)
+ * [terraform\_validate](#terraform_validate)
+ * [terraform\_wrapper\_module\_for\_each](#terraform_wrapper_module_for_each)
* [terrascan](#terrascan)
* [tfupdate](#tfupdate)
* [Docker Usage: File Permissions](#docker-usage-file-permissions)
@@ -70,7 +70,7 @@ If you are using `pre-commit-terraform` already or want to support its developme
* [`TFLint`](https://github.com/terraform-linters/tflint) required for `terraform_tflint` hook.
* [`TFSec`](https://github.com/liamg/tfsec) required for `terraform_tfsec` hook.
* [`infracost`](https://github.com/infracost/infracost) required for `infracost_breakdown` hook.
-* [`jq`](https://github.com/stedolan/jq) required for `infracost_breakdown` hook.
+* [`jq`](https://github.com/stedolan/jq) required for `terraform_validate` with `--retry-once-with-cleanup` flag, and for `infracost_breakdown` hook.
* [`tfupdate`](https://github.com/minamijoyo/tfupdate) required for `tfupdate` hook.
* [`hcledit`](https://github.com/minamijoyo/hcledit) required for `terraform_wrapper_module_for_each` hook.
@@ -228,7 +228,7 @@ pre-commit run -a
Or, using Docker ([available tags](https://github.com/antonbabenko/pre-commit-terraform/pkgs/container/pre-commit-terraform/versions)):
-**NOTE:** This command uses your user id and group id for the docker container to use to access the local files. If the files are owned by another user, update the `USERID` environment variable. See [File Permissions section](#docker-usage-file-permissions) for more information.
+> Note: This command uses your user id and group id for the docker container to use to access the local files. If the files are owned by another user, update the `USERID` environment variable. See [File Permissions section](#docker-usage-file-permissions) for more information.
```bash
TAG=latest
@@ -248,7 +248,7 @@ There are several [pre-commit](https://pre-commit.com/) hooks to keep Terraform
| Hook name | Description | Dependencies
[Install instructions here](#1-install-dependencies) |
-| ------------------------------------------------------ |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
+| ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| `checkov` and `terraform_checkov` | [checkov](https://github.com/bridgecrewio/checkov) static analysis of terraform templates to spot potential security issues. [Hook notes](#checkov-deprecated-and-terraform_checkov) | `checkov`
Ubuntu deps: `python3`, `python3-pip` |
| `infracost_breakdown` | Check how much your infra costs with [infracost](https://github.com/infracost/infracost). [Hook notes](#infracost_breakdown) | `infracost`, `jq`, [Infracost API key](https://www.infracost.io/docs/#2-get-api-key) |
| `terraform_docs` | Inserts input and output documentation into `README.md`. Recommended. [Hook notes](#terraform_docs) | `terraform-docs` |
@@ -258,11 +258,11 @@ There are several [pre-commit](https://pre-commit.com/) hooks to keep Terraform
| `terraform_providers_lock` | Updates provider signatures in [dependency lock files](https://www.terraform.io/docs/cli/commands/providers/lock.html). [Hook notes](#terraform_providers_lock) | - |
| `terraform_tflint` | Validates all Terraform configuration files with [TFLint](https://github.com/terraform-linters/tflint). [Available TFLint rules](https://github.com/terraform-linters/tflint/tree/master/docs/rules#rules). [Hook notes](#terraform_tflint). | `tflint` |
| `terraform_tfsec` | [TFSec](https://github.com/aquasecurity/tfsec) static analysis of terraform templates to spot potential security issues. [Hook notes](#terraform_tfsec) | `tfsec` |
-| `terraform_validate` | Validates all Terraform configuration files. [Hook notes](#terraform_validate) | - |
+| `terraform_validate` | Validates all Terraform configuration files. [Hook notes](#terraform_validate) | `jq`, only for `--retry-once-with-cleanup` flag |
| `terragrunt_fmt` | Reformat all [Terragrunt](https://github.com/gruntwork-io/terragrunt) configuration files (`*.hcl`) to a canonical format. | `terragrunt` |
| `terragrunt_validate` | Validates all [Terragrunt](https://github.com/gruntwork-io/terragrunt) configuration files (`*.hcl`) | `terragrunt` |
| `terraform_wrapper_module_for_each` | Generates Terraform wrappers with `for_each` in module. [Hook notes](#terraform_wrapper_module_for_each) | `hcledit` |
-| `terrascan` | [terrascan](https://github.com/tenable/terrascan) Detect compliance and security violations. [Hook notes](#terrascan) | `terrascan` |
+| `terrascan` | [terrascan](https://github.com/tenable/terrascan) Detect compliance and security violations. [Hook notes](#terrascan) | `terrascan` |
| `tfupdate` | [tfupdate](https://github.com/minamijoyo/tfupdate) Update version constraints of Terraform core, providers, and modules. [Hook notes](#tfupdate) | `tfupdate` |
@@ -274,8 +274,9 @@ Check the [source file](https://github.com/antonbabenko/pre-commit-terraform/blo
> All, except deprecated hooks: `checkov`, `terraform_docs_replace`
-You can use environment variables for the `--args` section.
-Note: You _must_ use the `${ENV_VAR}` definition, `$ENV_VAR` will not expand.
+You can use environment variables for the `--args` section.
+
+> **Warning**: You _must_ use the `${ENV_VAR}` definition, `$ENV_VAR` will not expand.
Config example:
@@ -476,7 +477,7 @@ Unlike most other hooks, this hook triggers once if there are any changed files
- --args=--config=.terraform-docs.yml
```
- Note: Avoid use `recursive.enabled: true` in config file, that can cause unexpected behavior.
+ > **Warning**: Avoid use `recursive.enabled: true` in config file, that can cause unexpected behavior.
5. If you need some exotic settings, it can be done too. I.e. this one generates HCL files:
@@ -517,7 +518,7 @@ Example:
1. The hook requires Terraform 0.14 or later.
2. The hook invokes two operations that can be really slow:
- * `terraform init` (in case `.terraform` directory is not initialised)
+ * `terraform init` (in case `.terraform` directory is not initialized)
* `terraform providers lock`
Both operations require downloading data from remote Terraform registries, and not all of that downloaded data or meta-data is currently being cached by Terraform.
@@ -652,7 +653,31 @@ Example:
- --tf-init-args=-lockfile=readonly
```
-3. It may happen that Terraform working directory (`.terraform`) already exists but not in the best condition (eg, not initialized modules, wrong version of Terraform, etc.). To solve this problem, you can find and delete all `.terraform` directories in your repository:
+3. It may happen that Terraform working directory (`.terraform`) already exists but not in the best condition (eg, not initialized modules, wrong version of Terraform, etc.). To solve this problem, you can delete broken `.terraform` directories in your repository:
+
+ **Option 1**
+
+ ```yaml
+ - id: terraform_validate
+ args:
+ - --hook-config=--retry-once-with-cleanup=true # Boolean. true or false
+ ```
+
+ > Note: The flag requires additional dependency to be installed: `jq`.
+
+ If `--retry-once-with-cleanup=true`, then in each failed directory the cached modules and providers from the `.terraform` directory will be deleted, before retrying once more. To avoid unnecessary deletion of this directory, the cleanup and retry will only happen if Terraform produces any of the following error messages:
+
+ * "Missing or corrupted provider plugins"
+ * "Module source has changed"
+ * "Module version requirements have changed"
+ * "Module not installed"
+ * "Could not load plugin"
+
+ **Warning:** When using `--retry-once-with-cleanup=true`, problematic `.terraform/modules/` and `.terraform/providers/` directories will be recursively deleted without prompting for consent. Other files and directories will not be affected, such as the `.terraform/environment` file.
+
+ **Option 2**
+
+ An alternative solution is to find and delete all `.terraform` directories in your repository:
```bash
echo "
@@ -666,22 +691,23 @@ Example:
`terraform_validate` hook will try to reinitialize them before running the `terraform validate` command.
- **Warning:** If you use Terraform workspaces, DO NOT use this workaround ([details](https://github.com/antonbabenko/pre-commit-terraform/issues/203#issuecomment-918791847)). Wait to [`force-init`](https://github.com/antonbabenko/pre-commit-terraform/issues/224) option implementation.
+ **Warning:** If you use Terraform workspaces, DO NOT use this option ([details](https://github.com/antonbabenko/pre-commit-terraform/issues/203#issuecomment-918791847)). Consider the first option, or wait for [`force-init`](https://github.com/antonbabenko/pre-commit-terraform/issues/224) option implementation.
4. `terraform_validate` in a repo with Terraform module, written using Terraform 0.15+ and which uses provider `configuration_aliases` ([Provider Aliases Within Modules](https://www.terraform.io/language/modules/develop/providers#provider-aliases-within-modules)), errors out.
When running the hook against Terraform code where you have provider `configuration_aliases` defined in a `required_providers` configuration block, terraform will throw an error like:
- >
- >
+
> Error: Provider configuration not present
- > To work with its original provider configuration at provider["registry.terraform.io/hashicorp/aws"]. is required, but it has been removed. This occurs when a provider configuration is removed while
- > objects created by that provider still exist in the state. Re-add the provider configuration to destroy , after which you can remove the provider configuration again.
+ > To work with `` its original provider configuration at provider `["registry.terraform.io/hashicorp/aws"].` is required, but it has been removed. This occurs when a provider configuration is removed while
+ > objects created by that provider still exist in the state. Re-add the provider configuration to destroy ``, after which you can remove the provider configuration again.
This is a [known issue](https://github.com/hashicorp/terraform/issues/28490) with Terraform and how providers are initialized in Terraform 0.15 and later. To work around this you can add an `exclude` parameter to the configuration of `terraform_validate` hook like this:
+
```yaml
- id: terraform_validate
exclude: '^[^/]+$'
```
+
This will exclude the root directory from being processed by this hook. Then add a subdirectory like "examples" or "tests" and put an example implementation in place that defines the providers with the proper aliases, and this will give you validation of your module through the example. If instead you are using this with multiple modules in one repository you'll want to set the path prefix in the regular expression, such as `exclude: modules/offendingmodule/[^/]+$`.
Alternately, you can use [terraform-config-inspect](https://github.com/hashicorp/terraform-config-inspect) and use a variant of [this script](https://github.com/bendrucker/terraform-configuration-aliases-action/blob/main/providers.sh) to generate a providers file at runtime:
@@ -699,6 +725,7 @@ Example:
```
Save it as `.generate-providers.sh` in the root of your repository and add a `pre-commit` hook to run it before all other hooks, like so:
+
```yaml
- repos:
- repo: local
@@ -715,7 +742,7 @@ Example:
[...]
```
- **Note:** The latter method will leave an "aliased-providers.tf.json" file in your repo. You will either want to automate a way to clean this up or add it to your `.gitignore` or both.
+ > Note: The latter method will leave an "aliased-providers.tf.json" file in your repo. You will either want to automate a way to clean this up or add it to your `.gitignore` or both.
### terraform_wrapper_module_for_each
@@ -761,7 +788,7 @@ If the generated name is incorrect, set them by providing the `module-repo-short
See the `terrascan run -h` command line help for available options.
-2. Use the `--args=--verbose` parameter to see the rule ID in the scaning output. Usuful to skip validations.
+2. Use the `--args=--verbose` parameter to see the rule ID in the scanning output. Useful to skip validations.
3. Use `--skip-rules="ruleID1,ruleID2"` parameter to skip one or more rules globally while scanning (e.g.: `--args=--skip-rules="ruleID1,ruleID2"`).
4. Use the syntax `#ts:skip=RuleID optional_comment` inside a resource to skip the rule for that resource.
diff --git a/hooks/_common.sh b/hooks/_common.sh
index 0e9454458..6bc867965 100644
--- a/hooks/_common.sh
+++ b/hooks/_common.sh
@@ -299,13 +299,15 @@ function common::terraform_init {
TF_INIT_ARGS+=("-no-color")
fi
- if [ ! -d .terraform ]; then
+ if [ ! -d .terraform/modules ] || [ ! -d .terraform/providers ]; then
init_output=$(terraform init -backend=false "${TF_INIT_ARGS[@]}" 2>&1)
exit_code=$?
if [ $exit_code -ne 0 ]; then
common::colorify "red" "'terraform init' failed, '$command_name' skipped: $dir_path"
echo -e "$init_output\n\n"
+ else
+ common::colorify "green" "Command 'terraform init' successfully done: $dir_path"
fi
fi
diff --git a/hooks/terraform_validate.sh b/hooks/terraform_validate.sh
index cd23a52a8..3e4694855 100755
--- a/hooks/terraform_validate.sh
+++ b/hooks/terraform_validate.sh
@@ -24,6 +24,43 @@ function main {
common::per_dir_hook "$HOOK_ID" "${#ARGS[@]}" "${ARGS[@]}" "${FILES[@]}"
}
+#######################################################################
+# Run `terraform validate` and match errors. Requires `jq`
+# Arguments:
+# validate_output (string with json) output of `terraform validate` command
+# Outputs:
+# Returns integer:
+# - 0 (no errors)
+# - 1 (matched errors; retry)
+# - 2 (no matched errors; do not retry)
+#######################################################################
+function match_validate_errors {
+ local validate_output=$1
+
+ local valid
+ local summary
+
+ valid=$(jq -rc '.valid' <<< "$validate_output")
+
+ if [ "$valid" == "true" ]; then
+ return 0
+ fi
+
+ # Parse error message for retry-able errors.
+ while IFS= read -r error_message; do
+ summary=$(jq -rc '.summary' <<< "$error_message")
+ case $summary in
+ "missing or corrupted provider plugins") return 1 ;;
+ "Module source has changed") return 1 ;;
+ "Module version requirements have changed") return 1 ;;
+ "Module not installed") return 1 ;;
+ "Could not load plugin") return 1 ;;
+ esac
+ done < <(jq -rc '.diagnostics[]' <<< "$validate_output")
+
+ return 2 # Some other error; don't retry
+}
+
#######################################################################
# Unique part of `common::per_dir_hook`. The function is executed in loop
# on each provided dir path. Run wrapped tool with specified arguments
@@ -43,16 +80,68 @@ function per_dir_hook_unique_part {
local -a -r args=("$@")
local exit_code
- local validate_output
+ #
+ # Get hook settings
+ #
+ local retry_once_with_cleanup
+
+ IFS=";" read -r -a configs <<< "${HOOK_CONFIG[*]}"
+
+ for c in "${configs[@]}"; do
+
+ IFS="=" read -r -a config <<< "$c"
+ key=${config[0]}
+ value=${config[1]}
+
+ case $key in
+ --retry-once-with-cleanup)
+ if [ $retry_once_with_cleanup ]; then
+ common::colorify "yellow" 'Invalid hook config. Make sure that you specify not more than one "--retry-once-with-cleanup" flag'
+ exit 1
+ fi
+ retry_once_with_cleanup=$value
+ ;;
+ esac
+ done
common::terraform_init 'terraform validate' "$dir_path" || {
exit_code=$?
return $exit_code
}
- # pass the arguments to hook
- validate_output=$(terraform validate "${args[@]}" 2>&1)
- exit_code=$?
+ if [ "$retry_once_with_cleanup" != "true" ]; then
+ # terraform validate only
+ validate_output=$(terraform validate "${args[@]}" 2>&1)
+ exit_code=$?
+ else
+ # terraform validate, plus capture possible errors
+ validate_output=$(terraform validate -json "${args[@]}" 2>&1)
+ exit_code=$?
+
+ # Match specific validation errors
+ local -i validate_errors_matched
+ match_validate_errors "$validate_output"
+ validate_errors_matched=$?
+
+ # Errors matched; Retry validation
+ if [ "$validate_errors_matched" -eq 1 ]; then
+ common::colorify "yellow" "Validation failed. Removing cached providers and modules from \"$dir_path/.terraform\" directory"
+ # `.terraform` dir may comprise some extra files, like `environment`
+ # which stores info about current TF workspace, so we can't just remove
+ # `.terraform` dir completely.
+ rm -rf .terraform/{modules,providers}/
+
+ common::colorify "yellow" "Re-validating: $dir_path"
+
+ common::terraform_init 'terraform validate' "$dir_path" || {
+ exit_code=$?
+ return $exit_code
+ }
+
+ validate_output=$(terraform validate "${args[@]}" 2>&1)
+ exit_code=$?
+ fi
+ fi
if [ $exit_code -ne 0 ]; then
common::colorify "red" "Validation failed: $dir_path"