Remove need for "backend" block in Terraform modules.

[josh-padnick]

One of the most common issues that Terragrunt users experience is that the "topmost" module that is called from your terraform.tfvars file is expected to have a backend block:

terraform.tfvars:

terragrunt = {
  terraform {
    source = "/my-module/"
  }
}

key1 = val1
key2 = val2

/my-module/main.tf:

terraform {
  # The offending block. The configuration for this backend will be filled in by Terragrunt.
  backend "s3" {}
}

variable "key1" {}
variable "key2" {}
...

In theory, a module should be unopinionated about which backend it implements and the calling code should express this opinion. In practice, Terragrunt works by copying the module to a temp directory and filling in the vars automatically, so it needs a backend block to "hook into".

An alternative way to implement this is to auto-generate a boilerplate Terraform template that calls the original module, but @brikis98 had a concern that we'd be forever playing catch-up to frequently changing Terraform code.

I don't currently agree with that argument, but it's still non-trivial to update Terragrunt to auto-generate new code, so for now, I just wanted to record the issue and let users know this is just about the most common issue we encounter with users.

[conorgil]

@josh-padnick Glad you created an issue for this, as I am running into it while creating some example configurations for #169. The discussion came up in that thread in this comment, where @brikis98 suggested exposing variables that you can use to change the configuration of the S3 backend.

However, to make sure I follow your suggestion, you're saying that the larger problem is that my-module/main.tf defines a backend at all, right? If someone wanted to use the Atlas backend, for example, then they would not be able to use that module because it hard codes the S3 backend inline. I am running into this problem during local development of my Terraform modules because I want to use a local backend during dev (working in Terraform only land) and then the S3 backend when running in AWS (Terragrunt land). Since I am new to the tool, my workflow might be off, so feel free to suggest general improvements there.

Can you explain more your idea to "auto-generate a boilerplate Terraform template that calls the original module"?

Here are my thoughts on a potential solution (not sure if it is the same as your idea or not). Basically, I think that because the root terraform.tfvars file already defines the backend configuration, Terragrunt can use that configuration for all child modules automatically by generating a backend.tf file in the temporary directory in which Terraform actually runs. For example:

live/terraform.tfvars:

terragrunt = {
  terraform {
    # The root terraform.tfvars file configures the backend, which should
    # be used as the default for all child environments and modules 
    remote_state {
     backend = "s3"
     config {
        bucket = "terragrunt-examples-remote-state"
        key = "${path_relative_to_include()}/terraform.tfstate"
        region = "us-east-1"
        encrypt = true
        lock_table = "terragrunt-examples-lock-table"      
      } 
    }
  }
}

live/staging/my-module/terraform.tfvars:

terragrunt = {
  # Load the root terraform.tfvars file to get the default backend configuration
  include {
    path = "${find_in_parent_folders()}"
  }

  terraform {
    # The environment terraform.tfvars file tell Terragrunt 
    # where to load the Terraform source.
    source = "/my-module/"

   # If we defined a remote_state here, it would override the default
   # configuration defined in the root terraform.tfvars. For example,
   # we could switch backends entirely and have this module use
   # Consul instead of S3 for some reason, but it would still execute
   # the same exact Terraform module code.
  }
}

key1 = val1
key2 = val2

/my-module/main.tf:

# The actual Terraform module does not define a backend in its
# .tf files. This means that by default, it will use the local backend.
variable "key1" {}
variable "key2" {}
...

/var/.../terragrunt-download/.../backend.tf:

# Terragrunt could use the configuration that the user has already defined
# in the root terraform.tfvars file to generate this backend.tf file in the
# temporary directory in which Terraform actually runs.
# Notice that Terragrunt has already resolved the interpolation
# syntax and this file only contains final raw values.
# Terraform would concat this file with all other *.tf files in the directory
# during its run and use whichever backend was defined in this backend.tf file.
terraform {
  backend "s3" {
    bucket = "terragrunt-examples-remote-state"
    key = "staging/my-module/terraform.tfvars"
    region = "us-east-1"
    encrypt = true
    lock_table = "terragrunt-examples-lock-table"
  }
}

Currently, it looks like Terragrunt passes all of backend configuration via the CLI when calling Terraform. Thoughts on instead generating a backend.tf file in the temporary directory where Terraform actually executes?

[conorgil]

If this issue is solved, then I think it would also resolve #212

[brikis98]

@conorgil I think your understanding is spot on.

I think it's a reasonable approach, with two reservations:

1. If you're using a terraform { source = "..." } block in your Terragrunt configuration, Terraform checks out the code into a temporary folder. Generating a backend.tf file into that folder is not a big deal, as developers won't see it anyway (for the most part). But what if you don't have a terraform { source = "..." } block in your Terragrunt configuration, generating a backend.tf file into a version-controlled folder gets messy, as now you have this extra file that pops up every time you run terragrunt, you have to add the file to .gitignore, you have to teach developers to never edit it by hand, and so on.

2. Getting into the code generation business is messy in general:

   1. What if there is already a backend.tf? Instead, we probably have to either generate a randomly-named file or something odd that is unlikely to conflict, such as __backend__.tf.
   2. Before generating the file, we have to scan the user's code to detect if a backend {} block already exists and log a warning and/or do no code generation if one is found.
   3. If Terraform changes something about the backend block, such as adding new params to it or adding support for interpolation, the generated code is more likely to be backwards incompatible than passing command-line params.

Thoughts?

[conorgil]

@brikis98 Great questions. I'll think on this more. Initial questions/responses below:

1. Hrm. I had not considered the case where there would not be a terraform { source = "..." } block in the Terragrunt configuration. Would the *.tf files just be defined in the same directory as the terraform.tfvars file which contains the Terragrunt configuration? What would the use-case for something like that be?

2. "Code generation..."

   1. Yea, I thought about the naming issue. We could somehow generate a unique ID and incorporate that into the filename. Something like terragrunt_backend_<id>.tf and also include the classic autogenerated comment saying "This file is automatically generated by Terragrunt. Changes to this file will be overridden. Please see for more details" and link to some docs.

   2. I guess if we didn't do this and they already had a backend, crazy insane things would happen when Terraform tried to run. Hopefully, it would just error and shout about multiple backends. This would be a pain, but shouldn't be too bad to write. I think the UX gain for most Terragrunt users would warrant the effort.

   3. Why do you think that the generated code would be more backwards fragile than CLI flags?

      If new params are added to the backend block, then the user would first have to update the Terragrunt configuration to add the new params/keys. Then, Terragrunt could still just copy that stanza into the backend.tf file.

      If Terraform updates to support interpolation in the backend stanza, then Terragrunt would already run into problems because we support interpolation in remote_state stanza already and it is the same syntax, right? Not quite sure how we would solve that off hand, but I also don't immediately see how CLI flags would make it easier to support. Am I missing something?

[brikis98]

Would the *.tf files just be defined in the same directory as the terraform.tfvars file which contains the Terragrunt configuration? What would the use-case for something like that be?

Yep. This was actually the "normal" way of using Terragrunt for most of its life! The support for downloading source code into a temporary folder was only added part of the way through :)

Why do you think that the generated code would be more backwards fragile than CLI flags?

My hunch is that what we're seeing now of the backend block is just version 0.1, and that many changes will be coming to it over the next few releases, whereas the CLI, in general, is a bit more mature and stable. I don't have any data to back that up, of course, and during the Terraform 0.9 upgrade, both the CLI and the code had backwards incompatibilities, but that's my guess :)

[conorgil]

Yep. This was actually the "normal" way of using Terragrunt for most of its life! The support for downloading source code into a temporary folder was only added part of the way through :)

Always interesting learning about a project after it has gone through some iterations. I did not realize that was the "normal" way of using Terragrunt historically. Though, now that I stopped to think and remembered that remote state locking was one of (if not the) primary feature for Terragrunt, it does kinda make sense that it could have been used inline for that.

My initial thought is that the automatically generated file still seems like a good solution. It would be very easy to debug what Terragrunt is doing under the hood by inspecting that file. For users using remote source, they won't run into problems because of the temp directory. For users using local source, they would have to add it to their gitignore. This would be a pain for those users, but we could make the filename consistent to make it less painful.

Also, given that Terraform now natively supports remote state locking and Terragrunt has correspondingly removed support for remote state locking (delegating to Terraform), do you think that using Terragrunt with local source is still a popular use-case? The "best practice" concepts outlined in #169 all favor a directory hierarchy that uses remote source as far as I can tell. Do you think that it is a reasonable stance to just say that as the tool has grown over time, remote source has emerged as the recommended best practice, so that we don't have to worry about legacy use-cases in this circumstance? Given, we'd need to provide a reasonable transition period for those types of things normally, but because there is an easy alternative (update .gitignore) it hopefully wouldn't be too painful in this case.

My hunch is that what we're seeing now of the backend block is just version 0.1, and that many changes will be coming to it over the next few releases, whereas the CLI, in general, is a bit more mature and stable. I don't have any data to back that up, of course, and during the Terraform 0.9 upgrade, both the CLI and the code had backwards incompatibilities, but that's my guess :)

I agree that the backend will likely undergo lots of changes since it is new and Terraform seems pretty comfortable with breaking changes as long as they are documented well and users have a clear transition path. However, what I am not following is why the CLI and backend block would change independently. If the keywords in the backend block change, then the keys in the CLI would have to change too, right? Even if the CLI continues to follow the pattern of -backend-config=key=value, the keys would both change in the CLI and the backend block, right?

A good example of this happening just recently is actually hashicorp/terraform#14949, where they updated the S3 backend to replace the lock_table keyword with dynamodb_table. In the PR, they updated both the S3 backend and the CLI. The lock_table keyword is deprecated as of the next release and once it is removed, I would expect using it in the backend block as lock_table = my_table and also on the CLI as -backend-config=lock_table=my_table would both break at the same time.

Thoughts?

[brikis98]

Also, given that Terraform now natively supports remote state locking and Terragrunt has correspondingly removed support for remote state locking (delegating to Terraform), do you think that using Terragrunt with local source is still a popular use-case?

Yes, it is. Since Terraform's backend doesn't support interpolation, many people use Terragrunt to avoid having to copy/paste the same backend settings to dozens of models. The support for declaratively defining CLI commands (extra_arguments) and running Terraform against multiple modules in parallel (apply-all, destroy-all) are also used with the "normal" Terraform folder structure.

Even if the CLI continues to follow the pattern of -backend-config=key=value, the keys would both change in the CLI and the backend block, right?

Yes, but Terragrunt currently blindly copies all the keys in the config {} block to the init command. In other words, it's largely agnostic to what the keys or values are, so it's more likely to continue to work, even as Terraform changes those keys.

[conorgil]

Yes, it is.

Thanks for the background. Makes sense.

Yes, but Terragrunt currently blindly copies all the keys in the config {} block to the init command. In other words, it's largely agnostic to what the keys or values are, so it's more likely to continue to work, even as Terraform changes those keys.

That makes sense. The file solution should be agnostic in the same way. It would just copy over the keys from the terraform.tfvars file into the generated backend.tf file. However, given that the logic for the CLI flags already exists, that is definitely reason to not write new code for the sake of writing new code. If the file solution isn't definitively better, then leveraging existing code first makes sense.

If a user already has defined a backend stanza, the CLI approach would have to figure out how to handle that in the same way as the file approach, right? Are there any other shared pieces of work between the two approaches?

[brikis98]

The file solution should be agnostic in the same way. It would just copy over the keys from the terraform.tfvars file into the generated backend.tf file.

It's a bit more complicated. First, we'd have to check if the code already has a terraform block. There very well could be one to define, for example, the minimum Terraform version. If there already is one, it's not clear if it's OK to define another one? Would Terraform merge them? Overwrite? Throw an error? Second, if there is already a terraform block, we should check if it already defines a backend. If so, I suppose we do nothing? Failing those two, we generate the nested blocks and fill in the keys as passed by the user. That'll work OK until Terraform decides to change the backend block syntax...

[conorgil]

Fair point on the potential for multiple terraform blocks. Quick test on the following main.tf:

terraform {
  required_version = "> 0.9.0"
}

terraform {
  backend "s3" {
    bucket = "some-other-bucket"
    key    = "some-other-key"
  }
}

yields an error:

Error loading configuration: Error loading /some/path/main.tf: only one 'terraform' block allowed per module

So, that would seem to rule out the simple solution of just generating a backend.tf file, especially if the user has already defined a terraform block somewhere in their configuration. It doesn't make sense to generate a file in some cases and not others, to I think this is evidence to support the CLI flag approach.

However, I think that the CLI approach will require scanning the Terraform code as well.

The following main.tf file:

terraform { }

and then running the command:

terraform init -backend-config=bucket=some-command-line-bucket -backend-config=key=some-command-line-key -backend-config=region=us-east-1

works without error, but does not write anything to .terraform/terraform.tfstate file, so it is essentially a noop.

However, the following main.tf:

terraform {
  backend "s3" {
    bucket = "some-bucket"
    key    = "some-key"
  }
}

and then running the command:

terraform init -backend-config=bucket=some-command-line-bucket -backend-config=key=some-command-line-key -backend-config=region=us-east-1

works without error and writes the following backend information to .terraform/terraform.tfstate:

"backend": {
        "type": "s3",
        "config": {
            "bucket": "some-command-line-bucket",
            "key": "some-command-line-key",
            "region": "us-east-1"
        }
}

Therefore, I believe that Terraform requires a backend block to be defined in the configuration for it to write the backend settings to the .terraform/terraform.tfstate file and the CLI flags to have an impact. Also, note that the CLI flags overwrite the values of the backend block defined in the configuration, so we'll need to figure out how to handle that situation (error, warn user, automatically overwrite, etc).

Implementing a solution for this issue is definitely trickier than I initially thought. Glad we're talking it through and getting ideas and details written down to figure out the best solution (which I am unsure of at this point)!

[brikis98]

Error loading configuration: Error loading /some/path/main.tf: only one 'terraform' block allowed per module

Ah, good to know.

Therefore, I believe that Terraform requires a backend block to be defined in the configuration for it to write the backend settings to the .terraform/terraform.tfstate file and the CLI flags to have an impact.

Yes, it does, and it's in our documentation that you have to define a backend block yourself.

Implementing a solution for this issue is definitely trickier than I initially thought.

Indeed. The CLI solution isn't great, but it's workable for now. My vote is to wait and see how the backend stuff evolves in Terraform 0.10 before investing too much more.

[conorgil]

My vote is to wait and see how the backend stuff evolves in Terraform 0.10 before investing too much more.

Sounds good

[brikis98]

As of #302, Terragrunt now checks if a backend is defined, and exits with an error if it's missing. I hope that's a decent compromise solution for this.