Render templates in a different module to where they are defined

[allenhumphreys]

Current Terraform Version

v0.13.3

Use-cases

As part of our Azure configuration, we're making extensive use of API management's policies to perform various request validations. The policies are in XML with embedded C# code and they can be quite long. As part of our complex usage of the policies, we compose smaller policy snippets into complete policies at varying levels of the API management policy scopes. When devising the organization for such a complex set of templates, I encountered a situation where a 2 pass template processing would've been very helpful. Specifically, I wanted to have an initial templatefile step capable of outputting a string that would then go through another step of templating, thus templatestring.

Attempted Solutions

Proposal

Add templatestring function to allow modules to output a template to be further processed and fed into the input of a resource or another module.

main.tf

variable "input" {}

output "template" {
  value = templatefile("${module.path}/template.xml", { input = var.input })
}

module "with_template_output" {
  source = "git:repo/templates"
}

resource "example" "example" {
  input = templatestring(module.with_template_output.template, { additional_input = "substitute" })
}

References

[rvaidya]

I have a similar use case that the templatefile function doesn't cover and I need to continue using the template_file module data source. In my case, the string templates are not coming directly from a file - they are coming from another terraform provider. The "awkward escaping" mentioned in the template_file documentation is a valid concern, but there isn't any other way to achieve this functionality that I know of without using the template provider.

https://registry.terraform.io/providers/hashicorp/template/latest/docs/data-sources/file

Due to this, the deprecation of the template provider is a real problem as functionality is being removed without a templatestring function to replace it. This is what my use case looks like:

provider "kustomization" {
  kubeconfig_raw = templatefile("${path.module}/templates/${var.kubernetes_platform}/kubeconfig.tpl", var.kube_config)
}

data "kustomization_build" "current" {
  # path to kustomization directory
  path = var.manifest_path
}

data "template_file" "current_overlay" {
  for_each = data.kustomization_build.current.ids

  # Since kustomize templates can include configmaps with embedded bash scripts, we cannot replace
  # all ${VAR} template expressions, because this overlaps with bash's own variable expression.
  # Instead, we escape all template expressions where the variable name is not prefixed by an underscore.
  # ex: ${VARNAME} - treat as a bash variable and escape it
  # ex: ${_VARNAME} - treat as a terraform template variable, do not escape it, the var will be substituted
  template = replace(replace(
    data.kustomization_build.current.manifests[each.value], "/\\$\\{([^_][^}]*)\\}/", "$$$${$1}"
  ), "/%\\{([^}]*)}/", "%%%{$1}")

  vars = var.template_vars
}

resource "kustomization_resource" "current" {
  for_each = data.kustomization_build.current.ids
  manifest = data.template_file.current_overlay[each.value].rendered
}

[Nuru]

@apparentlymart Why is this not a no-brainer? template_file took a string, which was usually read from a file, and evaluated it as a template. templatefile conveniently combines the file read with the template processing, but internally it still reads the file, turns it into a string, and performs template processing on the string. What is the hold up with skipping the file read part and just taking a string input for the template?

[apparentlymart]

Hi all,

We have intentionally not implemented any new features that treat strings as templates because our experience with template_file was that it constantly confused people that it required providing a string using Terraform's template syntax but with the need to escape that syntax in order to pass it through Terraform's own template syntax handling.

I understand that the underlying use-case here is to have a way for a module to accept as an argument something that describes how to construct a result based on data that the module itself will provide and which isn't accessible to the module caller. We do want to meet that use-case, but want to meet it in a way that avoids the user experience hazard that template_file created, which likely means that such a feature would require allowing the module caller to write an expression or template using the normal Terraform syntax with no special escaping, but we do not yet have a full technical design for such a mechanism.

The template_file data source remains available to meet this less-common use-case in the mean time. Although it is archived and no longer developed, the existing releases of that provider remain available in the registry and we typically do not remove already-published providers from the registry except in response to significant issues such as security problems. Given that, I would encourage you to keep using template_file when you have this particular use-case of dynamically rendering a template provided from elsewhere, and thus avoid blocking on there being a design for the requirement I described in the previous paragraph.

The templatefile function, along with just writing template strings inline in your main module code, remain the recommended path for the more common situation where the template and the inputs to the template are both defined by the same author in the same module. The template_file data source remains just as the interim solution to this situation where the template is defined by a module other than the one that will ultimately render it.

[nitrocode]

Here is my understanding

• The template_file provider can still be used but it does not work on darwin arm (M1 apple laptops)
• The template_file provider is archived so the provider cannot be updated to include darwin arm support
• The hashicorp team refuses to add the templatestring function because of the syntax escaping e.g. $${bucket_name} which is then replaced by a map key of bucket_name.

Our temporary solution now is to use multiple replace functions

provider "aws" {
  region = "us-east-1"
}

data "aws_iam_policy_document" "policy" {
  statement {
    actions = ["s3:GetObject"]
    resources = [
      "arn:aws:s3:::$${bucket_name}",
      "arn:aws:s3:::$${bucket_name}/*",
      "arn:aws:s3:::$${origin_path}/*",
      "arn:aws:s3:::$${something_else}/*",
    ]

    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::0123456789:root"]
    }
  }
}

locals {
  bucket_name    = "bucket_name_world"
  origin_path    = "origin_path_hello"
  something_else = "something_else_foobar"
}

output "override_policy" {
  value = replace(replace(replace(data.aws_iam_policy_document.policy.json,
    "$${bucket_name}", local.bucket_name),
    "$${origin_path}", local.origin_path),
    "$${something_else}", local.something_else)
}

Which returns

$ terraform plan
Changes to Outputs:
  + override_policy = jsonencode(
        {
          + Statement = [
              + {
                  + Action    = "s3:GetObject"
                  + Effect    = "Allow"
                  + Principal = {
                      + AWS = "arn:aws:iam::0123456789:root"
                    }
                  + Resource  = [
                      + "arn:aws:s3:::something_else_foobar/*",
                      + "arn:aws:s3:::origin_path_hello/*",
                      + "arn:aws:s3:::bucket_name_world/*",
                      + "arn:aws:s3:::bucket_name_world",
                    ]
                  + Sid       = ""
                },
            ]
          + Version   = "2012-10-17"
        }
    )

[allenhumphreys]

@nitrocode It sounds like the hashicorp team is acknowledging the need but don't have a clear solution yet, not necessarily refusing to do anything. They're making a fair user experience argument, even though adding templatestring seems like relatively low hanging fruit.

When I opened this ticket I did not know about the template_file provider, but when I found it I still avoided it because it is marked as deprecated.

It does really suck that it is not being supported but still being suggested as a work around for this gap.

[Nuru]

@apparentlymart @allenhumphreys I believe nitrocode's point is that the archived template_file provider is not a sufficient workaround because it does not run on Apple's new hardware. If Hashicorp wants to cut a new release of it that works on the M1 chip, then we can go back to using it, but being archived and deprecated, it does not look like that is happening.

[apparentlymart]

Indeed, it's not clear whether the template provider will ever see new releases for platforms that didn't exist when it was last released. For the moment Terraform CLI itself is not officially released for darwin_arm64 either, so that gap in support is not super relevant yet, but you're all right that it does increase the need for a new design to replace this last bit of functionality of template_file.

As @allenhumphreys noted, my answer above was not to say that we don't intend to design an implement a new mechanism for this, but rather that our experience with constant user friction with template_file tells us that templatestring (or similar) will not be a suitable solution for this use-case, and so instead we'll seek other designs that hopefully lead to better user experience tradeoffs while still meeting the main use-cases.

One thing that template_file can do in principle that I expect we won't aim to replicate in a new design is any means to dynamically generate a template from data gathered at runtime. Instead, the focus is on the more constrained use-case of situations where the template is defined in a different place to where it's evaluated, but yet the template is still statically defined with the normal Terraform string template syntax. Based on the use-cases we've seen for this so far, that seems like the best compromise to creating something which addresses most situations but without any need for unusual nested escaping or generating string templates from other string templates.

[adamdonahue]

Our use case is as follows. A user can specify an input variable which is UserData for an EC2 instance, and then leverage Terraform interpolation to bring in the relevant variables, for example, to mount a Terraform-managed EFS filesystem on instance startup. We might do this, for example:

launch_templates = {
  "foo" = {
    name = "foo"
    userdata = <<-EOF
      # Add EFS mount for shared log files.
      echo "${aws_efs_file_system.file_system["foo"].id} ..." >> /etc/fstab
    EOF
  }
}

It seems that even with templatefile this isn't easy, since you can't run templatefile across the entire output/data/variable space by default, and thus we'd also have to know upfront which variables are being used, and dynamically interpolate those into a map before calling templatefile. It starts getting messy, and I'm not even sure that's possible.

I do suspect that the order of interpolation as well as when templates can be evaluated is some of the complication, and that we may need to move to specific aws_launch_template resources rather than a general one we for_each over with a map, but that's unfortunate. The latter use case has served us nicely to date.

TL;DR -- We need some kind of Terraform eval expression, essentially.

[alamothe]

Second day using Terraform and I need this function...

[betancourtl]

I was hoping there was a solution for this... I just started using terraform.

[nitrocode]

An alternative is to use the copied template provider with an arm release. This has been done with the following provider and published to the registry.

https://registry.terraform.io/providers/cloudposse/template/latest
https://registry.terraform.io/providers/gxben/template/latest

You can achieve this by creating a new file to pin the provider.

Using cloudposse

# versions.tf
terraform {
  required_providers {
    template = {
      source  = "cloudposse/template"
      version = "~> 2.2.0"
    }
  }
}

Using gxben

# versions.tf
terraform {
  required_providers {
    template = {
      source  = "gxben/template"
      version = "= 2.2.0-m1"
    }
  }
}

e.g.

# main.tf
provider "template" {}
 
data "template_file" "greeting" {
  template = "my name is $${name}"
  vars = {
    name = "bob"
  }
}

# outputs.tf
output "greeting" {
  value = data.template_file.greeting.rendered
}

returns

Changes to Outputs:
  + greeting = "my name is bob"

More info: https://gist.github.com/nitrocode/cf40a24054a66afe2b19ca54b7be5d68

[allenhumphreys]

Since I’ve opened this issue I’ve gotten a lot more experience with Terraform. It’s interesting that I’ve actually never managed to run into the hard edge again and it’s fascinating that so many people new to terraform are ending up here.

I am looking forward to the improvements to templating flexibility that are coming because I think it’ll definitely unlock more patterns. But, I don’t currently find myself needing this, even for the thing that originally caused me to open the issue.

[MrE194]

Our use case is as follows:

We use a child module to create AWS API gateways (aws_api_gateway_rest_api). Within this resource is a parameter that contains the OpenAPI specification for setting up the gateway. As part of our plan/apply, our root module reads the documentation endpoint that the application exposes (that is, the app that lives behind the APIGW) and passes the result on to the child APIGW module.

Our methodology has a number of variable substitutions that need to be done to populate certain elements of the spec file, depending on which of our AWS accounts the deployment is being done in and which app environment we're talking about (volatile, perf testing, prod, and so forth). This is done by having the spec file get returned as a TF-friendly template, which is then populated with those necessary values by means of the template provider.

Why don't we just populate those values in the root module by pulling the spec, saving it as a temporary file, and then using templatefile() to render it for passing on to the child module? Some of them are generated within the child APIGW based on those account/environment-specific settings, and they're common among our various APIGW projects. Pulling those out into the root modules un-DRYs things that we centralized into the child module, because they're common among all of our projects.

[xophere]

An additional use case here is I only want to update the objects if a file has changed. So I create a data object by reading the file. A terraform_data resource to trigger the lifecycle on. But then because I can't process against a stream I have to reread the file. Seems like a bad design.

[joaocc]

We have a similar case to @MrE194, where we are working on a wrapper module to https://github.com/aws-ia/terraform-aws-eks-blueprints-addon where we receive a list of values (YAML strings) which we want to process as templates, injecting values (in this case IRSA names that are only known at the level of our module) and not at the level of whoever generates the templates.
With all due respect to all the good and hard work on terraform, I feel that the decision not to provide it intentionally can be seen as a bit too overprotective (if some users feel confused by the need to double-escape strings, then no one will be able to use it), instead of allowing people to decide for themselves whether the tradeoffs are worth the risk/effort/impact... From the comment, it also didn't seem to be due to any technical restriction or bring any benefit anything for the whole community.
This decision forces users to create even more contrived solutions (custom search-replace as pointed above probably works but is considerably less maintainable and more brittle) due to loss of language capabilities over time.
I am saying this from a point of admiration and love for the language and the tool, but which also makes me want to o make it even better and more useful.

[nitrocode]

@joaocc does #26838 (comment) unblock you?

[joaocc]

@nitrocode that is indeed the workaround we ended up using.
As much as I appreciate the fact that someone took the time to provide a good alternative, it is still a decrease in capabilities from the point of view of terraform, and forcing people to spend time on tracking alternatives without any discernible benefit based on the rationale presented above.

[crw]

Thank you for your continued interest in this issue.

Terraform version 1.8 launches with support of provider-defined functions. It is now possible to implement your own functions! We would love to see this implemented as a provider-defined function.

Please see the provider-defined functions documentation to learn how to implement functions in your providers. If you are new to provider development, learn how to create a new provider with the Terraform Plugin Framework. If you have any questions, please visit the Terraform Plugin Development category in our official forum.

We hope this feature unblocks future function development and provides more flexibility for the Terraform community. Thank you for your continued support of Terraform!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.