template_file: whether or not changes show up in plans feels inconsistent

[glasser]

Terraform Version

v0.7.0

Affected Resource(s)

Core (data sources)

Terraform Configuration Files

data template_file foo {
  template = "${bar}"
  vars {
    bar = "baz"
  }
}

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply
2. terraform plan

Expected Behavior

plan would only tell me about things that changed.

Actual Behavior

plan says this every time:

<= data.template_file.foo
    rendered: "<computed>"
    template: "${bar}"
    vars.%:   "1"
    vars.bar: "baz"

Reasoning

Yes, I get that data sources have a special format. But my project has a dozen or two resources that could be converted to data sources (template_files and a custom resource). And many of them have pretty verbose attributes. Having the complete contents of all of them print on every single terraform plan for the rest of time will make it hard to find the actual changes.

[apparentlymart]

Hi @glasser. Thanks for this report!

Is this problem unique to this particular configuration, or have you seen it happen with other data sources too?

I ask because I'm wondering if having that ${baz} interpolation in the Template is causing Terraform to think that this data resource has a computed configuration, which would then cause it to defer refresh until apply and include it in the plan.

It looks like the problem goes away if the interpolation in the template is escaped:

data "template_file" "foo" {
  template = "$${bar}"
  vars {
    bar = "baz"
  }
}

With the configuration written this way, the ${bar} interpolation gets handled by the template data source rather than by the main configuration parser, and thus Terraform sees it as immediately refreshable.

Does that solve the problem for you, or are you seeing similar issues with other data sources?

Thanks!

[glasser]

I can look more into this on Monday, but does anyone use template_file without computed attributes? I'm assuming without checking that ${file()} also counts as computed... Certainly, the identical template_file as a resource did not show up in the diff on every plan.

[glasser]

Also I'm moderately sure that even in my case, the interpolation is handled by template_file, since bar is not defined in any other context. Hmm.

[apparentlymart]

@glasser I suspect that the unescaped template is just working "by accident", due to Terraform failing to validate/error that the variable is undefined.

It seems like what's happening here is that Terraform is detecting ${bar} in the template attribute and thus the data source is deferred for refreshing until apply time, and thus shows up in the plan. Somehow (not sure how yet) Terraform is silently ignoring the erroneous ${bar} interpolation (which is not defined from Terraform's perspective) and leaving it in the attribute value, which then allows the template_file implementation to still "see" it and act on it.

I think the intended behavior here is that your original template would be an error (bar is not defined in Terraform's interpolation context).

If this is indeed the problem, and escaping the template interpolations is the solution, I think there's definitely some documentation work here (to mention in the template_file docs that the template should be escaped) and probably also some core Terraform work (to figure out why Terraform is allowing this and to make it actually be an error). I've optimistically added some tags to that effect but I'll re-tag if it turns out that there's a more general problem here.

[glasser]

@apparentlymart OK, I edited the title since it clearly seems to be hyperbolic. I agree that ensuring that the substitutions show up only after the initial round of substitutions seems to work better. And my theory that file() would trigger the same issue is wrong.

On the other hand, I'm now seeing problems that go in the opposite direction:

glasser@glasser-lyrid 0 /tmp/z $ cat x.tf
variable baz { }
data template_file foo {
  template = "$${bar}"
  vars {
    bar = "${var.baz}"
  }
}
glasser@glasser-lyrid 126 /tmp/z $ TF_VAR_baz=foo1 /usr/local/bin/terraform apply
data.template_file.foo: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
glasser@glasser-lyrid 0 /tmp/z $ grep rendered terraform.tfstate
                            "rendered": "foo1",
glasser@glasser-lyrid 0 /tmp/z $ TF_VAR_baz=foo2 /usr/local/bin/terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.

data.template_file.foo: Refreshing state...

No changes. Infrastructure is up-to-date. This means that Terraform
could not detect any differences between your configuration and
the real physical resources that exist. As a result, Terraform
doesn't need to do anything.
glasser@glasser-lyrid 0 /tmp/z $ TF_VAR_baz=foo2 /usr/local/bin/terraform apply
data.template_file.foo: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
glasser@glasser-lyrid 0 /tmp/z $ grep rendered terraform.tfstate
                            "rendered": "foo2",

So it seems like running tf plan doesn't show that the data source's value has changed even though it has. But maybe that's intentional — only resources are supposed to show up as changed?

[apparentlymart]

I think you're asking why running terraform plan (your second step) didn't show a diff saying that rendered would be re-computed on apply... if not, I apologize for misunderstanding and the rest of this comment probably won't help much, but let's see...

The reason you don't see any plan output for changes to a data block is that it was handled during the "refresh" phase, as you can see by this log line in all three of the commands you ran:

data.template_file.foo: Refreshing state...

The most significant difference between a data resource (a resource created with a data block) and a managed resource (a resource created with a resource block) is that the former is never "created"... it just exists immediately and refreshes in the initial run.

The exception to this rule is when the data resource's arguments interpolate values from managed resources that don't yet exist. In that case, we can't refresh the data until the resource has been created, which we signal in the plan with the "read" (<=) action as you saw:

<= data.template_file.foo
    rendered: "<computed>"
    template: "$${bar}"
    vars.%:   "1"
    vars.bar: "${aws_instance.example.id}"

In the above situation, the read action would appear only on the first run, since on subsequent runs we'd have a value for aws_instance.example.id already cached in the state.

These read actions are included in the plan in order to make the plan "complete": if you see a resource attribute in the plan that interpolates ${data.template_file.foo.rendered} then you can see in the plan what values that will be built from and thus decide if the plan is going to do what you want.

Terraform will only include the data resource in the plan if it depends on a resource that hasn't been created yet. Otherwise, it'll refresh it before the plan phase even begins, and shouldn't be mentioned in the plan at all.

I hope that clarifies the situation a little. In an early iteration of this feature we didn't ever include data reads in the plan, but we felt it was better to include it so that the user would be able to see the complete picture of what would happen once the plan is applied.

[glasser]

OK, that mostly makes sense to me. And when the top level interpolator saw ${bar} in my original example, it figured that since it didn't know what that meant, it might as well be the same as ${aws_instance.example.id}?

We've been using "inline templates" like this for a while but I guess we'll switch to either escaping the $ or using separate file()s. No big deal. And I guess that means this bug should be closed?

[apparentlymart]

I opened #8077 to capture the bug that this wasn't raised as an error. If we choose to update the docs about this gotcha then we could do it as part of that issue, so I'm going to close this one. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.