Mechanisms to allow for future language extension

[apparentlymart]

The Terraform language has always had a number of situations where reserved words defined in the language can appear in the same location as externally-defined names provided by either providers or modules. These are:

• In provider, resource, and data blocks Terraform overlays reserved meta-arguments onto the relevant provider schema, effectively masking any provider-defined argument with a conflicting name.
• In module blocks Terraform overlays reserved meta-arguments onto the set of input variable names defined in the module. Currently we block modules from declaring such conflicting names, but that means that adding any new meta-arguments here is a breaking change for existing modules, as we previously saw when we reserved count in the v0.12 release.
• In provisioner blocks Terraform overlays reserved meta-arguments onto the provisioner configuration schema, likewise masking any provisioner-defined argument with a conflicting name.
• References to managed resources use the resource type name with no prefix, like aws_instance.foo, but certain prefixes are reserved for other purposes (such as var., local., path., etc) which means that a managed resource of that name would not be referenceable.

Each of these situations is a liability for future language expansion because it makes any additional reserved word a potential breaking change. While we would always do research to select a name not used in any commonly-used providers or modules, there are lots of providers and modules kept private within organizations and so there is always a risk that we'd pick a name that collides with a provider or module used only within a single organization.

In #27941 we reserved a mechanism which we intend to use in future to allow per-module opt-in to new "editions" of the Terraform language that may have breaking changes such as these, which does at least allow for a gradual module-by-module migration up to a new edition. However, our intent is that there should usually be an largely-automated process for upgrading a module to a newer edition without changing its behavior, and that means that we must provide a way to continue using module and provider features that have been newly-masked by reserved words until there's a newer version of the provider or module that supports a non-reserved alternative name.

This PR, then, proposes two new language additions that will together serve as an escape hatch for all of the situations described earlier, so that if we do introduce a new reserved word that is in use by a provider or module we cannot see we can offer a graceful migration path to still opt in to the new language edition without having to wait for the provider or module to be updated.

The resource. Prefix

When we refer to data resources, we write data.type.name which means that there's no ambiguity that the second component type is always the name of a data resource type. The resource. prefix aims to get the same result for managed resources, so that resource.aws_instance.foo is just an equivalent alias for aws_instance.foo as long as aws_instance is never a reserved word (which would be weird!).

My intent is that we'd use this as part of the module upgrade tool for any new language edition which introduces a new reserved reference prefix. For example, if we'd introduced a new prefix foo. then the upgrade tool would search the module for any references that seem to refer to a resource type called "foo" and rewrite them to have the resource. prefix, leaving all other references unchanged. For example, foo.bar.baz would be come resource.foo.bar.baz, but aws_instance.foo would remain just aws_instance.foo.

I don't intend to recommend using this prefix in cases where it isn't needed, because it'll make such references unnecessarily long. Although I didn't implement this here yet, I could imagine a future version of terraform fmt automatically trimming off a resource. prefix for any reference where the resource type name isn't a reserved word in the currently-selected language edition, to help reinforce that this is for upgrade path use only.

Meta-argument Escaping Blocks

The idea of "escaping blocks" aims to address all of the variants above which involve a mixture of meta-arguments and normal arguments inside a single HCL block body. This applies to resource, data, provider, and module blocks at the top level, and also to provisioner blocks nested directly inside resource blocks.

This mechanism reserves the special and intentionally-odd-looking block type name _ (just a single underscore) to contain block body items for which the author intends to avoid interpretation as meta-arguments. For example, if a later language edition introduced a new meta-argument only_if (though there is no current plan to do so), it might be the case that there's an existing resource type or module that already has an argument named only_if which could be "escaped" using an escaping block, like this:

resource "example" "foo" {
  # most arguments stay out here to be
  # processed as normal
  beep = "boop"

  _ {
    # Arguments that conflict with a later-added
    # meta-argument can move in here to
    # retain the old interpretation.
    only_if = var.something
  }
}

Terraform interprets the content of the escaping block as if it were written outside of the escaping block except for bypassing the usual pre-processing to handle meta-arguments, and thus only_if above would always be understood as an argument of the resource type example rather than as a meta-argument.

Again, my intent is that we'd use this as part of a module upgrade tool for a new language edition which introduces a new meta-argument. It would search the module configuration for any argument names that are now shadowed by the new reserved word and move them into an escaping block to allow the module to retain its existing functionality.

I don't intend to recommend using escaping blocks in situations where they aren't needed, because it's an intentionally-odd-looking syntax to try to help make it obviously distinct from "normal" blocks. Although I didn't implement this here yet, I could imagine a future version of terraform fmt automatically hoisting items out of an escaping block if they don't conflict with any relevant meta-argument names in the currently-selected language edition, to help reinforce that this is for upgrade path use only.

It's interesting to note here that dynamic blocks actually already serve as a funny sort of escaping block which works only for reserved nested block types: a resource block containing a dynamic "lifecycle" block can produce an equivalent effect as an escaping block with a plain lifecycle block inside of it. Escaping blocks generalize that idea to include attributes too, and they offer a syntax that is easier to directly map from the normal syntax (retaining attached comments and all) in an upgrade tool.

---

Some Other Loose Ends

Over in #28700 there's an early proposal for a new feature to allow separating template definition from template evaluation, which if accepted would require adding a new reserved reference prefix. It seems unlikely that we'll fully accept and implement that proposal before we adopt a stricter compatibility promise, but it would also be a shame to hold it all the way to a new language edition that's likely at least a year away, and so out of a sense of pragmatism here I've also pre-emptively reserved some prefixes that seem unlikely to already be used as resource type names and that might be what we choose as the prefix for the template proposal, if accepted.

These reserved prefixes are template., lazy., and arg.. Only zero or one of these will ultimately be used, depending on the outcome of the proposal, so the unused ones can be un-reserved later and become available for use as unescaped resource type names again.

These additions also serve as a way to "rehearse" the use of the resource. escaping prefix to resolve any collisions: in the unlikely event that there's a private provider out there which has a resource type name which conflicts with one of these three, a module using that resource type can be rewritten to say e.g. resource.template. instead of just template. and otherwise retain the existing behavior. Since we are prior to stricter compatibility promises there is no explicit opt-in here, but I'd like to make this tradeoff as a compromise to allow a discussion already in progress to conclude relatively promptly, rather than retroactively become subject to a new blocker on its implementation.

The resource. prefix here does, of course, also theoretically conflict with a resource type literally named "resource". That also seems very unlikely in practice, but if it does arise then users can escape it by writing resource.resource., which looks silly but will buy some time to select a more reasonable resource type name without being blocked from upgrading Terraform.

Nothing in this PR actually requires the addition of meta-argument escaping blocks yet -- they could in principle be added by the first future language edition that needs them -- but having this mechanism in place will give us some peace of mind that this path is available and also represent concretely in code our intentions for future language expansion.

It's pretty irregular to introduce stuff like this in a patch release, but with stricter compatibility promises imminent I think it's a worthwhile compromise to give us necessarily flexibility for both work currently in progress and future work we've not yet imagined. Therefore I'm proposing to backport this to the v0.15 branch.

[alisdair]

These mechanisms all make sense to me! Thanks for the split into multiple commits.

[apparentlymart]

Backported to v0.15:

• 8c559b7
• ad3e99b
• db43867

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.