allow terraform to choose which .tf files to include / ignore

[FlorinAndrei]

By default, TF includes all *.tf files in the current directory. This is a bit rigid and makes it harder to add / remove parts of an environment dynamically.

Please allow the user to more easily select which *.tf files to include at runtime. There are many ways one could think of to accomplish this. Just a random example: allow an optional include_me statement in the file, with a value of either true (default) or false. Then the user could do things such as:

include_me "${var.include_api_cluster}"

Not sure if this is consistent with the current TF syntax, it's just an example.

[aholbreich]

+1. But another idea is to check if some custom.include is present to drop default behaviour (which is include everything) and read inclusion rules form that file...
Inclusin rules could be just line separatet filenames with wildcard support.

[FlorinAndrei]

@aholbreich I like your idea even better.

[apparentlymart]

Hi @FlorinAndrei! Thanks for this feature request.

I'd love to hear a bit more about the problem you're trying to solve here, since you hinted a bit by saying "add/remove parts of the environment dynamically" but I'm not sure I understand fully what you mean by "dynamically" here, since usually we think of dynamic behavior as having Terraform adjust part of the configuration depending on the outcome of another part (which isn't known until apply time), but I don't think that's what you're trying to do here since your proposal wouldn't address that need as far as I can tell.

Generally we try to encourage thinking of things at resource-level granularity rather than file-level granularity, since the decision for how to split things into files can be driven by lots of different motivations. For example, sometimes certain things are split into a separate file just because that part of the configuration happens to be generated by a pre-deploy script.

We currently have the -target argument which is intended to allow focusing a particular Terraform run on a subset of resources. It sounds like you'd like to be able to drive inclusions from a file on disk rather than from command line arguments. If there were a way to express targeting via an on-disk file rather than just on the command line, would that address the use-case you're trying to solve?

[FlorinAndrei]

Let's say I have a bunch of modules for prefab components. Stuff like a "network" module (including VPC, subnets, NAT, routes...), EC2 cluster module, RDS module, etc. Modules are all stashed in a separate top directory tree in the repo, and are source'd in my asset-building .tf files.

Built upon this, I have various environment templates. Let's say there's a QA env. It has a main.tf file that's sourcing the network module and some VPC peering modules and so on.

Then there's api.tf, calling the static EC2 cluster module (instances, block devices), some security group modules, some Route53 stuff, the works.

Then there's postgres.tf, calling the RDS module, with some security groups and DNS also.

Then there's redis.tf (similar as above), rabbitmq.tf, etc.

For a clean test, I need to tear down then rebuild the QA env. But there's no reason to tear down networking (VPC, subnets, NAT, routes). So main.tf can stay during planning. But I do need to tear down API, RDS, Redis, RMQ, etc. So all those .tf files need to go while I do the plan.

Currently I accomplish this via some awkward symlink create/delete in a toggle script, while api.tf, postgres.tf, etc are hidden in a subdirectory. But this seems fragile, it's repo-unfriendly with all those symlinks changing all the time, etc.

qa
├── backend.tf
├── disposable
│   ├── api.tf
│   ├── postgres.tf
│   ├── redis.tf
│   ├── rmq.tf
│   ├── speedy.tf
│   └── web.tf
├── init.sh
├── main.tf
├── plan
├── terraform.tfvars
└── toggle

It would be a lot better if I could just leave all .tf files in the current directory, and somehow tell terraform which ones to include and which ones to exclude, in a simple way. A foobar.include file, as proposed above, which could then be placed on the git ignore list, would be great. Now the Jenkins jobs could just tweak that one file on the fly, and dismantle/rebuild the environment with confidence.

Managing an include template file is easy, from bash here documents to Ansible templates and so on.

This is just an example, but really, the current aggressive "include all" policy seems rigid no matter what. Give some runtime leeway to the user, please.

[aholbreich]

@apparentlymart i have simmilar setup as @FlorinAndrei
Now i tried to work with destroy and -target argument and it was quite good. I also see your point about working on resource level not on file level. So indeed with the usage of target i saw some kind of dependency of destroyed things, that terraform cleverly resolved for me. Since several -target arguments are possible i'll try to work with that and maybe that is sufficient for the moment but can imaged this can be to messy if i need to list too much not dependent to each other resources by -target arguments

[apparentlymart]

Thanks for the additional context, @FlorinAndrei .

Some common solutions to what you're doing here are:

• Split the configuration you have into two parts, with one creating the more persistent resources and the other creating the more ephemeral resources. You can then apply and destroy each separately with its own lifecycle, perhaps using the terraform_remote_state data source to share ids from the former to the latter.
• Use terraform taint to instruct Terraform to replace a given specific resource on the next run.
• Use the -target argument to limit the scope of a destroy operation. For example, terraform plan -destroy -target=module.rabbitmq to plan to destroy only the resources in module.rabbitmq. Multiple -target arguments can be supplied to destroy more than one module at a time.

For the moment I'd recommend addressing such use-cases using wrapper scripts that encode the appropriate set of -target arguments for each of your scenarios, with your automation then running those scripts to ensure that only the desired resources get destroyed and recreated without any need to wrangle the filesystem at all.

There are some other issues open that should make this approach easier in future:

• Support recursive targeting on modules #9236 to make -target=module.foo also implicitly target any child module -target=module.foo.module.bar, which is not presently true.
• feature request: inverse targeting / exclude #2253 to allow "anti-targets" so that you could say "destroy everything except these resources..."

I believe that this, with the addition of the above fixes, should be able to address your needs, albeit in a different way than you imagined. We'd rather keep the current approach of thinking of things in a resource- and module-oriented way rather than a file-oriented way, since this gives more flexibility and allows Terraform to make better guarantees about properly handling dependencies.

I'm going to close this since I think what you need here can be addressed with existing features combined with work represented in other open issues. However, I do appreciate you taking the time to describe your use-case in detail, and we'll keep it in mind as we build on these existing features in future to see if we can make things smoother while remaining consistent with the existing model of Terraform.

Thanks!

[espoelstra]

terraform state list is a great way to see all the modules that are available as targets for terraform plan -target=module.name.

[juliovg]

you could use : terraform destroy -auto-approve -target=aws_instance.
to delete a resource, in my example i use it to destroy an ec2 instsance

[Skull0ne]

I know this issue is closed but the idea was great.

In my case I split my cluster configuration in several terraform files because I have things to deploy with packer/ansible before deploying the last part of my terraform configuration.

At the moment, the only I found to do it is by moving my configuration in another folder to deploy my first part, deploy packer/ansible, and then bring my configuration in my folder to finish the terraform deployment.

An option like -file first.tf -file second.tf could be great!

[ameyaagashe]

We really need to have the capability to plan and apply individual terraform tf files rather than resource level.
Help me address the below problem
Intergration_Env.tf
Performance_Env.tf
System_env.tf
Production_Env.tf

How to build individual environments using terrafom?

[kylejeske]

@ameyaagashe

If you see the documentation on their website, it speaks to how to use re-usable modules to handle different environments ...

In particular, organizations commonly want to create a strong separation between multiple deployments of the same infrastructure serving different development stages (e.g. staging vs. production) or different internal teams. In this case, the backend used for each deployment often belongs to that deployment, with different credentials and access controls. Named workspaces are not a suitable isolation mechanism for this scenario.

Instead, use one or more re-usable modules to represent the common elements, and then represent each instance as a separate configuration that instantiates those common elements in the context of a different backend. In that case, the root module of each configuration will consist only of a backend configuration and a small number of module blocks whose arguments describe any small differences between the deployments.

References:
https://www.terraform.io/docs/state/workspaces.html
https://www.terraform.io/docs/modules/index.html

My personal opinion is that workspaces (pretty much namespaces) seems more appropriate for this type of environment segmentation but, the documentation suggests otherwise. ¯\_(ツ)_/¯

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.