Allow better control over targeted apply.

[bcsgh]

Current Terraform Version

0.13

Use-cases

During development or operation processes, it's not uncommon for reality to diverge from the associated TF config and, even when the config is the desired end state, it is often not possible move directly from the current state to that ends state. For example, there may be cases where ACLs are temporarily and/or experimentally modified in an effort to address an active incident. It may then be necessary to make updates to some of the resource in the TF config while leaving other parts divergent. For example, updating the things that ACL tweak is protecting so it can be tested before the ACL tweak is removed. Currently there are approximately three options on how to do this:

1. If the resource to be left alone are not dependencies of the ones to be updated; --target= can be used.
2. The TF config can be modified to remove the changes to be ignored (normally temporally and locally because the changes are not a desired end state).
3. The resources to be updated can be manually modified outside of TF until plan shows they match what it would do.

Case 1 only works in some cases. Cases 2 &3 are far from convenient.

---

Preemptive comment: the way that terraform and it's documentation is set up, it looks to me like the way the terraform developers expect it to be used in this case is that the hypothetical ACL change (and in fact 100% of changes) would have originally been done via TF. While I like that as an ideal, it's a rather Utopian restriction that don't match well with reality. As soon as there are things not managed by TF (or managed by TF configs that are made inaccessible by their owners) that falls apart. It also falls apart during the process of migrating things into TF. And when constraints outside the reach of TF's control intrude.

In short: it is unavoidable that terraform will have to interact with other sources of change and can not always do so by just overwriting them. [end rant]

---

Proposal

Add a flag, like --target=, but where only the listed resource will be updated. An un-listed resource will be left in a divergent state or, where that's not possible (e.g. the dependent resource or attribute does not yet exist), cause an error.

[lijok]

What you're describing is exactly what --target does
It only applies a change to the targeted resource
You might be targeting a module and therefore seeing a diff of more than 1 resource?

it looks to me like the way the terraform developers expect it to be used in this case is that the hypothetical ACL change (and in fact 100% of changes) would have originally been done via TF. While I like that as an ideal, it's a rather Utopian restriction that don't match well with reality

If you think this is unachievable in your current project, then you have a process problem. It is not only achievable to get to the point where 100% of your changes go through terraform, it is also fairly easy to get to that point, with enough buy-in from the business, and enough drive from the developers.
My advice would be to evaluate why you're in a position where you think it is not feasible for you to manage all of your infra changes through terraform, and to try and fix those issues

[bcsgh]

I've seen more than a few cases where --target does not work that way. One I even had to work around today where running terraform apply --target=aws_instance.foo for would try and update both aws_instance.foo and the security group it is in. I know there are diff in the security group and the should be removed, but they can't be removed at this time.

---

(Sorry for the door stopper of a response, but you touched on a meta-concept that's a bit of a soapbox of mine.)

Regarding having 100% of stuff go through TF; I would like you to be correct, but:

1. That's not your decision to make. It is mine. If I, as the owner of the managed infrastructure, decide that the best way to resole a specific situation is to make edits outside terraform, then I am correct and if terraform refuses to deal with that then it is wrong. Tools exist to support there users. Tools that get in the way of user get removed. (That remains true even if it I could chose to do it via TF and that was just as fast.)

2. In general; whenever I can use terraform, I do. And where I can't; I try to move as far in that direction as time, priorities and other constraints allow. My goal is to get 100% of resource, from a preexisting setup, into TF. Much of what I'm dealing with is complexities associated with that still being in progress. Until that is done, it's unavoidable for there to be things that aren't how I would like them. But I still need to deal with them and also get other work done.

3. It may not ever be possible to get 100% of resource managed by TF. That "buy-in from the business", and "drive from the developers" are rather precarious assumptions. There are some components that consist of complex configurations maintained by third-parties who have no interest in supporting TF. There are things maintained by other teams who don't see the value in TF and don't answer to me. There is the (thankfully for me, hypothetical) PHB who for political reasons wants to inflate the billable hours for maintenance of their part of the project. (In other words, even if your claim is correct in theory, practice merely likes your sense of humor.) The more the system you build requires full "buy-in" and total "drive", the more you are limiting the market for your product. If you have a product that provides value from the start with minimal buy-in and effort, and then shows the user the value for using it more? If the product and usage patter actually is worth doing, that should happen naturally. (If it doesn't happen naturally, then maybe it's not actually the best solution.)

As for reasons that "it's not feasible to manage all infra changes through terraform"; some of them include:

• terraform does not support importing some resource types and thus can't manage any of them that predate the use of terraform. (FWIW: I'm repeatedly considered hand editing .tfstates)

• terrafrom doesn't provide effective tools for fine grained control of updates. For example multi stage modifications where the existing and desired end state are known but where the question of which bits can be updated at want point in time is a complex and state full question (e.g. "ask Bob if he is using the whatchamacallit") that would be prohibitive to code into terraform, Particularity, given it will only be used during the transition and then thrown away. (The best solution I've found for this involves multiple git branches or clients, copious use of --target, aggressive "hack at the .tf till it does what I need it to" and judicious "just apply the edit by hand" when the first bits looks like they won't be worth the effort.)

• The cost benefit analysis often comes down on the side of not using terraform for some things: I can spend 3 seconds tweaking a setting in the WebUI I'm already looking at to fix the issue now, work around that deltas for a few hours and then back it out tomorrow by applying at head. Or I can spend a few minutes finding which config/module/file that resource is in, a few more editing it, apply that, commit and send it for code review, merge it, and then tomorrow revert it (including fixing the merge conflicts), commit, review etc. Literally 10-100x the cost. And until things are merged (and fetched/pulled) other peoples still need to deal with their version of the configs not matching reality in ways they must not "fix".

• There is also the issue of "what needs to be deployed" and "what should be deployed" not always being the same thing. Which should be checked in? If you commit to master what is deployed, then how do you check if there are differences? If you leave master as what you are moving towards, then how do you make any changes other than jumping all the way there? (Note this still exists even if 100% of changes are made via TF but potentially by different people working on different branches.

• Some resources are fundamentally outside my TF configs' control (e.g. the current version of software replaced by some third party). Say I build the config to lookup the version to deploy to an instance where my goal is to run the latest version. Do I make the config use that automatically (which causes everything to update imminently as soon as I try to change anything in the config, regardless of how related, after there is a release)? Or do I pin it to a specific version (which losses the ability for TF to describe how things are supposed to be)?

Basically all of those are solved or massively mitigated by treating a TF config as aspirational; as the end goal, and give terraform the feature set to allow the user to use it to move towards that goal in whatever controlled manor they deem best. Right now I can accomplish that, but the "feature set" mostly consist of abusing the tools: modifying the config to make it match the parts of reality I can't yet fix (with no intent of ever committing that), breaking spurious dependencies via terraform state rm $x; terraform import $x $x_id, etc.