[WIP] Safety features for Terraform State #6540
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced May 8, 2016
apparentlymart
force-pushed
the
f-state-safety
branch
from
f66f674 to
de1310d
Compare
apparentlymart
force-pushed
the
f-state-safety
branch
8 times, most recently
from
b32e4a8 to
ee28ef8
Compare
apparentlymart
changed the title
|
@phinze I think this is ready for review now. This is the initial set of "state safety" changes we discussed a couple weeks ago for possible inclusion in 0.7. |
|
This is amazing! Would love to see this merged soon 🎉 |
This was referenced May 19, 2016
apparentlymart
changed the title
|
This is now broken by the change from #6811. An entirely different approach will be needed for preventing the application of stale plans now that planning doesn't actually update the persistent state. |
The lineage of a state is an identifier shared by a set of states whose serials are meaningfully comparable because they are produced by progressive Refresh/Apply operations from the same initial empty state. This is initialized as a type-4 (random) UUID when a new state is initialized and then preserved on all other changes. Since states before this change will not have lineage but users may wish to set a lineage for an existing state in order to get the safety benefits it will grow to imply, an empty lineage is considered to be compatible with all lineages.
After running "terraform plan -out=tfplan" and then "terraform apply tfplan" the plan file is left on disk and could previously potentially be applied a second time. Here we add a new constraint that prevents the use of a plan that was not produced from the current state, thus avoiding that problem. It will also reduce race conditions (on the human timescale) between running "plan" and later running "apply", in environments where multiple people/processes are using Terraform with the same remote state. This hazard cannot be eliminated entirely without proper locking, but the with this change in place the race condition is only for two concurrent *applies*, as opposed to overlapping of the whole time period between plan and apply.
Accidentally losing a state with resources in it can be anywhere from annoying to catastrophic. A common cause of such a problem is accidentally clobbering a remote state with an unrelated local state or vice-versa. Here we introduce safety checks that exploit the new "lineage" concept to ensure that once a state location is established it can only be updated with new states from the same lineage as the initial state. We also make the remote state caching mechanism treat a lineage mismatch as a confict, thus ensuring that automatic state syncing will stop if the user manages to get into a broken, mismatched state. As a special exception, we allow *empty* states to be "clobbered" regardless of lineage. This enables remote state to be configured easily in the common case where earlier user actions have implicitly caused an empty local state to be created, and also allows the remote state left behind from a config that has been destroyed to be overwritten by a new, unrelated state from a different config. The priority here is preventing actions that should never occur, so the UX is not polished. Later we may wish to make changes at higher levels of abstraction to either prevent these situations from arising in the first place (e.g. making remote configuration automatic based on config) or giving the user more guidance on resolution.
apparentlymart
force-pushed
the
f-state-safety
branch
from
ee28ef8 to
de82005
Compare
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
ghost
locked and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Through real-world use of Terraform in production I have encountered a variety of "gotchas" relating to state management.
This PR is an attempt to tackle a couple of these where I think I have a reasonable path forward. These changes are grouped into a single PR because they share a common building-block:
Serialis only expected to produce meaningful results whenLineagematches.The "features" themselves are:
terraform remote configto enable or re-configure remote state when a local state is already present. (formerly the separate PR [WIP] Tracking Remote State Lineage to Reduce Mistakes #4618)The primary goal here is to return an error when the user seems to be accidentally doing something dangerous, with as little impact as possible to "legitimate" workflows. In future version of Terraform we may make more fundamental changes to these features to help the user not make these mistakes in the first place, but this is intended as a short-term fix to reduce the risk of a state-related catastrophe.