API for atomic snapshot backups #8047
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 5 commits
May 21, 2020 20:04
Effectively Copied from https://github.com/hashicorp/consul/tree/v1.8.0-beta1/snapshot With addition of overall snapshot checksum file
drewbailey
reviewed
May 26, 2020
drewbailey
reviewed
May 26, 2020
drewbailey
reviewed
May 26, 2020
| err = r.forwardLeader(remoteServer, method, args, reply) | ||
| return true, err | ||
| } | ||
|
|
There was a problem hiding this comment.
even though it isn't exported a comment here explaining that nil means we are the leader would be helpful
drewbailey
reviewed
May 26, 2020
nomad/operator_endpoint.go
Outdated
| op.srv.setQueryMeta(&reply.QueryMeta) | ||
|
|
||
| // Take the snapshot and capture the index. | ||
| snap, err := snapshot.New(op.logger, op.srv.raft) |
There was a problem hiding this comment.
nit: it might be helpful to rename the logger that's passed in to differentiate from the server logger
drewbailey
reviewed
May 26, 2020
nomad/structs/operator.go
Outdated
|
|
||
| // SnapshotSaveResponse is the header for the streaming snapshot endpoint, | ||
| // and followed by the snapshot file content. | ||
| // It is written to the |
drewbailey
approved these changes
May 26, 2020
Co-authored-by: Drew Bailey <[email protected]>
The callers for `forward` and old implementation expect failures to be accompanied with a true value! This fixes the issue and have tests passing!
|
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This introduces a generate snapshot API to ease taking backups of the Nomad state, along with a CLI command.
The PR introduce an endpoint (i.e.
/v1/operator/snapshot) and new CLI subcommands (i.e.nomad operator snapshot savefor generating the snapshot andnomad operator snapshot inspectfor inspecting a snapshot file). These generate atomic consistent snapshots of the nomad state, representing latest view the servers have.The snapshots include all cluster data: job definitions, ACL policies, etc. It includes sensitive information (e.g. ACL Tokens, Vault Tokens), so they need to maintained with care. Also, it may contain ephemeral information about external world that may no longer be relevant - allocations/evaluations about nodes from two days ago may no longer relevant.
I'm mulling over some design consideration for restore capability to accommodate few use cases (disaster recovery vs provisioning a new cluster with recovered jobs but targeting new nodes), and to properly handle ephemeral state. I'll follow up with a new PR.