vault: fix token revocation during workflow migration #19689
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lgfa29
commented
Jan 9, 2024
lgfa29
commented
Jan 9, 2024
lgfa29
force-pushed
the
b-fix-vault-noop-client
branch
from
89afc21 to
9d394e4
Compare
lgfa29
commented
Jan 9, 2024
Comment on lines
+765
to
+766
| * Resubmit Nomad jobs that need access to Vault to redeploy them with a new | ||
| workload identity for Vault. |
There was a problem hiding this comment.
I was trying to think of ways we could help with this. Two ideas I had were:
- A command that looks for the following artifacts and tell users if their clusters are ready for the migration.
- jobs with
vaultblocks (and maybetemplateswith Vault-related functions? Avaultblock is not always required to access Vault) but no Vault identities. - unused
VaultAccessorsin state store (and potentially clean them without needing a leadership transition)
- Some kind of metric that operators can monitor and wait until it reaches zero. But I'm not sure yet exactly what to measure.
There was a problem hiding this comment.
I like the idea of a command... maybe we could bake this into a -check flag for nomad setup vault?
A metric would be nice, but the only reasonable place for it would be in the state store and we'd have to repopulate it during snapshot restore. Kind of messy.
There was a problem hiding this comment.
Cool! I will try to think a bit more where to place the command. I was thinking somewhere under nomad operator to indicate that this may require a management token since it needs to look all over the state store.
It could also be used for other upgrade checks in the future.
tgross
approved these changes
Jan 9, 2024
Comment on lines
+765
to
+766
| * Resubmit Nomad jobs that need access to Vault to redeploy them with a new | ||
| workload identity for Vault. |
There was a problem hiding this comment.
I like the idea of a command... maybe we could bake this into a -check flag for nomad setup vault?
A metric would be nice, but the only reasonable place for it would be in the state store and we'd have to repopulate it during snapshot restore. Kind of messy.
lgfa29
force-pushed
the
b-fix-vault-noop-client
branch
from
9d394e4 to
2dac161
Compare
lgfa29
force-pushed
the
b-fix-vault-noop-client
branch
from
2dac161 to
984f7fc
Compare
When transitioning from the legacy token-based workflow to the new JWT workflow for Vault the previous code would instantiate a no-op Vault if the server configuration had a `default_identity` block. This no-op client returned an error for some of its operations were called, such as `LookupToken` and `RevokeTokens`. The original intention was that, in the new JWT workflow, none of these methods should be called, so returning an error could help surface potential bugs. But the `RevokeTokens` and `MarkForRevocation` methods _are_ called even in the JWT flow. When a leadership transition happens, the new server looks for unused Vault accessors from state and tries to revoke them. Similarly, the `RevokeTokens` method is called every time the `Node.UpdataStatus` and `Node.UpdateAlloc` RPCs are made by clients, as the Nomad server tries to find unused Vault tokens for the node/alloc. Since the new JWT flow does not require Nomad servers to contact Vault, calling `RevokeTokens` and `MarkForRevocation` is not able to complete without a Vault token, so this commit changes the logic to use the no-op Vault client when no token is configured. It also updates the client itself to not error if these methods are called, but to rather just log so operators can be made aware that there are Vault tokens created by Nomad that have not been force-expired.
When migrating an existing cluster to the new workload identity based flow, Nomad operators must first upgrade the Nomad version without removing any of the existing Vault configuration. Doing so can prevent Nomad servers from managing and cleaning-up existing Vault tokens during a leadership transition and node or alloc updates. Operators must also resubmit all jobs with a `vault` block so they are updated with an `identity` for Vault. Skipping this step may cause allocations to fail if their Vault token expires (if, for example, the Nomad client stops running for TTL/2) or if they are rescheduled, since the new client will try to follow the legacy flow which will fail if the Nomad server configuration for Vault has already been updated to remove the Vault address and token.
lgfa29
force-pushed
the
b-fix-vault-noop-client
branch
from
984f7fc to
315acaf
Compare
|
We just hit this problem. We had rollback our config without workload identity. We'll wait for the next release. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When transitioning from the legacy token-based workflow to the new JWT workflow for Vault the previous code would instantiate a no-op Vault if the server configuration had a
default_identityblock.This no-op client returned an error for some of its operations were called, such as
LookupTokenandRevokeTokens. The original intention was that, in the new JWT workflow, none of these methods should be called, so returning an error could help surface potential bugs.But the
RevokeTokensandMarkForRevocationmethods are called even in the JWT flow. When a leadership transition happens, the new server looks for unused Vault accessors from state and tries to revoke them. Similarly, theRevokeTokensmethod is called every time theNode.UpdataStatusandNode.UpdateAllocRPCs are made by clients, as the Nomad server tries to find unused Vault tokens for the node/alloc.Since the new JWT flow does not require Nomad servers to contact Vault, calling
RevokeTokensandMarkForRevocationis not able to complete without a Vault address and token, so this commit changes the logic to use the no-op Vault client when one of them is not passed. It also updates the client itself to not error if these methods are called, but rather raise a warning log so operators can be made aware that there are Vault tokens created by Nomad that have not been force-expired.There are also updates to the documentation of the migration process. When migrating an existing cluster to the new workload identity based flow, Nomad operators must first upgrade the Nomad version without removing any of the existing Vault configuration. Doing so can prevent Nomad servers from managing and cleaning-up existing Vault tokens during a leadership transition and node or alloc updates.
Operators must also resubmit all jobs with a
vaultblock so they are updated with anidentityfor Vault. Skipping this step may cause allocations to fail if their Vault token expires (if, for example, the Nomad client stops running for TTL/2) or if they are rescheduled, since the new client will try to follow the legacy flow which will fail if the Nomad server configuration for Vault has already been updated to remove the Vault address and token.