-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
keyring: Fix a panic when decrypting aead with empty RSA block. #24442
Conversation
Clusters that have gone through several upgrades have be found to include keyring material which has an empty RSA block. In more recent versions of Nomad, an empty RSA block is omitted from being written to disk. This results in the panic not being present. Older versions, however, did not have this struct tag meaning we wrote an empty JSON block which is not accounted for in the current version.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Is there a way to manually heal a cluster that ran into this issue? edit: we have a workaround for this now: we build nomad 1.9.4-dev from source and inject the nomad binary to the servers. They pick up the new version and start working as expected. |
Adding to @devdrik's workaround, I grabbed a binary artifact from the latest build action on the releases/1.9.x branch: https://github.com/hashicorp/nomad/actions/workflows/build.yml?query=branch%3Arelease%2F1.9.x (click a run, scroll to "Artifacts"). |
Description
Clusters that have gone through several upgrades have be found to include keyring material which has an empty RSA block.
In more recent versions of Nomad, an empty RSA block is omitted from being written to disk. This results in the panic not being present. Older versions, however, did not have this struct tag meaning we wrote an empty JSON block which is not accounted for in the current version.
Spot check for this locally in the code:
Links
closes #24441
related #24383
Contributor Checklist
changelog entry using the
make cl
command.ensure regressions will be caught.
and job configuration, please update the Nomad website documentation to reflect this. Refer to
the website README for docs guidelines. Please also consider whether the
change requires notes within the upgrade guide.
Reviewer Checklist
backporting document.
in the majority of situations. The main exceptions are long-lived feature branches or merges where
history should be preserved.
within the public repository.