Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

keyring: Fix a panic when decrypting aead with empty RSA block. #24442

Merged
merged 2 commits into from
Nov 12, 2024

Conversation

jrasell
Copy link
Member

@jrasell jrasell commented Nov 12, 2024

Description

Clusters that have gone through several upgrades have be found to include keyring material which has an empty RSA block.

In more recent versions of Nomad, an empty RSA block is omitted from being written to disk. This results in the panic not being present. Older versions, however, did not have this struct tag meaning we wrote an empty JSON block which is not accounted for in the current version.

Spot check for this locally in the code:

ag "WrappedRSAKey != nil {"
ag "WrappedRSAKey != nil && "
nomad/encrypter.go
480:		if wrappedKey.WrappedRSAKey != nil && len(wrappedKey.WrappedRSAKey.Ciphertext) > 0 {
808:	if kekWrapper.WrappedRSAKey != nil && len(kekWrapper.WrappedRSAKey.Ciphertext) > 0 {

Links

closes #24441
related #24383

Contributor Checklist

  • Changelog Entry If this PR changes user-facing behavior, please generate and add a
    changelog entry using the make cl command.
  • Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
    ensure regressions will be caught.
  • Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
    and job configuration, please update the Nomad website documentation to reflect this. Refer to
    the website README for docs guidelines. Please also consider whether the
    change requires notes within the upgrade guide.

Reviewer Checklist

  • Backport Labels Please add the correct backport labels as described by the internal
    backporting document.
  • Commit Type Ensure the correct merge method is selected which should be "squash and merge"
    in the majority of situations. The main exceptions are long-lived feature branches or merges where
    history should be preserved.
  • Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
    within the public repository.

Clusters that have gone through several upgrades have be found
to include keyring material which has an empty RSA block.

In more recent versions of Nomad, an empty RSA block is omitted
from being written to disk. This results in the panic not being
present. Older versions, however, did not have this struct tag
meaning we wrote an empty JSON block which is not accounted for
in the current version.
@jrasell jrasell added backport/ent/1.8.x+ent Changes are backported to 1.8.x+ent backport/ent/1.9.x+ent Changes are backported to 1.9.x+ent backport/1.9.x backport to 1.9.x release line labels Nov 12, 2024
@jrasell jrasell marked this pull request as ready for review November 12, 2024 11:56
@jrasell jrasell self-assigned this Nov 12, 2024
Copy link
Contributor

@pkazmierczak pkazmierczak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Member

@tgross tgross left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jrasell jrasell merged commit 4e7496d into main Nov 12, 2024
35 checks passed
@jrasell jrasell deleted the b-gh-24441 branch November 12, 2024 14:26
@devdrik
Copy link

devdrik commented Nov 14, 2024

Is there a way to manually heal a cluster that ran into this issue?

edit: we have a workaround for this now: we build nomad 1.9.4-dev from source and inject the nomad binary to the servers. They pick up the new version and start working as expected.

@blalor
Copy link
Contributor

blalor commented Nov 18, 2024

Adding to @devdrik's workaround, I grabbed a binary artifact from the latest build action on the releases/1.9.x branch: https://github.com/hashicorp/nomad/actions/workflows/build.yml?query=branch%3Arelease%2F1.9.x (click a run, scroll to "Artifacts").

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport/ent/1.8.x+ent Changes are backported to 1.8.x+ent backport/ent/1.9.x+ent Changes are backported to 1.9.x+ent backport/1.9.x backport to 1.9.x release line
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Nomad 1.9.3: panic: runtime error: slice bounds out of range [:12] with capacity 0
5 participants