logs: fix missing allocation logs after update to Nomad 1.5.4 #17087
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tgross
force-pushed
the
fix-log-collection-upgrade
branch
from
d54c610 to
45af110
Compare
tgross
force-pushed
the
fix-log-collection-upgrade
branch
from
45af110 to
775c665
Compare
When the server restarts for the upgrade, it loads the `structs.Job` from the Raft snapshot/logs. The jobspec has long since been parsed, so none of the guards around the default value are in play. The empty field value for `Enabled` is the zero value, which is false. This doesn't impact any running allocation because we don't replace running allocations when either the client or server restart. But as soon as any allocation gets rescheduled (ex. you drain all your clients during upgrades), it'll be using the `structs.Job` that the server has, which has `Enabled = false`, and logs will not be collected. This changeset fixes the bug by adding a new field `Disabled` which defaults to false (so that the zero value works), and deprecates the old field. Fixes #17076
tgross
force-pushed
the
fix-log-collection-upgrade
branch
from
775c665 to
4ef8b96
Compare
schmichael
reviewed
May 4, 2023
|
@schmichael I've addressed your comments and added some extra tests around those cases. I've also done bench testing across upgrades: both from 1.5.3 to 1.5.5-dev and from 1.5.3 to 1.5.4 to 1.5.5-dev. The nice thing I discovered with the fix is that allocations with missing |
schmichael
approved these changes
May 4, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When the server restarts for the upgrade, it loads the
structs.Jobfrom the Raft snapshot/logs. The jobspec has long since been parsed, so none of the guards around the default value are in play. The empty field value forEnabledis the zero value, which is false.This doesn't impact any running allocation because we don't replace running allocations when either the client or server restart. But as soon as any allocation gets rescheduled (ex. you drain all your clients during upgrades), it'll be using the
structs.Jobthat the server has, which hasEnabled = false, and logs will not be collected.This changeset fixes the bug by adding a new field
Disabledwhich defaults to false (so that the zero value works), and deprecates the old field.Fixes #17076