[wip] server: ensure settings are up-to-date #50271
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In an ideal world a KV node would not declare itself as ready until it has received the current cluster settings. However, we cannot indiscriminately block on that because of the chicken-and-egg problem that comes up when the node currently starting is needed for quorum on the system config range. For example, if a three node cluster is completely down, at least two of the nodes must be online before current settings are received. Instead do the following: 1. persist the settings locally on the first store whenever they are received, so that restarting nodes can come up with settings that are no staler than the ones they had when they went down. 2. if a new node joins the cluster, we can wait for the settings to show up (since the node is just joining, it is not required for any quorum). Fixes cockroachdb#48005. Release note: None
|
This change is |
ajwerner
reviewed
Jun 16, 2020
There was a problem hiding this comment.
Should we also write the settings values to the store during bootstrap?
Reviewed 3 of 3 files at r1.
Reviewable status:complete! 0 of 0 LGTMs obtained (waiting on @tbg)
pkg/server/init.go, line 238 at r1 (raw file):
var initialSettingsKVs []roachpb.KeyValue select { case <-time.After(5 * time.Second):
Should we log in this case?
|
I like the approach proposed here. The guarantee that a node's settings will never regress across restarts is a big improvement. |
tbg
added a commit
to tbg/cockroach
that referenced
this pull request
Aug 27, 2020
Pulling the gossip info via HTTP can fail if the freshly restarted nodes didn't gossip the actual cluster settings yet. Fixing the initialization is tracked via cockroachdb#50271. Closes cockroachdb#48005. Release justification: testing Release note: None
craig bot
pushed a commit
that referenced
this pull request
Nov 18, 2020
55166: server: ensure settings are up-to-date. r=tbg a=vrongmeal [WIP] Context: #50271 Fixes #48005. Release note: None Signed-off-by: Vaibhav <[email protected]> Co-authored-by: Vaibhav <[email protected]>
|
@vrongmeal took care of this! #55166 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In an ideal world a KV node would not declare itself as ready until
it has received the current cluster settings.
However, we cannot indiscriminately block on that because of the
chicken-and-egg problem that comes up when the node currently starting
is needed for quorum on the system config range. For example, if a three
node cluster is completely down, at least two of the nodes must be
online before current settings are received.
Instead do the following:
received, so that restarting nodes can come up with settings that are no
staler than the ones they had when they went down.
up (since the node is just joining, it is not required for any quorum).
Fixes #48005.
Release note: None