roachtest: acceptance/gossip/peerings failed #48005
Assignees
Labels
branch-master
Failures and bugs on the master branch.
C-test-failure
Broken test (automatically or manually discovered).
O-roachtest
O-robot
Originated from a bot.
Milestone
Comments
cockroach-teamcity
added
branch-master
andreimatei
removed
the
release-blocker
Looks like we're trying to hit the admin UI ( |
|
One thing we see in the logs from this node is that: fires a few times after: |
|
Yes, that sounds reasonable. The settings are not persisted on the node, so we're always running with the default settings for a little bit of time even after signaling readiness. Something we could do here is to wait until a system config has been ingested before returning from |
|
(roachtest).acceptance/gossip/peerings failed on master@61f18db7dd9a054d9a4648f67546202f760b5000: More
Artifacts: /acceptance/gossip/peerings See this test on roachdash |
omg that explains so much about many other test failures I've investigated in the past. It also explains why certain SQL clients which should get some defaults initialized by settings don't get them when they connect immediately after a node starts. I'm going to file this as a separate issue under the "rolling restarts" project. |
|
(roachtest).acceptance/gossip/peerings failed on master@20916a30cf9356683c973f8653e8b69613a75fe4: More
Artifacts: /acceptance/gossip/peerings See this test on roachdash |
|
We cannot indiscriminately block on receiving the settings before signaling ready because of the chicken-and-egg problem that comes up when the node currently starting is needed for quorum on the system config range. For example, if a three node cluster is completely down, at least two of the nodes must be online before current settings are received. I think we should do two things here:
|
tbg
added a commit
to tbg/cockroach
that referenced
this issue
Jun 16, 2020
In an ideal world a KV node would not declare itself as ready until it has received the current cluster settings. However, we cannot indiscriminately block on that because of the chicken-and-egg problem that comes up when the node currently starting is needed for quorum on the system config range. For example, if a three node cluster is completely down, at least two of the nodes must be online before current settings are received. Instead do the following: 1. persist the settings locally on the first store whenever they are received, so that restarting nodes can come up with settings that are no staler than the ones they had when they went down. 2. if a new node joins the cluster, we can wait for the settings to show up (since the node is just joining, it is not required for any quorum). Fixes cockroachdb#48005. Release note: None
vrongmeal
added a commit
to vrongmeal/cockroach
that referenced
this issue
Oct 2, 2020
Add functions to persist settings key values with the local store prefix so restarting nodes can come up with settings that are no staler than the ones they had when they went down. Fixes cockroachdb#48005. Release note: None Signed-off-by: Vaibhav <[email protected]>
vrongmeal
added a commit
to vrongmeal/cockroach
that referenced
this issue
Oct 20, 2020
In an ideal world a KV node would not declare itself as ready until it has received the current cluster settings. However, we cannot indiscriminately block on that because of the chicken-and-egg problem that comes up when the node currently starting is needed for quorum on the system config range. For example, if a three node cluster is completely down, at least two of the nodes must be online before current settings are received. Instead do the following: 1. persist the settings locally on the first store whenever they are received, so that restarting nodes can come up with settings that are no staler than the ones they had when they went down. 2. if a new node joins the cluster, we can wait for the settings to show up (since the node is just joining, it is not required for any quorum). Fixes cockroachdb#48005. Release note: None Signed-off-by: Vaibhav <[email protected]>
vrongmeal
added a commit
to vrongmeal/cockroach
that referenced
this issue
Nov 3, 2020
In an ideal world a KV node would not declare itself as ready until it has received the current cluster settings. However, we cannot indiscriminately block on that because of the chicken-and-egg problem that comes up when the node currently starting is needed for quorum on the system config range. For example, if a three node cluster is completely down, at least two of the nodes must be online before current settings are received. Instead do the following: 1. persist the settings locally on the first store whenever they are received, so that restarting nodes can come up with settings that are no staler than the ones they had when they went down. 2. if a new node joins the cluster, we can wait for the settings to show up (since the node is just joining, it is not required for any quorum). Fixes cockroachdb#48005. Release note: None Signed-off-by: Vaibhav <[email protected]>
vrongmeal
added a commit
to vrongmeal/cockroach
that referenced
this issue
Nov 9, 2020
In an ideal world a KV node would not declare itself as ready until it has received the current cluster settings. However, we cannot indiscriminately block on that because of the chicken-and-egg problem that comes up when the node currently starting is needed for quorum on the system config range. For example, if a three node cluster is completely down, at least two of the nodes must be online before current settings are received. Instead do the following: 1. persist the settings locally on the first store whenever they are received, so that restarting nodes can come up with settings that are no staler than the ones they had when they went down. 2. if a new node joins the cluster, we can wait for the settings to show up (since the node is just joining, it is not required for any quorum). Fixes cockroachdb#48005. Release note: None Signed-off-by: Vaibhav <[email protected]>
vrongmeal
added a commit
to vrongmeal/cockroach
that referenced
this issue
Nov 10, 2020
In an ideal world a KV node would not declare itself as ready until it has received the current cluster settings. However, we cannot indiscriminately block on that because of the chicken-and-egg problem that comes up when the node currently starting is needed for quorum on the system config range. For example, if a three node cluster is completely down, at least two of the nodes must be online before current settings are received. Instead do the following: 1. persist the settings locally on the first store whenever they are received, so that restarting nodes can come up with settings that are no staler than the ones they had when they went down. 2. if a new node joins the cluster, we can wait for the settings to show up (since the node is just joining, it is not required for any quorum). Fixes cockroachdb#48005. Release note: None Signed-off-by: Vaibhav <[email protected]>
vrongmeal
added a commit
to vrongmeal/cockroach
that referenced
this issue
Nov 18, 2020
In an ideal world a KV node would not declare itself as ready until it has received the current cluster settings. However, we cannot indiscriminately block on that because of the chicken-and-egg problem that comes up when the node currently starting is needed for quorum on the system config range. For example, if a three node cluster is completely down, at least two of the nodes must be online before current settings are received. Instead do the following: 1. persist the settings locally on the first store whenever they are received, so that restarting nodes can come up with settings that are no staler than the ones they had when they went down. 2. if a new node joins the cluster, we can wait for the settings to show up (since the node is just joining, it is not required for any quorum). Fixes cockroachdb#48005. Release note: None Signed-off-by: Vaibhav <[email protected]>
craig bot
pushed a commit
that referenced
this issue
Nov 18, 2020
55166: server: ensure settings are up-to-date. r=tbg a=vrongmeal [WIP] Context: #50271 Fixes #48005. Release note: None Signed-off-by: Vaibhav <[email protected]> Co-authored-by: Vaibhav <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
(roachtest).acceptance/gossip/peerings failed on master@0e16cc15f139b816b8e46fe6571691a8ec0e6937:
More
Artifacts: /acceptance/gossip/peerings
See this test on roachdash
powered by pkg/cmd/internal/issues
The text was updated successfully, but these errors were encountered: