ccl/multiregionccl: TestMultiRegionDataDriven failed #98020
Assignees
Labels
A-kv-distribution
Relating to rebalancing and leasing.
A-kv-test-failure-complex
A kv C-test-failure which requires a medium-large amount of work to address.
branch-master
Failures and bugs on the master branch.
C-bug
Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
C-test-failure
Broken test (automatically or manually discovered).
O-robot
Originated from a bot.
skipped-test
T-kv
KV Team
Milestone
Comments
cockroach-teamcity
added
branch-master
|
This is a manifestation of #92235. |
|
ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 4dc9e98d8cc1b137ee4cc950a4a42e5895bb890b: Parameters: Same failure on other branches
|
|
ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 4dc9e98d8cc1b137ee4cc950a4a42e5895bb890b: Parameters: Same failure on other branches
|
|
ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 01e14f68f7c757cab1c878cebfecd73c5119046b: Parameters: Same failure on other branches
|
|
ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 5b2a5670cbbe895d76602c230390816e783e0caa: Parameters: Same failure on other branches
|
|
ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 95684388bfbb372ac8c7e1a8ebd4a0d6f447e147: Same failure on other branches
|
|
ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 95684388bfbb372ac8c7e1a8ebd4a0d6f447e147: Parameters: Same failure on other branches
|
|
ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ e4924e2b9be4a36d466beab53a80df9241df4783: Parameters: Same failure on other branches
|
|
This failure more seems new as of four days ago. |
|
ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 36d39aadc5e7b88732e1318363177507193eb52e: Same failure on other branches
|
|
ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 4215b86090794a78649aa4463c7bf95d74c57c37: Parameters: Same failure on other branches
|
|
I used It pointed to 3a4e0c8 as the first failing commit. The failure is: cc @matthewtodd in case you have any ideas. I wonder if the changes in pkg/sql/executor_statement_metrics.go are relevant to this test. |
|
Thanks, @rafiss, I'm guessing there's something in this |
|
ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 6c99966f604f3521acdb925b9f689529ffd46df3: Same failure on other branches
|
|
ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 53dbb86acb1d48309530181b94838faf937084d3: Parameters: Same failure on other branches
|
|
ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 9acc75317aebfdfe69fb097a8a28e0040c7a67fd: Parameters: Same failure on other branches
|
kvoli
added a commit
to kvoli/cockroach
that referenced
this issue
Sep 6, 2023
Update the cluster setting statement to correctly surround the value duration in quotations.`5m` to `'5m'`. This was fixed in backports to release-23.1 (cockroachdb#110052) and release-22.2 (cockroachdb#110053). However, this was not caught on master due to `TestMultiRegionDataDriven` being skipped. Informs: cockroachdb#98020 Release note: None
kvoli
added a commit
to kvoli/cockroach
that referenced
this issue
Sep 6, 2023
`TestMultiRegionDataDriven` was skipped in cockroachdb#108107 due to finish (tracing span) being called twice in raft reproposals cockroachdb#107521, which cockroachdb#108775 fixed. Unskip the top level `TestMultiRegionDataDriven` test. Note `/secondary_region` is still skipped due to a known allocator bug. Informs: cockroachdb#98020 Epic: none Release note: None
craig bot
pushed a commit
that referenced
this issue
Sep 6, 2023
110063: ccl/multiregionccl: unskip test multi region dd parent test r=rafiss a=kvoli `TestMultiRegionDataDriven` was skipped in #108107 due to finish (tracing span) being called twice in raft reproposals #107521, which #108775 fixed. Unskip the top level `TestMultiRegionDataDriven` test. Note `/secondary_region` is still skipped due to a known allocator bug. Informs: #98020 Epic: none Release note: None Co-authored-by: Austen McClernon <[email protected]>
kvoli
added a commit
to kvoli/cockroach
that referenced
this issue
Oct 3, 2023
Previously, it was possible for a satisfiable voter constraint to never be satisfied when: 1. There were a correct number of `VOTER` and `NON_VOTER` replicas. 2. All existing replicas were necessary to satisfy a replica constraint, or voter constraint. The allocator relies on the `RebalanceVoter` path to resolve voter constraint violations when there are a correct number of each replica type. Candidates which are `necessary` to satisfy a constraint are ranked higher as rebalance targets than those which are not. Under most circumstances this leads to constraint conformance. However, when every existing replica is necessary to satisfy a replica constraint, and a voter constraint is unsatisfied -- `RebalanceVoter` would not consider swapping a `VOTER` and `NON_VOTER` to satisfy the constraint. For example, consider a setup where there are two stores, one in locality `a` and the other `b`, where some range has the following config and initial placement: ``` replicas = a(non-voter) b(voter) constraints = a:1 b:1 voter_constraints = a:1 ``` In this example, the only satisfiable placement is `a(voter)` `b(non-voter)`, which would require promoting `a(non-voter) -> a(voter)`, and demoting `b(voter)->b(non-voter)`. However, both are necessary to satisfy `constraints` leading to no rebalance occurring. Add an additional field to the allocator candidate struct, which is used to sort rebalance candidates. The new field, `voterNecessary` is sorted strictly after `necessary`, but before `diversityScore`. The `voterNecessary` field can be true only when rebalancing voters, and when the rebalance candidate is necessary to satisfy a voter constraint, the rebalance candidate already has a non-voter, and the existing voter is not necessary to satisfy *any* voter constraint. Note these rebalances are turned into swaps (promotion and demotion) in `plan.ReplicationChangesForRebalance`, so incur no snapshots. Fixes: cockroachdb#98020 Fixes: cockroachdb#106559 Fixes: cockroachdb#108127 Release note (bug fix): Voter constraints which were never satisfied due to all existing replicas being considered necessary to satisfy a replica constraint, will now be satisfied by promoting existing non-voters.
craig bot
pushed a commit
that referenced
this issue
Oct 6, 2023
111609: allocator: prioritize non-voter promotion to satisfy voter constraints r=sumeerbhola a=kvoli Previously, it was possible for a satisfiable voter constraint to never be satisfied when: 1. There were a correct number of `VOTER` and `NON_VOTER` replicas. 2. All existing replicas were necessary to satisfy a replica constraint, or voter constraint. The allocator relies on the `RebalanceVoter` path to resolve voter constraint violations when there are a correct number of each replica type. Candidates which are `necessary` to satisfy a constraint are ranked higher as rebalance targets than those which are not. Under most circumstances this leads to constraint conformance. However, when every existing replica is necessary to satisfy a replica constraint, and a voter constraint is unsatisfied -- `RebalanceVoter` would not consider swapping a `VOTER` and `NON_VOTER` to satisfy the constraint. For example, consider a setup where there are two stores, one in locality `a` and the other `b`, where some range has the following config and initial placement: ``` replicas = a(non-voter) b(voter) constraints = a:1 b:1 voter_constraints = a:1 ``` In this example, the only satisfiable placement is `a(voter)` `b(non-voter)`, which would require promoting `a(non-voter) -> a(voter)`, and demoting `b(voter)->b(non-voter)`. However, both are necessary to satisfy `constraints` leading to no rebalance occurring. Add an additional field to the allocator candidate struct, which is used to sort rebalance candidates. The new field, `voterNecessary` is sorted strictly after `necessary`, but before `diversityScore`. The `voterNecessary` field can be true only when rebalancing voters, and when the rebalance candidate is necessary to satisfy a voter constraint, the rebalance candidate already has a non-voter, and the existing voter is not necessary to satisfy *any* voter constraint. Note these rebalances are turned into swaps (promotion and demotion) in `plan.ReplicationChangesForRebalance`, so incur no snapshots. Fixes: #98020 Fixes: #106559 Fixes: #108127 Release note (bug fix): Voter constraints which were never satisfied due to all existing replicas being considered necessary to satisfy a replica constraint, will now be satisfied by promoting existing non-voters. Co-authored-by: Austen McClernon <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ fcea283ebca17a6d923c5d4b0401697438b77dbd:
Help
See also: How To Investigate a Go Test Failure (internal)
Same failure on other branches
This test on roachdash | Improve this report!
Jira issue: CRDB-25019
The text was updated successfully, but these errors were encountered: