roachtest/bank: don't start chaos until nodes are serving SQL #30989
Conversation
|
Please hold off on this. I thought it would fix the second class of failures we saw in #30064, but I just had a repro so it doesn't seem like it did. |
|
I still have this cleanup on my plate, including the l156 fix, but I'd like to actually get this test working without flakes first, which is turning out to be surprisingly tricky. |
nvanbenschoten
force-pushed
the
nvabenschoten/backFix
branch
from
9280b9d to
5bd0dec
Compare
|
This change is |
nvanbenschoten
changed the title
There was a problem hiding this comment.
Reviewed 1 of 1 files at r1, 1 of 1 files at r2.
Reviewable status:complete! 0 of 0 LGTMs obtained (and 1 stale)
pkg/cmd/roachtest/bank.go, line 210 at r1 (raw file):
consistentIdx int, ) { s.waitGroup.Add(1)
Is this kosher? You're calling Add from a goroutine and then calling Wait from the main routine. I think this would hit an internal assertion in WaitGroup, except that the Wait doesn't happen right away.
pkg/cmd/roachtest/bank.go, line 215 at r3 (raw file):
// Don't begin the chaos monkey until all nodes are serving SQL connections. // This ensures that we don't test cluster initialization under chaos. for i := 0; i < c.nodes; i++ {
Nit: for i := 0; i <= c.nodes; i++. Then in the body you can use i instead of i+1.
There was a problem hiding this comment.
Reviewable status:
pkg/cmd/roachtest/bank.go, line 210 at r1 (raw file):
Previously, petermattis (Peter Mattis) wrote…
Is this kosher? You're calling
Addfrom a goroutine and then callingWaitfrom the main routine. I think this would hit an internal assertion inWaitGroup, except that theWaitdoesn't happen right away.
Yeah, you're absolutely right.
pkg/cmd/roachtest/bank.go, line 215 at r3 (raw file):
Previously, petermattis (Peter Mattis) wrote…
Nit:
for i := 0; i <= c.nodes; i++. Then in the body you can useiinstead ofi+1.
Make it for i := 1; i <= c.nodes; i++ and you've got a deal.
There was a problem hiding this comment.
Reviewed 1 of 1 files at r1, 1 of 1 files at r2, 1 of 1 files at r3.
Reviewable status:
nvanbenschoten
force-pushed
the
nvabenschoten/backFix
branch
from
5bd0dec to
0cff6f2
Compare
There was a problem hiding this comment.
Reviewable status:
pkg/cmd/roachtest/bank.go, line 210 at r1 (raw file):
Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Yeah, you're absolutely right.
Done.
pkg/cmd/roachtest/bank.go, line 215 at r3 (raw file):
Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Make it
for i := 1; i <= c.nodes; i++and you've got a deal.
Done.
There was a problem hiding this comment.
Reviewable status:
There was a problem hiding this comment.
Having just read through the recent comments on #30064 I agree this is the right change to make. We should also reduce the migration lease duration, though. There's no need for it to be so long given that we renew it in the background while running migrations.
Release note: None
Release note: None
Informs cockroachdb#30064. This prevents us from running into issues with SQL migrations. Release note: None
nvanbenschoten
force-pushed
the
nvabenschoten/backFix
branch
from
0cff6f2 to
b1ab7dd
Compare
|
TFTRs.
I was hoping for a quick fix here, but it looks like this is tied to the bors r+ |
30989: roachtest/bank: don't start chaos until nodes are serving SQL r=nvanbenschoten a=nvanbenschoten Informs #30064. Release note: None Co-authored-by: Nathan VanBenschoten <[email protected]>
Build succeeded |
Informs #30064.
Release note: None