roachtest: election-after-restart failed [skipped]

[cockroach-teamcity]

(roachtest).election-after-restart failed on release-20.2@09786094a013b45e4d794f9cb02a4256f70b9158:

The test failed on branch=release-20.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/election-after-restart/run_1
	election.go:95,test_runner.go:754: expected query to succeed in less than 15s, took 19.712360991s

More

Artifacts: /election-after-restart

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[cockroach-teamcity]

(roachtest).election-after-restart failed on release-20.2@7ab6963f9c48e61df8f138894f69f056fe1aa8f5:

The test failed on branch=release-20.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/election-after-restart/run_1
	election.go:95,test_runner.go:754: expected query to succeed in less than 15s, took 22.020802541s

More

Artifacts: /election-after-restart

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[nvanbenschoten]

@knz can I assign you to take a look at this?

[cockroach-teamcity]

(roachtest).election-after-restart failed on release-20.2@56c72f47ae31ad3373b69f87250d57fdfee176ce:

The test failed on branch=release-20.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/election-after-restart/run_1
	election.go:95,test_runner.go:754: expected query to succeed in less than 15s, took 17.534954179s

More

Artifacts: /election-after-restart
Related:

• roachtest: unexpected message 'C' in simple query execution in election-after-restart #54389 roachtest: election-after-restart failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[tbg]

@andreimatei didn't you want to assign yourself?

[irfansharif]

I'm back in office today, taking a look now.

[irfansharif]

This test was unskipped recently (#53689) after being skipped over a year ago (#37507). I'm able to repro it pretty easily on my machine (which runs counter to Rafa's experience in #35047 (comment)?).

I used just the following:

rm -r artifacts; make buildshort; make bin/roachtest; roachprod destroy local;
roachtest run election-after-restart --cockroach ./cockroachshort --local --debug --count=10

Looking through the symptoms they're identical to what was described a year ago in #35047 (comment). Our thinking at the time was to address #21849 to address the lag we're tracking in this failure ("expected query to succeed in less than 15s, took 22.020802541s"). In any case, this is not a release blocker. I'm going to re-skip the test and keep this issue open. It also affects master currently though I don't see any issues filed for it. I'm going to defer investigation here, I have a few other tests to skim through.

[irfansharif]

+cc @knz you may be interested in the discussion above as it relates to "crdb post restart behavior". It's not something we can do much about in 20.2 but might finally nudge us to pick up #21849 if we're looking to close out another loose end in the area.