roachtest: c2c/shutdown/dest/coordinator failed - failed ingestion job resume after node failure

[cockroach-teamcity]

roachtest.c2c/shutdown/dest/coordinator failed with artifacts on master @ 245c3b9db56567d9820babe460c5ed8a3786890e:

test artifacts and logs in: /artifacts/c2c/shutdown/dest/coordinator/run_1
(sql_runner.go:218).Scan: error scanning '&{0xc0062706c0 <nil>}': pq: system-jobs-scan: rpc error: code = Unavailable desc = connection error: desc = "transport: error while dialing: connection interrupted (did the remote node shut down or are there networking issues?)"
(1) pq: system-jobs-scan: rpc error: code = Unavailable desc = connection error: desc = "transport: error while dialing: connection interrupted (did the remote node shut down or are there networking issues?)"
Error types: (1) *pq.Error
(monitor.go:127).Wait: monitor failure: monitor task failed: context canceled while waiting for job to finish: context canceled
(test_runner.go:1089).func1: 1 dead node(s) detected

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-27786

[msbutler]

this infra flake will be fixed by #102382

[cockroach-teamcity]

roachtest.c2c/shutdown/dest/coordinator failed with artifacts on master @ 107ad579b1ff8311ac4f4df4783985010a0322f9:

test artifacts and logs in: /artifacts/c2c/shutdown/dest/coordinator/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: unexpectedly found job 865597372800925699 in state pause-requested
(test_runner.go:1108).func1: 1 dead node(s) detected

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

• roachtest: c2c/shutdown/dest/coordinator failed #103392 roachtest: c2c/shutdown/dest/coordinator failed [C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-23.1 release-blocker]

This test on roachdash | Improve this report!

[msbutler]

This latest failure seems legit. Here's the timeline:

1. We issue a cutover command
2. We kill the destination job coordinator (node 6)
3. Node 5 attempts to take over the job lease and resume the job, but fails to do so because:

 producer job is not active and in status STREAM_INACTIVE) but is being paused›

4. After this error, I see some very strange log lines about the ingestion job stepping through the succeed state.

I230516 09:43:20.725988 41597 jobs/registry.go:1592 ⋮ [T1,n1] 630  STREAM INGESTION job 865597372800925699: stepping through state succeeded
E230516 09:43:20.735804 41597 jobs/adopt.go:479 ⋮ [T1,n1] 631  job 865597372800925699: adoption completed with error job 865597372800925699: could not mark as succeeded: job 865597372800925699: job with status pause-requested cannot be marked as succeeded

[stevendanna]

We believe this one is fixed.