roachtest: weekly/tpcc/headroom failed

[cockroach-teamcity]

roachtest.weekly/tpcc/headroom failed with artifacts on master @ 2a5e231716c436781f12452d800651f51c6383b7:

(monitor.go:154).Wait: monitor failure: full command output in run_110525.127724219_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/weekly/tpcc/headroom/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=true
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-37345

[srosenberg]

The workload fails with,

Error: error in payment: ERROR: result is ambiguous: error=failed to send RPC: sending to all replicas failed; last error: r593/2	:(n2,s2) is unavailable (circuit breaker tripped): context canceled [exhausted] (last error: failed to send RPC: sending to all replicas failed; last error: r593/2:(n2,s2) is unavailable (circuit breaker tripped): context canceled) (SQLSTATE 40003)

Having gone through the logs and grafana dashboards nothing really jumps out. System-wise and runtime-wise, there is no sign of n2 being overloaded, yet raft applies are trailing behind. We see multiple instances of node might be overloaded,

grep "node might be overloaded" cockroach.log|awk '{print $2}'|awk -F":" '{print $1, $2}'|sort |uniq
14 11
14 14
14 33
14 50
15 18
15 32
17 43
18 36
18 49
18 50

Coincidentally, these minute offsets co-occur with the scheduled backup job running every ~15 mins. It's unclear why n2 continued to flap, culminating in the failure of the workload. Why couldn't the lease be transferred to a different node?

Removing release-blocker since it doesn't appear to be a new issue per se, but this warrants a further investigation.

[andrewbaptist]

This is most likely fixed by #121317. When the system is "stressed" its possible there is a transaction conflct or other non-ambiguous error that gets escalated to an ambiguous error if its going through a proxy. If this reoccurs, please reopen, but the ambiguous errors during situations like this are exactly what this change fixed.