roachtest: tpcc/mixed-headroom/n5cpu16 failed

[cockroach-teamcity]

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on release-20.2@e1d01d047e833da659ee0c2bd570e5c4143c784e:

		  |   356.0s        0            0.0           18.2      0.0      0.0      0.0      0.0 delivery
		  |   356.0s        0            0.0          175.3      0.0      0.0      0.0      0.0 newOrder
		  |   356.0s        0            0.0           18.4      0.0      0.0      0.0      0.0 orderStatus
		  |   356.0s        0            0.0          181.1      0.0      0.0      0.0      0.0 payment
		  |   356.0s        0            0.0           18.2      0.0      0.0      0.0      0.0 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |   357.0s        0            0.0           18.2      0.0      0.0      0.0      0.0 delivery
		  |   357.0s        0            0.0          174.8      0.0      0.0      0.0      0.0 newOrder
		  |   357.0s        0            0.0           18.3      0.0      0.0      0.0      0.0 orderStatus
		  |   357.0s        0            0.0          180.6      0.0      0.0      0.0      0.0 payment
		  |   357.0s        0            0.0           18.1      0.0      0.0      0.0      0.0 stockLevel
		  |   358.0s        0            0.0           18.1      0.0      0.0      0.0      0.0 delivery
		  |   358.0s        0            0.0          174.3      0.0      0.0      0.0      0.0 newOrder
		  |   358.0s        0            0.0           18.3      0.0      0.0      0.0      0.0 orderStatus
		  |   358.0s        0            0.0          180.1      0.0      0.0      0.0      0.0 payment
		  |   358.0s        0            0.0           18.1      0.0      0.0      0.0      0.0 stockLevel
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2655,tpcc.go:174,tpcc.go:269,test_runner.go:755: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2643
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2651
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:174
		  | main.registerTPCC.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:269
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2699
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2613
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6309
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:208
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /tpcc/mixed-headroom/n5cpu16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[erikgrinaker]

This failed with:

Error: error in newOrder: ERROR: result is ambiguous (intent missing and record aborted) (SQLSTATE 40003)

That seems to be a valid error that might occur due to e.g. conflicts, does this warrant further investigation or should we just make TPCC tolerant to these errors?

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 1059 to 1072 in ecc031c

	case roachpb.ABORTED:
	// The transaction has either already been finalized as aborted or has been
	// finalized as committed and already had its transaction record GCed. Both
	// these cases return an ABORTED txn; in the GC case the record has been
	// synthesized.
	// If the record has been GC'ed, then we can't distinguish between the
	// two cases, and so we're forced to return an ambiguous error. On the other
	// hand, if the record exists, then we know that the transaction did not
	// commit because a committed record cannot be GC'ed and the recreated as
	// ABORTED.
	if resp.TxnRecordExists {
	return false, roachpb.NewTransactionAbortedError(roachpb.ABORT_REASON_ABORTED_RECORD_FOUND)
	}
	return false, roachpb.NewAmbiguousResultErrorf("intent missing and record aborted")