roachtest: tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed #93623

cockroach-teamcity · 2022-12-14T17:34:42Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 052acc88ad9d7296ce6b8b441627fb469cc74d95:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(test_impl.go:291).Fatal: 4: expected version 1000022.2-16, got 1000022.1-18
(test_impl.go:291).Fatal: monitor failure: monitor task failed: output in run_172308.167707597_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/test-eng _{This test on roachdash | Improve this report!

Jira issue: CRDB-22441}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2022-12-15T17:09:41Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ d6f98e90684894fd36f53596e6aac355676d232e:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(test_impl.go:291).Fatal: 4: expected version 1000022.2-16, got 1000022.1-18
(test_impl.go:291).Fatal: monitor failure: monitor task failed: output in run_165806.942392400_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

renatolabs · 2022-12-16T15:53:45Z

A couple of errors here. First one is in the AUTO CREATE STATS job:

I221215 16:57:43.391334 43903 jobs/registry.go:1371 â‹® [n1] 266  AUTO CREATE STATS job 822649560166596611: stepping through state reverting with error: AS OF SYSTEM TIME: cannot specify timestamp older than 5m0s for this operation
I221215 16:57:43.391334 43903 jobs/registry.go:1371 â‹® [n1] 266 +(1) attached stack trace
I221215 16:57:43.391334 43903 jobs/registry.go:1371 â‹® [n1] 266 +  -- stack trace:
I221215 16:57:43.391334 43903 jobs/registry.go:1371 â‹® [n1] 266 +  | github.com/cockroachdb/cockroach/pkg/sql/row.(*Fetcher).StartInconsistentScan
I221215 16:57:43.391334 43903 jobs/registry.go:1371 â‹® [n1] 266 +  | 	github.com/cockroachdb/cockroach/pkg/sql/row/fetcher.go:528
I221215 16:57:43.391334 43903 jobs/registry.go:1371 â‹® [n1] 266 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*tableReader).startScan
I221215 16:57:43.391334 43903 jobs/registry.go:1371 â‹® [n1] 266 +  | 	github.com/cockroachdb/cockroach/pkg/sql/rowexec/tablereader.go:221
I221215 16:57:43.391334 43903 jobs/registry.go:1371 â‹® [n1] 266 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*tableReader).Next
I221215 16:57:43.391334 43903 jobs/registry.go:1371 â‹® [n1] 266 +  | 	github.com/cockroachdb/cockroach/pkg/sql/rowexec/tablereader.go:259
I221215 16:57:43.391334 43903 jobs/registry.go:1371 â‹® [n1] 266 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*samplerProcessor).mainLoop
I221215 16:57:43.391334 43903 jobs/registry.go:1371 â‹® [n1] 266 +  | 	github.com/cockroachdb/cockroach/pkg/sql/rowexec/sampler.go:253
I221215 16:57:43.391334 43903 jobs/registry.go:1371 â‹® [n1] 266 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*samplerProcessor).Run
I221215 16:57:43.391334 43903 jobs/registry.go:1371 â‹® [n1] 266 +  | 	github.com/cockroachdb/cockroach/pkg/sql/rowexec/sampler.go:229
I221215 16:57:43.391334 43903 jobs/registry.go:1371 â‹® [n1] 266 +  | github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).StartInternal.func1
I221215 16:57:43.391334 43903 jobs/registry.go:1371 â‹® [n1] 266 +  | 	github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:440
I221215 16:57:43.391334 43903 jobs/registry.go:1371 â‹® [n1] 266 +  | runtime.goexit
I221215 16:57:43.391334 43903 jobs/registry.go:1371 â‹® [n1] 266 +  | 	GOROOT/src/runtime/asm_amd64.s:1594

The other is in the migration itself (create-stmt-diag-reqs-v3-index), making the test fail:

I221215 17:06:35.985586 1415313 jobs/registry.go:1446 â‹® [n1] 780  job 822652423205912577: running execution encountered retriable error: non-cancelable: running migration for 1000022.1-20: error while validating descriptors during operation create-stmt-diag-reqs-v3-index: expected descriptor doesn't match with found descriptor: â€¹StoreColumnNames[1]: "min_execution_latency" != "sampling_probability"â€º
I221215 17:06:35.985586 1415313 jobs/registry.go:1446 â‹® [n1] 780 +(1) attached stack trace
I221215 17:06:35.985586 1415313 jobs/registry.go:1446 â‹® [n1] 780 +  -- stack trace:
I221215 17:06:35.985586 1415313 jobs/registry.go:1446 â‹® [n1] 780 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
I221215 17:06:35.985586 1415313 jobs/registry.go:1446 â‹® [n1] 780 +  | 	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1444
I221215 17:06:35.985586 1415313 jobs/registry.go:1446 â‹® [n1] 780 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
I221215 17:06:35.985586 1415313 jobs/registry.go:1446 â‹® [n1] 780 +  | 	github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:412
I221215 17:06:35.985586 1415313 jobs/registry.go:1446 â‹® [n1] 780 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1
I221215 17:06:35.985586 1415313 jobs/registry.go:1446 â‹® [n1] 780 +  | 	github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:332
I221215 17:06:35.985586 1415313 jobs/registry.go:1446 â‹® [n1] 780 +  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
I221215 17:06:35.985586 1415313 jobs/registry.go:1446 â‹® [n1] 780 +  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470

cc @cockroachdb/sql-schema

srosenberg · 2022-12-16T16:00:03Z

The other is in the migration itself (create-stmt-diag-reqs-v3-index), making the test fail.

I thought the fix [1] from @ajwerner would have resolved it?

[1] #93487

cockroach-teamcity · 2022-12-16T17:06:04Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ dcdf599e9eaee8010137167f419509ccb7627406:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(test_impl.go:291).Fatal: 2: expected version 1000022.2-16, got 1000022.1-18
(test_impl.go:291).Fatal: monitor failure: monitor task failed: output in run_165423.625240424_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-17T17:23:26Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 7c8e7012b73a598f3aa1da9bbf422e833706f5d3:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(test_impl.go:291).Fatal: 1: expected version 1000022.2-20, got 1000022.1-18
(test_impl.go:291).Fatal: monitor failure: monitor task failed: output in run_171134.050486516_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-18T17:24:17Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 93ed65565357538c9048ff45c878a493f2ed9b45:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(test_impl.go:291).Fatal: 4: expected version 1000022.2-20, got 1000022.1-18
(test_impl.go:291).Fatal: monitor failure: monitor task failed: output in run_171244.212814318_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-19T18:02:20Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 94e03c016955dfd64d4e15358ea92226a8f362aa:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(test_impl.go:291).Fatal: 3: expected version 1000022.2-20, got 1000022.1-18
(test_impl.go:291).Fatal: monitor failure: monitor task failed: output in run_175045.604210776_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=zfs , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-20T17:14:46Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 61f7c816574848f804f1adf7f42f7315bfcaaeba:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(test_impl.go:291).Fatal: 2: expected version 1000022.2-20, got 1000022.1-18
(test_impl.go:291).Fatal: monitor failure: monitor task failed: output in run_170313.440444325_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-21T17:57:14Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 10266a323f94c3cf397d5de590e512d987e63e22:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(test_impl.go:291).Fatal: 1: expected version 1000022.2-20, got 1000022.1-18
(test_impl.go:291).Fatal: monitor failure: monitor task failed: output in run_174533.316484357_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-22T17:23:41Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 15bc0c4780681d95e705c22dbe2e7d8d190054ef:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(test_impl.go:291).Fatal: 2: expected version 1000022.2-22, got 1000022.1-18
(test_impl.go:291).Fatal: monitor failure: monitor task failed: output in run_171155.822538377_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-23T17:44:31Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 4232883add85a151c423c45904ac4096d04656c5:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(test_impl.go:291).Fatal: 4: expected version 1000022.2-22, got 1000022.1-18
(test_impl.go:291).Fatal: monitor failure: monitor task failed: output in run_173300.609397712_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=zfs , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-24T16:40:58Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ c4bde8b72cdd4016845ae70ef5162b3f11fab1fb:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(test_impl.go:291).Fatal: 4: expected version 1000022.2-22, got 1000022.1-18
(test_impl.go:291).Fatal: monitor failure: monitor task failed: output in run_162921.443026955_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-25T16:42:32Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ c4bde8b72cdd4016845ae70ef5162b3f11fab1fb:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(test_impl.go:291).Fatal: 4: expected version 1000022.2-22, got 1000022.1-18
(test_impl.go:291).Fatal: monitor failure: monitor task failed: output in run_163048.742621611_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-26T17:28:56Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ c4bde8b72cdd4016845ae70ef5162b3f11fab1fb:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(test_impl.go:291).Fatal: 4: expected version 1000022.2-22, got 1000022.1-18
(test_impl.go:291).Fatal: monitor failure: monitor task failed: output in run_171722.466602735_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=zfs , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-27T16:44:09Z

roachtest.tpcc/mixed-headroom/n5cpu16 failed with artifacts on master @ 9c5375f6a7375724cdbcbaa0029ed97a230d7abe:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/n5cpu16/run_1
(test_impl.go:291).Fatal: 1: expected version 1000022.2-22, got 1000022.1-16
(test_impl.go:291).Fatal: monitor failure: monitor task failed: output in run_163239.555324330_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/mixed-headroom/n5cpu16 failed [OOM during import while running 21.2] #74892 roachtest: tpcc/mixed-headroom/n5cpu16 failed [OOM during import while running 21.2] [C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-21.2]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-27T17:00:38Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 9c5375f6a7375724cdbcbaa0029ed97a230d7abe:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(test_impl.go:291).Fatal: 4: expected version 1000022.2-22, got 1000022.1-18
(test_impl.go:291).Fatal: monitor failure: monitor task failed: output in run_164903.795879750_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

srosenberg · 2022-12-28T00:48:16Z

The other is in the migration itself (create-stmt-diag-reqs-v3-index), making the test fail.

I thought the fix [1] from @ajwerner would have resolved it?

[1] #93487

@ajwerner We're still seeing the same failure mode in the nightly runs. Furthermore, we expect systemschema/validate-after-version-upgrade to also fail in this case, right?

cockroach-teamcity · 2022-12-28T16:10:24Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 6723e00a46aaa3ea575093bd82a02b7d6f6b131b:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(test_impl.go:291).Fatal: 4: expected version 1000022.2-22, got 1000022.1-18
(test_impl.go:291).Fatal: monitor failure: monitor task failed: output in run_155840.873407413_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-29T16:51:15Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 64e4fc9faa4e0ab19fe5ba78f053bc2b1390cb5e:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:515).func1: 2: expected version 1000022.2-22, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_163940.682852727_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-30T16:55:58Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 250238cd29102391dddbc8cc71380090c49ce509:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:515).func1: 2: expected version 1000022.2-22, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_164418.278123745_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-31T15:55:58Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 0725273ac7f789ba8ed78aacaf73cc953ca47fe8:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:515).func1: 4: expected version 1000022.2-22, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_154420.592326253_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-01T15:41:51Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 0725273ac7f789ba8ed78aacaf73cc953ca47fe8:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:515).func1: 2: expected version 22.2, got 22.1-52
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_152856.953591401_n5_cockroach_workload_run_tpcc: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=10m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

srosenberg · 2023-01-18T17:13:35Z

I think it should be fixed once that backport is released and the released version becomes the previous version.

Yep, thanks for the clarification. In my dream world it would have been resolved immediately after the backport was merged. In the real world, it won't happen until the next patch release which also coincides with the update of

cockroach/pkg/cmd/roachtest/tests/predecessor_version.json

Line 12 in 51005e4

"23.1": "22.2.2"

cockroach-teamcity · 2023-01-19T17:06:19Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 942b55ef3f329d2e5e8142c8fc5282ed56173ea7:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:534).func1: 3: expected version 1000022.2-30, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_165434.113259379_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-20T17:10:02Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ d698e5eaf36bee19901beeff7db3425315402907:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:534).func1: 2: expected version 1000022.2-30, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_165817.880577796_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-21T16:59:37Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 2861a4ee05fac1cc13f87bf00f368a65ea8700dc:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:318).func1: 3: expected cluster version 1000022.2-30, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_164759.664382629_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-22T17:19:30Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ b21379bb56dd206e6f63cc7d07ca72e85db7a4c4:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:318).func1: 2: expected cluster version 1000022.2-30, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_170752.370578898_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-23T16:51:31Z

roachtest.tpcc/mixed-headroom/n5cpu16 failed with artifacts on master @ b21379bb56dd206e6f63cc7d07ca72e85db7a4c4:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/n5cpu16/run_1
(versionupgrade.go:318).func1: 2: expected cluster version 1000022.2-30, got 1000022.2-4
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_163954.329917868_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/mixed-headroom/n5cpu16 failed [OOM during import while running 21.2] #74892 roachtest: tpcc/mixed-headroom/n5cpu16 failed [OOM during import while running 21.2] [C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-21.2]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-23T17:03:10Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ b21379bb56dd206e6f63cc7d07ca72e85db7a4c4:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:318).func1: 3: expected cluster version 1000022.2-30, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_165115.993144281_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-24T17:09:30Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ dc974cd698364a4db5acb630162dd8b1856cfad6:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:318).func1: 4: expected cluster version 1000022.2-30, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_165756.621252376_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-25T17:39:03Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 6eabc2f348710f4146ff984bc346cd8cbe96f83b:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:318).func1: 2: expected cluster version 1000022.2-32, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_172726.552910927_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=zfs , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-26T17:49:25Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 2ad8df3df3272110705984efc32f1453631ce602:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:318).func1: 4: expected cluster version 1000022.2-34, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_173741.042447171_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=zfs , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-27T17:11:27Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 95a3a11f11b09481bbdf56820eea4420120f04f9:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:318).func1: 2: expected cluster version 1000022.2-34, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_165950.836846061_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-28T17:16:15Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 78fe59d0536be35fcec68280142746fb91515551:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:318).func1: 4: expected cluster version 1000022.2-34, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_170437.312506071_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-28T17:40:00Z

roachtest.tpcc/mixed-headroom/n5cpu16 failed with artifacts on master @ 78fe59d0536be35fcec68280142746fb91515551:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/n5cpu16/run_1
(versionupgrade.go:318).func1: 4: expected cluster version 1000022.2-34, got 1000022.2-12
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_172824.032197693_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=zfs , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/mixed-headroom/n5cpu16 failed [OOM during import while running 21.2] #74892 roachtest: tpcc/mixed-headroom/n5cpu16 failed [OOM during import while running 21.2] [C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-21.2]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-29T17:47:57Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 69dd453d0e61e258f402c5751de310405743cd18:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:318).func1: 3: expected cluster version 1000022.2-34, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_173619.215606140_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=zfs , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-30T17:07:49Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 69dd453d0e61e258f402c5751de310405743cd18:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:318).func1: 4: expected cluster version 1000022.2-34, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_165606.678584782_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T17:39:34Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 1fb7236f7b1f53e28bde833c28bed83a976138cf:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:318).func1: 1: expected cluster version 1000022.2-34, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_172757.905904183_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=zfs , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-02-01T17:50:32Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ b59fa09d971e92b8b5bcc874437159ef252e0313:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:318).func1: 4: expected cluster version 1000022.2-34, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_173854.905102190_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-02-02T18:29:34Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 22244a780dcfaca48162dde8e0f90b5ba9b6bb9c:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:318).func1: 2: expected cluster version 1000022.2-34, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_181743.116948362_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-02-03T18:02:44Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 44d9f3c8b7bd46839187cea69eaec640c080ac05:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:318).func1: 2: expected cluster version 1000022.2-36, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_175106.410685884_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-02-04T18:25:42Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 5fbcd8a8deac0205c7df38e340c1eb9692854383:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:318).func1: 2: expected cluster version 1000022.2-36, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_181405.764581848_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-02-05T18:19:14Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 8e24570fa366ed038c6ae65f50db5d8e22826db0:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:318).func1: 1: expected cluster version 1000022.2-36, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_180739.761141073_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-02-06T18:56:06Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 8e24570fa366ed038c6ae65f50db5d8e22826db0:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:318).func1: 2: expected cluster version 1000022.2-36, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_184424.288093143_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-02-07T18:36:42Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ eb158026c50d8fa856e42f928d844831ea9e6b28:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(versionupgrade.go:318).func1: 4: expected cluster version 1000022.2-36, got 1000022.1-18
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_182501.928590394_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-02-09T20:33:21Z

roachtest.tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed with artifacts on master @ 09188370d82e163ff1d44c62fe611104502c548d:

test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_191239.288875628_n5_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=1h40m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-4} returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_191239.965931834_n5_cockroach-workload-r.log: exit status 1
(mixed_version_jobs.go:75).wait: monitor failure: monitor task failed: t.Fatal() was called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=zfs , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

renatolabs · 2023-02-15T00:06:20Z

Closing this issue now that the min binary version has been bumped (#96656) and a new patch version for 22.2 has been released (#97060).

srosenberg · 2023-02-22T16:04:28Z

The above failure mode is the same as seen in [1]; n2 invoked,

log.Fatalf(ctx, "finishing proposal with outstanding reproposal at a higher max lease index: %+v", cmd)

[1] #97102

cockroach-teamcity added this to the 23.1 milestone Dec 14, 2022

blathers-crl bot added the T-testeng TestEng Team label Dec 14, 2022

cockroach-teamcity mentioned this issue Feb 10, 2023

roachtest: tpcc/mixed-headroom/n5cpu16 failed #96923

Closed

renatolabs closed this as completed Feb 15, 2023

renatolabs mentioned this issue Mar 31, 2023

sql: inconsistent scan can return error cannot specify timestamp older than 5m0s for this operation if cluster is overloaded #100304

Open

roachtest: tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed #93623

roachtest: tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed #93623

Comments

cockroach-teamcity commented Dec 14, 2022 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Dec 15, 2022

renatolabs commented Dec 16, 2022

srosenberg commented Dec 16, 2022 • edited Loading

cockroach-teamcity commented Dec 16, 2022

cockroach-teamcity commented Dec 17, 2022

cockroach-teamcity commented Dec 18, 2022

cockroach-teamcity commented Dec 19, 2022

cockroach-teamcity commented Dec 20, 2022

cockroach-teamcity commented Dec 21, 2022

cockroach-teamcity commented Dec 22, 2022

cockroach-teamcity commented Dec 23, 2022

cockroach-teamcity commented Dec 24, 2022

cockroach-teamcity commented Dec 25, 2022

cockroach-teamcity commented Dec 26, 2022

cockroach-teamcity commented Dec 27, 2022

cockroach-teamcity commented Dec 27, 2022

srosenberg commented Dec 28, 2022

cockroach-teamcity commented Dec 28, 2022

cockroach-teamcity commented Dec 29, 2022

cockroach-teamcity commented Dec 30, 2022

cockroach-teamcity commented Dec 31, 2022

cockroach-teamcity commented Jan 1, 2023

srosenberg commented Jan 18, 2023 • edited Loading

cockroach-teamcity commented Jan 19, 2023

cockroach-teamcity commented Jan 20, 2023

cockroach-teamcity commented Jan 21, 2023

cockroach-teamcity commented Jan 22, 2023

cockroach-teamcity commented Jan 23, 2023

cockroach-teamcity commented Jan 23, 2023

cockroach-teamcity commented Jan 24, 2023

cockroach-teamcity commented Jan 25, 2023

cockroach-teamcity commented Jan 26, 2023

cockroach-teamcity commented Jan 27, 2023

cockroach-teamcity commented Jan 28, 2023

cockroach-teamcity commented Jan 28, 2023

cockroach-teamcity commented Jan 29, 2023

cockroach-teamcity commented Jan 30, 2023

cockroach-teamcity commented Jan 31, 2023

cockroach-teamcity commented Feb 1, 2023

cockroach-teamcity commented Feb 2, 2023

cockroach-teamcity commented Feb 3, 2023

cockroach-teamcity commented Feb 4, 2023

cockroach-teamcity commented Feb 5, 2023

cockroach-teamcity commented Feb 6, 2023

cockroach-teamcity commented Feb 7, 2023

cockroach-teamcity commented Feb 9, 2023

renatolabs commented Feb 15, 2023

srosenberg commented Feb 22, 2023

cockroach-teamcity commented Dec 14, 2022 •

edited by cockroach-jira-scripts

Loading

srosenberg commented Dec 16, 2022 •

edited

Loading

srosenberg commented Jan 18, 2023 •

edited

Loading