roachtest: schemachange/indexrollback/tpcc/w=1000 failed #36321

cockroach-teamcity · 2019-03-29T03:49:26Z

SHA: https://github.com/cockroachdb/cockroach/commits/3aadd20bbf0940ef65f8b2cdcda498401ba5d9c6

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1206925&tab=buildLog

The test failed on release-19.1:
	cluster.go:1293,tpcc.go:130,cluster.go:1631,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1206925-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		    2.1      0.0      0.0      0.0      0.0 delivery
		  50m50s    68081            0.0           20.5      0.0      0.0      0.0      0.0 newOrder
		  50m50s    68081            0.0            2.1      0.0      0.0      0.0      0.0 orderStatus
		  50m50s    68081            0.0           18.9      0.0      0.0      0.0      0.0 payment
		  50m50s    68081            0.0            2.0      0.0      0.0      0.0      0.0 stockLevel
		E190329 03:45:02.793850 1 workload/cli/run.go:420  error in newOrder: dial tcp 10.128.0.30:26257: connect: connection refused
		  50m51s    93052            0.0            2.1      0.0      0.0      0.0      0.0 delivery
		  50m51s    93052            0.0           20.5      0.0      0.0      0.0      0.0 newOrder
		  50m51s    93052            0.0            2.1      0.0      0.0      0.0      0.0 orderStatus
		  50m51s    93052            1.0           18.9  42949.7  42949.7  42949.7  42949.7 payment
		  50m51s    93052            0.0            2.0      0.0      0.0      0.0      0.0 stockLevel
		: signal: killed
	cluster.go:1652,tpcc.go:140,schemachange.go:375,test.go:1223: unexpected node event: 1: dead
	cluster.go:953,context.go:90,cluster.go:942,asm_amd64.s:522,panic.go:397,test.go:774,test.go:760,cluster.go:1652,tpcc.go:140,schemachange.go:375,test.go:1223: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1206925-schemachange-indexrollback-tpcc-w-1000 --oneshot --ignore-empty-nodes: exit status 1

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2019-03-30T14:28:39Z

SHA: https://github.com/cockroachdb/cockroach/commits/23f9707873abbd2de91a42055535529d7ff296ce

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1209900&tab=buildLog

The test failed on release-19.1:
	cluster.go:1293,tpcc.go:130,cluster.go:1631,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1209900-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		       0            0.0            3.1      0.0      0.0      0.0      0.0 stockLevel
		   9m43s        0            5.0            3.2   2281.7   2952.8   2952.8   2952.8 delivery
		   9m43s        0           35.9           28.3   3355.4  85899.3 103079.2 103079.2 newOrder
		   9m43s        0            3.0            3.1   1275.1   1476.4   1476.4   1476.4 orderStatus
		   9m43s        0           32.9           23.8   3087.0 103079.2 103079.2 103079.2 payment
		   9m43s        0            2.0            3.1   1543.5   1879.0   1879.0   1879.0 stockLevel
		   9m44s        0            0.0            3.2      0.0      0.0      0.0      0.0 delivery
		   9m44s        0           21.0           28.3  11274.3 103079.2 103079.2 103079.2 newOrder
		   9m44s        0            2.0            3.1    805.3    973.1    973.1    973.1 orderStatus
		   9m44s        0           20.0           23.7   2080.4 103079.2 103079.2 103079.2 payment
		   9m44s        0            3.0            3.1    906.0    906.0    906.0    906.0 stockLevel
		: signal: killed
	cluster.go:1652,tpcc.go:140,schemachange.go:375,test.go:1223: sql: no rows in result set

vivekmenezes · 2019-04-01T14:27:45Z

the latest failure seems to be a situation where the query : SELECT status FROM [SHOW JOBS] WHERE job_id = $1 given the rollback job id failed. It's a harmless failure but we should look into finding out why.

cockroach-teamcity · 2019-04-03T01:43:03Z

SHA: https://github.com/cockroachdb/cockroach/commits/877ebd1ece299b9ee621aa0d091657621593d844

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1215372&tab=buildLog

The test failed on release-19.1:
	cluster.go:1329,tpcc.go:130,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1215372-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		.0      0.0 newOrder
		   7m37s    63516            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   7m37s    63516            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   7m37s    63516            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		E190402 14:16:51.182444 1 workload/cli/run.go:420  error in newOrder: dial tcp 10.142.0.67:26257: connect: connection refused
		   7m38s   103769            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   7m38s   103769            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   7m38s   103769            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   7m38s   103769            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   7m38s   103769            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		E190402 14:16:52.181646 1 workload/cli/run.go:420  error in payment: dial tcp 10.142.0.67:26257: connect: connection refused
		: signal: killed
	test.go:1214: test timed out (3h0m0s)
	cluster.go:1688,tpcc.go:140,schemachange.go:375,test.go:1226: unexpected node event: 3: dead
	cluster.go:1405,cluster.go:1424,cluster.go:1528,cluster.go:968,asm_amd64.s:522,panic.go:397,test.go:774,test.go:760,cluster.go:1688,tpcc.go:140,schemachange.go:375,test.go:1226: context canceled

cockroach-teamcity · 2019-04-05T14:13:25Z

SHA: https://github.com/cockroachdb/cockroach/commits/b88a6ce86bfe507e14e1e80fdaefd219b5f0b046

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1222890&tab=buildLog

The test failed on release-19.1:
	cluster.go:1329,tpcc.go:130,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1222890-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		Error: read tcp 10.142.0.77:54644->10.142.0.68:26257: read: connection reset by peer
		Error:  exit status 1
		: exit status 1
	cluster.go:1688,tpcc.go:140,schemachange.go:375,test.go:1228: Goexit() was called

cockroach-teamcity · 2019-04-09T15:50:18Z

SHA: https://github.com/cockroachdb/cockroach/commits/c03c307b1a4100a8a1edd2804bdfaf3903097756

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1229036&tab=buildLog

The test failed on release-19.1:
	test.go:1225: test timed out (3h0m0s)
	cluster.go:1329,tpcc.go:169,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1229036-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		eating load: dial tcp :26257: connect: connection refused
		I190409 15:46:27.943959 1 workload/cli/run.go:337  retrying after error while creating load: dial tcp :26257: connect: connection refused
		I190409 15:46:39.205529 1 workload/cli/run.go:337  retrying after error while creating load: dial tcp :26257: connect: connection refused
		I190409 15:46:51.616948 1 workload/cli/run.go:337  retrying after error while creating load: dial tcp :26257: connect: connection refused
		I190409 15:47:04.566491 1 workload/cli/run.go:337  retrying after error while creating load: dial tcp :26257: connect: connection refused
		I190409 15:47:16.429664 1 workload/cli/run.go:337  retrying after error while creating load: dial tcp :26257: connect: connection refused
		I190409 15:47:27.551983 1 workload/cli/run.go:337  retrying after error while creating load: dial tcp :26257: connect: connection refused
		I190409 15:47:39.100901 1 workload/cli/run.go:337  retrying after error while creating load: dial tcp :26257: connect: connection refused
		: signal: killed
	cluster.go:1688,tpcc.go:179,schemachange.go:375,test.go:1237: Goexit() was called

cockroach-teamcity · 2019-04-11T15:03:42Z

SHA: https://github.com/cockroachdb/cockroach/commits/bf399d2677783dc1eea7f5ede6d4561f95c0ea10

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1234662&tab=buildLog

The test failed on master:
	cluster.go:1329,tpcc.go:169,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1234662-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		Initializing 10000 connections...
		Initializing 10000 workers and preparing statements...
		: signal: killed
	cluster.go:1688,tpcc.go:179,schemachange.go:375,test.go:1237: pq: crdb-internal-jobs-table: no inbound stream connection

cockroach-teamcity · 2019-04-12T04:15:59Z

SHA: https://github.com/cockroachdb/cockroach/commits/134478e4dde16919eb86efb81fb22d8cce8a9701

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1236263&tab=buildLog

The test failed on release-19.1:
	cluster.go:1329,tpcc.go:169,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1236263-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		n.go:428  error in newOrder: dial tcp 10.128.0.109:26257: connect: connection refused
		   4m47s     6380            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   4m47s     6380            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   4m47s     6380            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   4m47s     6380            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   4m47s     6380            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		   4m48s     6380            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   4m48s     6380            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   4m48s     6380            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   4m48s     6380            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   4m48s     6380            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		: signal: killed
	cluster.go:1688,tpcc.go:179,schemachange.go:375,test.go:1237: unexpected node event: 4: dead
	cluster.go:953,context.go:90,cluster.go:942,asm_amd64.s:522,panic.go:397,test.go:785,test.go:771,cluster.go:1688,tpcc.go:179,schemachange.go:375,test.go:1237: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1236263-schemachange-indexrollback-tpcc-w-1000 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		4: dead
		1: 3140
		3: 3306
		2: 3185
		Error:  4: dead

cockroach-teamcity · 2019-04-12T08:13:23Z

SHA: https://github.com/cockroachdb/cockroach/commits/509c5b130fb1ad0042beb74e083817aa68e4fc92

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1237068&tab=buildLog

The test failed on release-19.1:
	cluster.go:1329,tpcc.go:169,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1237068-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		un.go:428  error in payment: dial tcp 10.128.0.134:26257: connect: connection refused
		   3m51s    11346            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   3m51s    11346            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   3m51s    11346            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   3m51s    11346            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   3m51s    11346            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		   3m52s    11346            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   3m52s    11346            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   3m52s    11346            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   3m52s    11346            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   3m52s    11346            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		: signal: killed
	cluster.go:1688,tpcc.go:179,schemachange.go:375,test.go:1237: unexpected node event: 2: dead
	cluster.go:953,context.go:90,cluster.go:942,asm_amd64.s:522,panic.go:397,test.go:785,test.go:771,cluster.go:1688,tpcc.go:179,schemachange.go:375,test.go:1237: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1237068-schemachange-indexrollback-tpcc-w-1000 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		2: dead
		1: 3011
		4: 3033
		3: 3251
		Error:  2: dead

cockroach-teamcity · 2019-04-18T14:35:44Z

SHA: https://github.com/cockroachdb/cockroach/commits/83de585d331b05a4aa02a65b353bed6bf829b696

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1247383&tab=buildLog

The test failed on master:
	cluster.go:1329,tpcc.go:168,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1247383-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   0.0 delivery
		   6m29s    16869            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   6m29s    16869            0.0            0.4      0.0      0.0      0.0      0.0 orderStatus
		   6m29s    16869            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   6m29s    16869            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		E190418 14:31:53.585047 1 workload/cli/run.go:428  error in newOrder: ERROR: operation "intent_resolver_ir_batcher.sendBatch" timed out after 30s (SQLSTATE XXUUU)
		   6m30s    16870            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   6m30s    16870            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   6m30s    16870            0.0            0.4      0.0      0.0      0.0      0.0 orderStatus
		   6m30s    16870            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   6m30s    16870            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		: signal: killed
	cluster.go:1688,tpcc.go:178,schemachange.go:375,test.go:1237: dial tcp 35.196.209.187:26257: connect: connection refused
	cluster.go:953,context.go:89,cluster.go:942,asm_amd64.s:522,panic.go:397,test.go:785,test.go:771,cluster.go:1688,tpcc.go:178,schemachange.go:375,test.go:1237: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1247383-schemachange-indexrollback-tpcc-w-1000 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		1: dead
		2: 3947
		3: 4174
		4: 3955
		Error:  1: dead

cockroach-teamcity · 2019-04-19T01:49:28Z

SHA: https://github.com/cockroachdb/cockroach/commits/837e946efc272bd8a9e0e08484733f8755ff5ab1

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1247401&tab=buildLog

The test failed on release-19.1:
	cluster.go:1329,tpcc.go:168,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1247401-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		un.go:428  error in newOrder: dial tcp 10.142.0.68:26257: connect: connection refused
		   7m42s    11446            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   7m42s    11446            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   7m42s    11446            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   7m42s    11446            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   7m42s    11446            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		   7m43s    11446            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   7m43s    11446            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   7m43s    11446            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   7m43s    11446            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   7m43s    11446            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		: signal: killed
	test.go:1225: test timed out (3h0m0s)
	cluster.go:1688,tpcc.go:178,schemachange.go:375,test.go:1237: unexpected node event: 4: dead
	cluster.go:1405,cluster.go:1424,cluster.go:1528,cluster.go:968,asm_amd64.s:522,panic.go:397,test.go:785,test.go:771,cluster.go:1688,tpcc.go:178,schemachange.go:375,test.go:1237: context canceled

cockroach-teamcity · 2019-04-22T14:45:00Z

SHA: https://github.com/cockroachdb/cockroach/commits/46f8608c4fe2d94b771beb37bcee19136040fd74

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1253450&tab=buildLog

The test failed on master:
	cluster.go:1329,tpcc.go:168,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1253450-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		operation "intent_resolver_ir_batcher.sendBatch" timed out after 30s (SQLSTATE XXUUU)
		  12m10s    20979            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		  12m10s    20979            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  12m10s    20979            0.0            0.2      0.0      0.0      0.0      0.0 orderStatus
		  12m10s    20979            0.0            0.0      0.0      0.0      0.0      0.0 payment
		  12m10s    20979            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		  12m11s    20979            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		  12m11s    20979            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  12m11s    20979            0.0            0.2      0.0      0.0      0.0      0.0 orderStatus
		  12m11s    20979            0.0            0.0      0.0      0.0      0.0      0.0 payment
		  12m11s    20979            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		: signal: killed
	cluster.go:1688,tpcc.go:178,schemachange.go:375,test.go:1237: unexpected node event: 1: dead

cockroach-teamcity · 2019-04-23T14:47:56Z

SHA: https://github.com/cockroachdb/cockroach/commits/ec4728ae986b46d4f57009233b86971198b275ed

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1255121&tab=buildLog

The test failed on master:
	cluster.go:1329,tpcc.go:168,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1255121-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		.0      0.0      0.0      0.0 orderStatus
		   7m14s    17981            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   7m14s    17981            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		E190423 14:45:20.129701 1 workload/cli/run.go:428  error in newOrder: ERROR: operation "intent_resolver_ir_batcher.sendBatch" timed out after 30s (SQLSTATE XXUUU)
		   7m15s    18008            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   7m15s    18008            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   7m15s    18008            0.0            0.2      0.0      0.0      0.0      0.0 orderStatus
		   7m15s    18008            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   7m15s    18008            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		E190423 14:45:21.157249 1 workload/cli/run.go:428  error in newOrder: ERROR: operation "intent_resolver_ir_batcher.sendBatch" timed out after 30s (SQLSTATE XXUUU)
		: signal: killed
	cluster.go:1688,tpcc.go:178,schemachange.go:375,test.go:1237: unexpected node event: 1: dead
	cluster.go:953,context.go:89,cluster.go:942,asm_amd64.s:522,panic.go:397,test.go:785,test.go:771,cluster.go:1688,tpcc.go:178,schemachange.go:375,test.go:1237: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1255121-schemachange-indexrollback-tpcc-w-1000 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		1: dead
		2: 4198
		4: 4089
		3: 4086
		Error:  1: dead

vivekmenezes · 2019-04-23T15:25:56Z

I'm seeing node liveness problems on this cluster. Will run one by hand and check

cockroach-teamcity · 2019-04-24T14:52:52Z

SHA: https://github.com/cockroachdb/cockroach/commits/a22a3780ebb5482be04b6f31344c764b2fb5f765

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1257361&tab=buildLog

The test failed on release-19.1:
	cluster.go:1423,tpcc.go:168,cluster.go:1761,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1257361-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		l
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   4m57s        0            4.0            3.0   6174.0  22548.6  22548.6  22548.6 delivery
		   4m57s        0           70.2           25.9  38654.7 103079.2 103079.2 103079.2 newOrder
		   4m57s        0            3.0            3.1    125.8    369.1    369.1    369.1 orderStatus
		   4m57s        0           17.0           19.2  98784.2 103079.2 103079.2 103079.2 payment
		   4m57s        0            5.0            2.9   2013.3  30064.8  30064.8  30064.8 stockLevel
		   4m58s        0            3.0            3.0  11274.3  23622.3  23622.3  23622.3 delivery
		   4m58s        0           27.9           25.9  64424.5 103079.2 103079.2 103079.2 newOrder
		   4m58s        0            0.0            3.1      0.0      0.0      0.0      0.0 orderStatus
		   4m58s        0           24.0           19.2  57982.1 103079.2 103079.2 103079.2 payment
		   4m58s        0            1.0            2.9    318.8    318.8    318.8    318.8 stockLevel
		: signal: killed
	cluster.go:1782,tpcc.go:178,schemachange.go:375,test.go:1245: sql: no rows in result set

cockroach-teamcity · 2019-04-25T16:54:02Z

SHA: https://github.com/cockroachdb/cockroach/commits/99306ec3e9fcbba01c05431cbf496e8b5b8954b4

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1260033&tab=buildLog

The test failed on master:
	cluster.go:1423,tpcc.go:168,cluster.go:1761,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1260033-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		d
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  14m25s    18711            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		  14m25s    18711            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  14m25s    18711            0.0            0.1      0.0      0.0      0.0      0.0 orderStatus
		  14m25s    18711            0.0            0.0      0.0      0.0      0.0      0.0 payment
		  14m25s    18711            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		  14m26s    18711            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		  14m26s    18711            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  14m26s    18711            0.0            0.1      0.0      0.0      0.0      0.0 orderStatus
		  14m26s    18711            0.0            0.0      0.0      0.0      0.0      0.0 payment
		  14m26s    18711            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		: signal: killed
	test.go:1231: test timed out (3h0m0s)
	cluster.go:1782,tpcc.go:178,schemachange.go:375,test.go:1245: unexpected node event: 2: dead

36744: storage/rangefeed: use fine-grained locking around Processor, add metrics r=nvanbenschoten a=nvanbenschoten It is critical for correctness that operations like providing the processor with logical ops or informing the processor of closed timestamp updates be properly synchronized via the raftMu. This ensures that events published on the processors eventC accurately reflect the order of events in the Raft log. However, it is not critical that lifecycle events around starting a Rangefeed processor, stopping a Rangefeed processor, and registering with a Rangefeed processor be synchronized with Raft application. This change exploits this to break up the locking around rangefeed.Processor. Using more fine-grained locking opens up the opportunity to interact with the rangefeed Processor without needing to synchronize with the Range's raft application, which can be very expensive. The next commit uses this improvement to add a new `rangefeed_registrations` metric to `RangeInfo`. 36999: roachtest: add schemachange/bulkingest test r=lucy-zhang a=lucy-zhang Add a test to index the random `payload` column for the `bulkingest` workload, to test the index backfiller for an index on randomly ordered values relative to the primary key. Release note: None 37169: opt: don't use delegateQuery for ShowZoneConfig r=RaduBerinde a=RaduBerinde Part of a larger work item to remove `delegateQuery`, as it doesn't work with the optimizer. I could not use the new delegate infrastructure because it doesn't support hiding columns returned by the delegated query. Release note: None 37182: roachtest: provision more cpu for schemachange tpcc tests r=vivekmenezes a=vivekmenezes related to #36094 #36024 #36321 Release note: None Co-authored-by: Nathan VanBenschoten <[email protected]> Co-authored-by: Lucy Zhang <[email protected]> Co-authored-by: Radu Berinde <[email protected]> Co-authored-by: Vivek Menezes <[email protected]>

cockroach-teamcity · 2019-04-30T14:32:45Z

SHA: https://github.com/cockroachdb/cockroach/commits/73765b6d168fb999466756b112fd590747a3a8c4

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1266059&tab=buildLog

The test failed on branch=release-19.1, cloud=gce:
	cluster.go:1476,tpcc.go:168,cluster.go:1814,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1266059-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		l
		   4m20s        0           54.9           28.1   1040.2   5637.1  12348.0  18253.6 delivery
		   4m20s        0          365.3          275.0   2818.6  26843.5  47244.6  85899.3 newOrder
		   4m20s        0           34.9           28.1     48.2   1140.9   3489.7   3489.7 orderStatus
		   4m20s        0          332.4          267.7   2550.1  62277.0 103079.2 103079.2 payment
		   4m20s        0           31.9           27.9    201.3   6710.9  19327.4  19327.4 stockLevel
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   4m21s        0           26.0           28.1   1073.7   4160.7   5100.3   5100.3 delivery
		   4m21s        0          186.2          274.7   1879.0  26843.5  55834.6 103079.2 newOrder
		   4m21s        0           16.0           28.1    151.0   1476.4   8589.9   8589.9 orderStatus
		   4m21s        0          194.2          267.4   2415.9  90194.3 103079.2 103079.2 payment
		   4m21s        0           17.0           27.8    318.8   1543.5   2147.5   2147.5 stockLevel
		: signal: killed
	cluster.go:1835,tpcc.go:178,schemachange.go:375,test.go:1253: sql: no rows in result set

cockroach-teamcity · 2019-04-30T16:51:53Z

SHA: https://github.com/cockroachdb/cockroach/commits/efb45869b242137e5c178b10c646c3ed025fff36

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1266041&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1476,tpcc.go:168,cluster.go:1814,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1266041-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		.0      0.0      0.0 orderStatus
		   2m40s    11822            0.0           16.2      0.0      0.0      0.0      0.0 payment
		   2m40s    11822            0.0            0.2      0.0      0.0      0.0      0.0 stockLevel
		E190430 14:22:12.588196 1 workload/cli/run.go:428  error in payment: dial tcp 10.142.0.74:26257: connect: connection refused
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   2m41s    11915            0.0            2.5      0.0      0.0      0.0      0.0 delivery
		   2m41s    11915            2.0           20.4 103079.2 103079.2 103079.2 103079.2 newOrder
		   2m41s    11915            0.0            3.0      0.0      0.0      0.0      0.0 orderStatus
		   2m41s    11915            0.0           16.1      0.0      0.0      0.0      0.0 payment
		   2m41s    11915            0.0            0.2      0.0      0.0      0.0      0.0 stockLevel
		E190430 14:22:13.588267 1 workload/cli/run.go:428  error in newOrder: dial tcp 10.142.0.74:26257: connect: connection refused
		: signal: killed
	cluster.go:1835,tpcc.go:178,schemachange.go:375,test.go:1253: unexpected node event: 4: dead
	test.go:1239: test timed out (3h0m0s)

cockroach-teamcity · 2019-05-01T15:36:12Z

SHA: https://github.com/cockroachdb/cockroach/commits/24feca7a4106f08c73534e16ebb79d949a479f35

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1268176&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1476,tpcc.go:168,cluster.go:1814,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1268176-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		       0           15.0           22.3    503.3  16643.0  22548.6  22548.6 stockLevel
		    1m2s        0           12.0           22.7   1275.1   2147.5  28991.0  28991.0 delivery
		    1m2s        0          113.8          211.2   1610.6  31138.5  51539.6  60129.5 newOrder
		    1m2s        0           22.0           23.0    285.2  14495.5  17179.9  17179.9 orderStatus
		    1m2s        0          209.7          207.5   1140.9  28991.0  60129.5  62277.0 payment
		    1m2s        0           24.0           22.3    419.4   1543.5  20401.1  20401.1 stockLevel
		    1m3s        0           18.0           22.6   1208.0   3221.2   9663.7   9663.7 delivery
		    1m3s        0          313.3          212.8   1677.7  34359.7  47244.6  55834.6 newOrder
		    1m3s        0           19.0           23.0    285.2  16106.1  28991.0  28991.0 orderStatus
		    1m3s        0          119.1          206.1   1476.4  40802.2  55834.6  60129.5 payment
		    1m3s        0           17.0           22.2    335.5    805.3  21474.8  21474.8 stockLevel
		: signal: killed
	cluster.go:1835,tpcc.go:178,schemachange.go:375,test.go:1251: sql: no rows in result set

cockroach-teamcity · 2019-05-02T14:51:29Z

SHA: https://github.com/cockroachdb/cockroach/commits/9f039a1237bb7571d910b3a3b0e4f2ffc5a84357

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1270756&tab=buildLog

The test failed on branch=release-19.1, cloud=gce:
	cluster.go:1476,tpcc.go:168,cluster.go:1814,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1270756-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		l
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   3m17s        0           33.0           28.1   1208.0   1677.7   1811.9   1811.9 delivery
		   3m17s        0          199.7          270.8   1342.2  15569.3  66572.0  94489.3 newOrder
		   3m17s        0           19.0           27.7    352.3    704.6    872.4    872.4 orderStatus
		   3m17s        0          268.7          266.5   1476.4  73014.4 103079.2 103079.2 payment
		   3m17s        0           13.0           27.6    234.9    453.0    486.5    486.5 stockLevel
		   3m18s        0           23.0           28.0   1275.1   3623.9   3892.3   3892.3 delivery
		   3m18s        0          211.3          270.5   1744.8  22548.6  53687.1 103079.2 newOrder
		   3m18s        0           35.0           27.8    234.9    906.0   1208.0   1208.0 orderStatus
		   3m18s        0          261.4          266.5   1409.3   5368.7  24696.1  81604.4 payment
		   3m18s        0           26.0           27.6    335.5  10200.5  22548.6  22548.6 stockLevel
		: signal: killed
	cluster.go:1835,tpcc.go:178,schemachange.go:375,test.go:1251: sql: no rows in result set

cockroach-teamcity · 2019-05-02T15:38:54Z

SHA: https://github.com/cockroachdb/cockroach/commits/ff969dd6cb0e087327f0b210f728f588b2f5aeb0

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1270738&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1476,tpcc.go:168,cluster.go:1814,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1270738-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		       0           29.0           21.1    151.0  23622.3  28991.0  28991.0 stockLevel
		   1m38s        0           15.0           20.6   1946.2  26843.5  27917.3  27917.3 delivery
		   1m38s        0          254.0          188.9   2684.4  47244.6  77309.4  85899.3 newOrder
		   1m38s        0           14.0           21.5    184.5    604.0  22548.6  22548.6 orderStatus
		   1m38s        0          106.0          180.5   2281.7  47244.6  94489.3  94489.3 payment
		   1m38s        0           17.0           21.0    704.6  42949.7  68719.5  68719.5 stockLevel
		   1m39s        0            9.0           20.4   2013.3  24696.1  24696.1  24696.1 delivery
		   1m39s        0          224.0          189.2   2415.9  49392.1  81604.4  85899.3 newOrder
		   1m39s        0           25.0           21.5    151.0  10737.4  12884.9  12884.9 orderStatus
		   1m39s        0          288.0          181.6   1811.9  57982.1  81604.4  94489.3 payment
		   1m39s        0           25.0           21.1    570.4  23622.3  30064.8  30064.8 stockLevel
		: signal: killed
	cluster.go:1835,tpcc.go:178,schemachange.go:375,test.go:1251: sql: no rows in result set

cockroach-teamcity · 2019-05-03T14:53:40Z

SHA: https://github.com/cockroachdb/cockroach/commits/bb50fd396f6ce79258b744e1f8efa2d1bc9dfbd2

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1272551&tab=buildLog

The test failed on branch=release-19.1, cloud=gce:
	cluster.go:1474,tpcc.go:168,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1272551-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		       0           35.1           32.4    184.5   6979.3  15032.4  15032.4 stockLevel
		   1m46s        0           44.9           32.4    872.4   1543.5   3087.0   3087.0 delivery
		   1m46s        0          390.4          321.1   1476.4  15032.4  30064.8  40802.2 newOrder
		   1m46s        0           30.9           32.4     33.6    130.0    184.5    184.5 orderStatus
		   1m46s        0          393.4          314.8    771.8  28991.0  49392.1  68719.5 payment
		   1m46s        0           33.9           32.4    142.6   1073.7   4563.4   4563.4 stockLevel
		   1m47s        0           41.0           32.5    805.3   1140.9   1275.1   1275.1 delivery
		   1m47s        0          270.2          320.6    771.8  22548.6  27917.3  32212.3 newOrder
		   1m47s        0           33.0           32.4     58.7    352.3    385.9    385.9 orderStatus
		   1m47s        0          276.2          314.4    805.3  15569.3  38654.7  47244.6 payment
		   1m47s        0           46.0           32.5    125.8   6979.3  15569.3  15569.3 stockLevel
		: signal: killed
	cluster.go:1833,tpcc.go:178,schemachange.go:375,test.go:1251: sql: no rows in result set

cockroach-teamcity · 2019-05-03T15:38:49Z

SHA: https://github.com/cockroachdb/cockroach/commits/29bf8a1eb0b5b943127206f49333337f7f3079b4

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1272533&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,tpcc.go:168,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1272533-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		l
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   3m41s        0           22.0           20.5    436.2  16643.0  25769.8  25769.8 delivery
		   3m41s        0          245.9          193.0    906.0  62277.0 103079.2 103079.2 newOrder
		   3m41s        0           23.0           20.4     39.8  13958.6  19327.4  19327.4 orderStatus
		   3m41s        0          225.9          187.7    637.5  57982.1 103079.2 103079.2 payment
		   3m41s        0           34.0           20.1    130.0  42949.7 103079.2 103079.2 stockLevel
		   3m42s        0           14.0           20.5    469.8  15032.4  51539.6  51539.6 delivery
		   3m42s        0          143.8          192.8    805.3  64424.5 103079.2 103079.2 newOrder
		   3m42s        0           18.0           20.4    234.9  51539.6  57982.1  57982.1 orderStatus
		   3m42s        0          178.8          187.7    771.8  42949.7 103079.2 103079.2 payment
		   3m42s        0           21.0           20.1    260.0  24696.1  25769.8  25769.8 stockLevel
		: signal: killed
	cluster.go:1833,tpcc.go:178,schemachange.go:375,test.go:1251: sql: no rows in result set

cockroach-teamcity · 2019-05-04T14:56:51Z

SHA: https://github.com/cockroachdb/cockroach/commits/b0c0444481f40a772798c0f6eef3af2fd6a8ada0

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1274193&tab=buildLog

The test failed on branch=release-19.1, cloud=gce:
	cluster.go:1474,tpcc.go:168,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1274193-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		l
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   3m33s        0           30.0           30.8    503.3   1208.0   1275.1   1275.1 delivery
		   3m33s        0          262.0          301.1    838.9  18253.6  31138.5  45097.2 newOrder
		   3m33s        0           35.0           30.6     62.9    260.0    260.0    260.0 orderStatus
		   3m33s        0          394.9          293.2   1040.2  53687.1 103079.2 103079.2 payment
		   3m33s        0           36.0           30.8    151.0   4563.4  40802.2  40802.2 stockLevel
		   3m34s        0           29.0           30.8   1208.0   3892.3   3892.3   3892.3 delivery
		   3m34s        0          195.9          300.6   1040.2  21474.8  53687.1  68719.5 newOrder
		   3m34s        0           21.0           30.6     65.0    369.1    503.3    503.3 orderStatus
		   3m34s        0          315.9          293.3   1342.2  45097.2 103079.2 103079.2 payment
		   3m34s        0           21.0           30.7    142.6    453.0  32212.3  32212.3 stockLevel
		: signal: killed
	cluster.go:1833,tpcc.go:178,schemachange.go:375,test.go:1251: sql: no rows in result set

cockroach-teamcity · 2019-05-05T01:46:21Z

SHA: https://github.com/cockroachdb/cockroach/commits/856ba9108f112f85d406bbe88d2208651859336e

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1274175&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,tpcc.go:168,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1274175-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		    0.0      0.0      0.0      0.0      0.0 delivery
		   3m31s    10043            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   3m31s    10043            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   3m31s    10043            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   3m31s    10043            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		   3m32s    10043            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   3m32s    10043            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   3m32s    10043            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   3m32s    10043            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   3m32s    10043            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		E190504 14:43:39.885336 1 workload/cli/run.go:428  error in payment: dial tcp 10.142.0.161:26257: connect: connection refused
		: signal: killed
	test.go:1237: test timed out (3h0m0s)
	cluster.go:1833,tpcc.go:178,schemachange.go:375,test.go:1251: unexpected node event: 3: dead
	cluster.go:1550,cluster.go:1569,cluster.go:1673,cluster.go:1053,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1833,tpcc.go:178,schemachange.go:375,test.go:1251: context canceled

cockroach-teamcity · 2019-05-07T14:35:32Z

SHA: https://github.com/cockroachdb/cockroach/commits/23155799e92e54915ae66259d06a630e981afbeb

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1277061&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,tpcc.go:168,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1277061-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		4  15032.4 newOrder
		   1m13s    94392            3.0            5.5   1140.9   7516.2   7516.2   7516.2 orderStatus
		   1m13s    94392           11.0           47.1   3892.3  24696.1  27917.3  27917.3 payment
		   1m13s    94392            0.0            0.5      0.0      0.0      0.0      0.0 stockLevel
		E190507 14:32:07.988144 1 workload/cli/run.go:428  error in payment: dial tcp 10.142.0.165:26257: connect: connection refused
		   1m14s   212773            2.0            4.6  17179.9  22548.6  22548.6  22548.6 delivery
		   1m14s   212773           13.0           40.1  13958.6  19327.4  36507.2  36507.2 newOrder
		   1m14s   212773            3.0            5.4   1946.2   5100.3   5100.3   5100.3 orderStatus
		   1m14s   212773           26.0           46.8   6174.0  22548.6  51539.6  51539.6 payment
		   1m14s   212773            0.0            0.5      0.0      0.0      0.0      0.0 stockLevel
		E190507 14:32:08.988142 1 workload/cli/run.go:428  error in payment: dial tcp 10.142.0.165:26257: connect: connection refused
		: signal: killed
	cluster.go:1833,tpcc.go:178,schemachange.go:375,test.go:1251: unexpected node event: 4: dead
	cluster.go:1038,context.go:89,cluster.go:1027,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1833,tpcc.go:178,schemachange.go:375,test.go:1251: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1277061-schemachange-indexrollback-tpcc-w-1000 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		4: dead
		2: 4446
		1: 4481
		3: 4463
		Error:  4: dead

cockroach-teamcity · 2019-05-07T14:36:39Z

SHA: https://github.com/cockroachdb/cockroach/commits/72d089cfc1a0b67c299439d328587345f1125b74

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1277079&tab=buildLog

The test failed on branch=release-19.1, cloud=gce:
	cluster.go:1474,tpcc.go:168,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1277079-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		l
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		     41s        0           38.9           29.8   2281.7   3355.4   3489.7   3489.7 delivery
		     41s        0          524.1          305.6   1342.2   5100.3   6442.5   9126.8 newOrder
		     41s        0           38.9           31.7     30.4    113.2    142.6    142.6 orderStatus
		     41s        0          281.5          296.9    402.7   5100.3   8589.9  11811.2 payment
		     41s        0           38.9           30.8    100.7   1409.3   4563.4   4563.4 stockLevel
		     42s        0           54.1           30.4   1006.6   1946.2   2952.8   3221.2 delivery
		     42s        0          433.7          308.6    704.6   3221.2   8053.1  10737.4 newOrder
		     42s        0           33.1           31.7     18.9     37.7     50.3     50.3 orderStatus
		     42s        0          300.5          297.0    419.4   5905.6  10737.4  16106.1 payment
		     42s        0           31.0           30.8     96.5    385.9   1275.1   1275.1 stockLevel
		: signal: killed
	cluster.go:1833,tpcc.go:178,schemachange.go:375,test.go:1251: sql: no rows in result set

cockroach-teamcity · 2019-05-08T15:11:53Z

SHA: https://github.com/cockroachdb/cockroach/commits/8abb47a1c9795c1463183bc44e776b054bece682

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1279683&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,tpcc.go:168,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1279683-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		run.go:428  error in newOrder: dial tcp 10.142.0.5:26257: connect: connection refused
		   6m10s     9455            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   6m10s     9455            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   6m10s     9455            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   6m10s     9455            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   6m10s     9455            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		   6m11s     9455            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   6m11s     9455            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   6m11s     9455            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   6m11s     9455            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   6m11s     9455            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		: signal: killed
	cluster.go:1833,tpcc.go:178,schemachange.go:375,test.go:1251: read tcp 172.17.0.2:43292->34.74.137.171:26257: read: connection reset by peer

vivekmenezes · 2019-05-09T17:43:44Z

this test has been deleted,

cockroach-teamcity added this to the 19.1 milestone Mar 29, 2019

cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Mar 29, 2019

tbg assigned vivekmenezes Apr 1, 2019

vivekmenezes mentioned this issue Apr 29, 2019

roachtest: provision more cpu for schemachange tpcc tests #37182

Merged

vivekmenezes closed this as completed May 9, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: schemachange/indexrollback/tpcc/w=1000 failed #36321

roachtest: schemachange/indexrollback/tpcc/w=1000 failed #36321

cockroach-teamcity commented Mar 29, 2019

cockroach-teamcity commented Mar 30, 2019

vivekmenezes commented Apr 1, 2019

cockroach-teamcity commented Apr 3, 2019

cockroach-teamcity commented Apr 5, 2019

cockroach-teamcity commented Apr 9, 2019

cockroach-teamcity commented Apr 11, 2019

cockroach-teamcity commented Apr 12, 2019

cockroach-teamcity commented Apr 12, 2019

cockroach-teamcity commented Apr 18, 2019

cockroach-teamcity commented Apr 19, 2019

cockroach-teamcity commented Apr 22, 2019

cockroach-teamcity commented Apr 23, 2019

vivekmenezes commented Apr 23, 2019

cockroach-teamcity commented Apr 24, 2019

cockroach-teamcity commented Apr 25, 2019

cockroach-teamcity commented Apr 30, 2019

cockroach-teamcity commented Apr 30, 2019

cockroach-teamcity commented May 1, 2019

cockroach-teamcity commented May 2, 2019

cockroach-teamcity commented May 2, 2019

cockroach-teamcity commented May 3, 2019

cockroach-teamcity commented May 3, 2019

cockroach-teamcity commented May 4, 2019

cockroach-teamcity commented May 5, 2019

cockroach-teamcity commented May 7, 2019

cockroach-teamcity commented May 7, 2019

cockroach-teamcity commented May 8, 2019

vivekmenezes commented May 9, 2019

roachtest: schemachange/indexrollback/tpcc/w=1000 failed #36321

roachtest: schemachange/indexrollback/tpcc/w=1000 failed #36321

Comments

cockroach-teamcity commented Mar 29, 2019

cockroach-teamcity commented Mar 30, 2019

vivekmenezes commented Apr 1, 2019

cockroach-teamcity commented Apr 3, 2019

cockroach-teamcity commented Apr 5, 2019

cockroach-teamcity commented Apr 9, 2019

cockroach-teamcity commented Apr 11, 2019

cockroach-teamcity commented Apr 12, 2019

cockroach-teamcity commented Apr 12, 2019

cockroach-teamcity commented Apr 18, 2019

cockroach-teamcity commented Apr 19, 2019

cockroach-teamcity commented Apr 22, 2019

cockroach-teamcity commented Apr 23, 2019

vivekmenezes commented Apr 23, 2019

cockroach-teamcity commented Apr 24, 2019

cockroach-teamcity commented Apr 25, 2019

cockroach-teamcity commented Apr 30, 2019

cockroach-teamcity commented Apr 30, 2019

cockroach-teamcity commented May 1, 2019

cockroach-teamcity commented May 2, 2019

cockroach-teamcity commented May 2, 2019

cockroach-teamcity commented May 3, 2019

cockroach-teamcity commented May 3, 2019

cockroach-teamcity commented May 4, 2019

cockroach-teamcity commented May 5, 2019

cockroach-teamcity commented May 7, 2019

cockroach-teamcity commented May 7, 2019

cockroach-teamcity commented May 8, 2019

vivekmenezes commented May 9, 2019