roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [#37866] #37250

cockroach-teamcity · 2019-05-01T19:48:37Z

SHA: https://github.com/cockroachdb/cockroach/commits/24feca7a4106f08c73534e16ebb79d949a479f35

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=6/cpu=16/multi-az PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1268176&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1835,tpcc.go:853,tpcc.go:554,test.go:1251: unexpected node event: 4: dead
	cluster.go:1038,context.go:89,cluster.go:1027,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1835,tpcc.go:853,tpcc.go:554,test.go:1251: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1268176-tpccbench-nodes-6-cpu-16-multi-az --oneshot --ignore-empty-nodes: exit status 1 3: skipped
		4: dead
		6: 3679
		5: 3748
		2: 3684
		1: 4489
		7: 3645
		Error:  4: dead

The text was updated successfully, but these errors were encountered:

andreimatei · 2019-05-02T19:38:10Z

Node died because of no disk space left :S
@nvanbenschoten ?

cockroach-teamcity · 2019-05-08T09:28:51Z

SHA: https://github.com/cockroachdb/cockroach/commits/d554884a4e474cc06213230d5ba7d757a88e9e46

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=6/cpu=16/multi-az PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1279548&tab=buildLog

The test failed on branch=release-2.1, cloud=gce:
	cluster.go:1833,tpcc.go:853,tpcc.go:554,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1279548-tpccbench-nodes-6-cpu-16-multi-az:3 -- ./workload fixtures load tpcc --warehouses=5000 --split --scatter --checks=false {pgurl:1} returned:
		stderr:
		
		stdout:
		3 +0x1 fp=0xc000c227e0 sp=0xc000c227d8 pc=0x466f61
		created by runtime.(*timersBucket).addtimerLocked
			/usr/local/go/src/runtime/time.go:170 +0x114
		
		goroutine 978 [timer goroutine (idle)]:
		runtime.gopark(0x15ed1a8, 0x28cb3c0, 0x8cb17471414, 0x1)
			/usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000529728 sp=0xc000529708 pc=0x438f8b
		runtime.goparkunlock(0x28cb3c0, 0xc000811414, 0x1)
			/usr/local/go/src/runtime/proc.go:308 +0x53 fp=0xc000529758 sp=0xc000529728 pc=0x439033
		runtime.timerproc(0x28cb3c0)
			/usr/local/go/src/runtime/time.go:280 +0x288 fp=0xc0005297d8 sp=0xc000529758 pc=0x456168
		runtime.goexit()
			/usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0005297e0 sp=0xc0005297d8 pc=0x466f61
		created by runtime.(*timersBucket).addtimerLocked
			/usr/local/go/src/runtime/time.go:170 +0x114
		bash: line 1:  3622 Aborted                 (core dumped) bash -c "./workload fixtures load tpcc --warehouses=5000 --split --scatter --checks=false 'postgres://[email protected]:26257?sslmode=disable'"
		Error:  exit status 134
		: exit status 1

danhhz · 2019-05-08T16:40:13Z

Most recent failure is fixed by #37401

cockroach-teamcity · 2019-05-08T17:39:25Z

SHA: https://github.com/cockroachdb/cockroach/commits/8abb47a1c9795c1463183bc44e776b054bece682

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=6/cpu=16/multi-az PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1279683&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1833,tpcc.go:853,tpcc.go:554,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1279683-tpccbench-nodes-6-cpu-16-multi-az:3 -- ./workload fixtures load tpcc --warehouses=5000 --split --scatter --checks=false {pgurl:1} returned:
		stderr:
		
		stdout:
		3 +0x1 fp=0xc0007707e0 sp=0xc0007707d8 pc=0x466f61
		created by runtime.(*timersBucket).addtimerLocked
			/usr/local/go/src/runtime/time.go:170 +0x114
		
		goroutine 562 [timer goroutine (idle)]:
		runtime.gopark(0x15ed1a8, 0x28cb0c0, 0x59d2a7d1414, 0x1)
			/usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000b0d728 sp=0xc000b0d708 pc=0x438f8b
		runtime.goparkunlock(0x28cb0c0, 0xc000881414, 0x1)
			/usr/local/go/src/runtime/proc.go:308 +0x53 fp=0xc000b0d758 sp=0xc000b0d728 pc=0x439033
		runtime.timerproc(0x28cb0c0)
			/usr/local/go/src/runtime/time.go:280 +0x288 fp=0xc000b0d7d8 sp=0xc000b0d758 pc=0x456168
		runtime.goexit()
			/usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000b0d7e0 sp=0xc000b0d7d8 pc=0x466f61
		created by runtime.(*timersBucket).addtimerLocked
			/usr/local/go/src/runtime/time.go:170 +0x114
		bash: line 1:  4248 Aborted                 (core dumped) bash -c "./workload fixtures load tpcc --warehouses=5000 --split --scatter --checks=false 'postgres://[email protected]:26257?sslmode=disable'"
		Error:  exit status 134
		: exit status 1

cockroach-teamcity · 2019-05-14T22:51:48Z

SHA: https://github.com/cockroachdb/cockroach/commits/ba5c092a726134b73e789c2047f7ec151be7c1a1

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=6/cpu=16/multi-az PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1288263&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1833,tpcc.go:853,tpcc.go:554,test.go:1251: unexpected node event: 4: dead
	cluster.go:1038,context.go:89,cluster.go:1027,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1833,tpcc.go:853,tpcc.go:554,test.go:1251: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1288263-tpccbench-nodes-6-cpu-16-multi-az --oneshot --ignore-empty-nodes: exit status 1 3: skipped
		2: dead
		4: dead
		6: dead
		5: 3730
		7: 3653
		1: 3775
		Error:  2: dead, 4: dead, 6: dead

cockroach-teamcity · 2019-05-17T19:37:17Z

SHA: https://github.com/cockroachdb/cockroach/commits/7b2651400b2003d0a381cba9dbfc0b7bc0dfee00

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=6/cpu=16/multi-az PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1293898&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1833,tpcc.go:853,tpcc.go:554,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1293898-tpccbench-nodes-6-cpu-16-multi-az:3 -- ./workload fixtures load tpcc --warehouses=5000 --split --scatter --checks=false {pgurl:1} returned:
		stderr:
		
		stdout:
		open -> drain
		debug2: channel 0: obuf empty
		debug2: channel 0: close_write
		debug2: channel 0: output drain -> closed
		debug3: receive packet: type 98
		debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
		debug3: receive packet: type 97
		debug2: channel 0: rcvd close
		debug3: channel 0: will not send data after close
		debug2: channel 0: almost dead
		debug2: channel 0: gc: notify user
		debug2: channel 0: gc: user detached
		debug2: channel 0: send close
		debug3: send packet: type 97
		debug2: channel 0: is dead
		debug2: channel 0: garbage collecting
		debug1: channel 0: free: client-session, nchannels 1
		debug3: channel 0: status: The following connections are open:
		  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
		
		debug3: send packet: type 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug3: fd 2 is not O_NONBLOCK
		Transferred: sent 12952, received 7924 bytes, in 16582.0 seconds
		Bytes per second: sent 0.8, received 0.5
		debug1: Exit status 1
		: exit status 1
		: exit status 1

cockroach-teamcity · 2019-05-18T19:20:46Z

SHA: https://github.com/cockroachdb/cockroach/commits/923a3b2a6f4a6492883141092280d1041de1381a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=6/cpu=16/multi-az PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1295056&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1833,tpcc.go:853,tpcc.go:554,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1295056-tpccbench-nodes-6-cpu-16-multi-az:3 -- ./workload fixtures load tpcc --warehouses=5000 --split --scatter --checks=false {pgurl:1} returned:
		stderr:
		
		stdout:
		open -> drain
		debug2: channel 0: obuf empty
		debug2: channel 0: close_write
		debug2: channel 0: output drain -> closed
		debug3: receive packet: type 98
		debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
		debug3: receive packet: type 97
		debug2: channel 0: rcvd close
		debug3: channel 0: will not send data after close
		debug2: channel 0: almost dead
		debug2: channel 0: gc: notify user
		debug2: channel 0: gc: user detached
		debug2: channel 0: send close
		debug3: send packet: type 97
		debug2: channel 0: is dead
		debug2: channel 0: garbage collecting
		debug1: channel 0: free: client-session, nchannels 1
		debug3: channel 0: status: The following connections are open:
		  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
		
		debug3: send packet: type 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug3: fd 2 is not O_NONBLOCK
		Transferred: sent 13168, received 7988 bytes, in 16887.8 seconds
		Bytes per second: sent 0.8, received 0.5
		debug1: Exit status 1
		: exit status 1
		: exit status 1

cockroach-teamcity · 2019-05-19T19:04:20Z

SHA: https://github.com/cockroachdb/cockroach/commits/cab299a0ef983f8b4ffe5d724e44587d9665d3a3

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=6/cpu=16/multi-az PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1295811&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1833,tpcc.go:853,tpcc.go:554,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1295811-tpccbench-nodes-6-cpu-16-multi-az:3 -- ./workload fixtures load tpcc --warehouses=5000 --split --scatter --checks=false {pgurl:1} returned:
		stderr:
		
		stdout:
		status reply 0
		debug3: receive packet: type 96
		debug2: channel 0: rcvd eof
		debug2: channel 0: output open -> drain
		debug2: channel 0: obuf empty
		debug2: channel 0: close_write
		debug2: channel 0: output drain -> closed
		debug3: receive packet: type 97
		debug2: channel 0: rcvd close
		debug3: channel 0: will not send data after close
		debug2: channel 0: almost dead
		debug2: channel 0: gc: notify user
		debug2: channel 0: gc: user detached
		debug2: channel 0: send close
		debug3: send packet: type 97
		debug2: channel 0: is dead
		debug2: channel 0: garbage collecting
		debug1: channel 0: free: client-session, nchannels 1
		debug3: channel 0: status: The following connections are open:
		  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
		
		debug3: send packet: type 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug3: fd 2 is not O_NONBLOCK
		Transferred: sent 12088, received 7628 bytes, in 15106.7 seconds
		Bytes per second: sent 0.8, received 0.5
		debug1: Exit status 1
		: exit status 1
		: exit status 1

nvanbenschoten · 2019-05-20T05:43:49Z

The failures over the past 3 days are because of #37590.

cockroach-teamcity · 2019-05-20T18:59:48Z

SHA: https://github.com/cockroachdb/cockroach/commits/58c567a325056033b326cb9c4ed9ba490e8956da

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=6/cpu=16/multi-az PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1296592&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1833,tpcc.go:853,tpcc.go:554,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1296592-tpccbench-nodes-6-cpu-16-multi-az:3 -- ./workload fixtures load tpcc --warehouses=5000 --split --scatter --checks=false {pgurl:1} returned:
		stderr:
		
		stdout:
		status reply 0
		debug3: receive packet: type 96
		debug2: channel 0: rcvd eof
		debug2: channel 0: output open -> drain
		debug2: channel 0: obuf empty
		debug2: channel 0: close_write
		debug2: channel 0: output drain -> closed
		debug3: receive packet: type 97
		debug2: channel 0: rcvd close
		debug3: channel 0: will not send data after close
		debug2: channel 0: almost dead
		debug2: channel 0: gc: notify user
		debug2: channel 0: gc: user detached
		debug2: channel 0: send close
		debug3: send packet: type 97
		debug2: channel 0: is dead
		debug2: channel 0: garbage collecting
		debug1: channel 0: free: client-session, nchannels 1
		debug3: channel 0: status: The following connections are open:
		  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
		
		debug3: send packet: type 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug3: fd 2 is not O_NONBLOCK
		Transferred: sent 13060, received 7960 bytes, in 16848.3 seconds
		Bytes per second: sent 0.8, received 0.5
		debug1: Exit status 1
		: exit status 1
		: exit status 1

nvanbenschoten · 2019-05-21T20:13:41Z

Previous two issues addressed by #37701.

cockroach-teamcity · 2019-05-23T01:41:39Z

SHA: https://github.com/cockroachdb/cockroach/commits/7009f8750d5c3af32d5c43011869048ea7a311ae

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=6/cpu=16/multi-az PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1300930&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1869,tpcc.go:853,tpcc.go:554,test.go:1251: signal: interrupt
	cluster.go:1586,cluster.go:1605,cluster.go:1709,cluster.go:1094,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1869,tpcc.go:853,tpcc.go:554,test.go:1251: context canceled

cockroach-teamcity · 2019-05-24T00:13:16Z

SHA: https://github.com/cockroachdb/cockroach/commits/c9301cf71ea69da451fe5e5ba2c3074a4fe53831

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=6/cpu=16/multi-az PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1303699&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1875,tpcc.go:853,tpcc.go:554,test.go:1251: unexpected node event: 6: dead
	cluster.go:1038,context.go:89,cluster.go:1027,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1875,tpcc.go:853,tpcc.go:554,test.go:1251: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1303699-tpccbench-nodes-6-cpu-16-multi-az --oneshot --ignore-empty-nodes: exit status 1 3: skipped
		6: dead
		5: 1110
		7: 1434
		1: 1613
		4: 1670
		2: 1498
		Error:  6: dead

cockroach-teamcity · 2019-05-24T23:57:17Z

SHA: https://github.com/cockroachdb/cockroach/commits/630a6e9cb3771912cd138f9aa3bea1f0ca9fa7c9

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=6/cpu=16/multi-az PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1306250&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1875,tpcc.go:828,tpcc.go:554,test.go:1251: unexpected node event: 1: dead
	cluster.go:1038,context.go:89,cluster.go:1027,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1875,tpcc.go:828,tpcc.go:554,test.go:1251: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1306250-tpccbench-nodes-6-cpu-16-multi-az --oneshot --ignore-empty-nodes: exit status 1 7: skipped
		1: dead
		3: 3174
		2: 2959
		4: 3223
		6: 2613
		5: 3435
		Error:  1: dead

tbg · 2019-05-27T14:32:48Z

F190524 23:53:29.817116 24309 kv/txn_coord_sender.go:913  [n1,client=10.128.0.82:40908,user=root] unexpected txn state: "sql txn" id=ca81044b key=/Table/54/1/804/0 rw=true pri=0.02619792 stat=COMMITTED epo=1 ts=1558742002.436278411,0 orig=1558742002.436278411,0 max=1558742002.936278411,0 wto=false seq=7 int=6
goroutine 24309 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0xc000057b01, 0xc000057b60, 0x557b400, 0x16)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1020 +0xd4
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x5d0a200, 0xc000000004, 0x557b485, 0x16, 0x391, 0xc0089f8400, 0xfc)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:878 +0x93d
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x3b6b180, 0xc0293c5b30, 0x4, 0x2, 0x3404e91, 0x18, 0xc0476450c0, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:85 +0x2d8
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x3b6b180, 0xc0293c5b30, 0x1, 0x4, 0x3404e91, 0x18, 0xc0476450c0, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:71 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(0x3b6b180, 0xc0293c5b30, 0x3404e91, 0x18, 0xc0476450c0, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:182 +0x7e
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).maybeRejectClientLocked(0xc02f92a900, 0x3b6b180, 0xc0293c5b30, 0xc03563ad00, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:913 +0x180
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc02f92a900, 0x3b6b180, 0xc0293c5b30, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:748 +0x107

#37866 for both of the above.

cockroach-teamcity added this to the 19.2 milestone May 1, 2019

cockroach-teamcity assigned nvanbenschoten May 1, 2019

cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels May 1, 2019

nvanbenschoten mentioned this issue May 20, 2019

workload/tpcc: duplicate FK detection during initialization broken #37590

Closed

This comment has been minimized.

Sign in to view

tbg changed the title ~~roachtest: tpccbench/nodes=6/cpu=16/multi-az failed~~ roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [#37866] May 27, 2019

tbg closed this as completed May 27, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [#37866] #37250

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [#37866] #37250

cockroach-teamcity commented May 1, 2019

andreimatei commented May 2, 2019

cockroach-teamcity commented May 8, 2019

danhhz commented May 8, 2019

cockroach-teamcity commented May 8, 2019

cockroach-teamcity commented May 14, 2019

cockroach-teamcity commented May 17, 2019

cockroach-teamcity commented May 18, 2019

cockroach-teamcity commented May 19, 2019

nvanbenschoten commented May 20, 2019

cockroach-teamcity commented May 20, 2019

This comment has been minimized.

This comment has been minimized.

nvanbenschoten commented May 21, 2019

cockroach-teamcity commented May 23, 2019

cockroach-teamcity commented May 24, 2019

cockroach-teamcity commented May 24, 2019

tbg commented May 27, 2019

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [#37866] #37250

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [#37866] #37250

Comments

cockroach-teamcity commented May 1, 2019

andreimatei commented May 2, 2019

cockroach-teamcity commented May 8, 2019

danhhz commented May 8, 2019

cockroach-teamcity commented May 8, 2019

cockroach-teamcity commented May 14, 2019

cockroach-teamcity commented May 17, 2019

cockroach-teamcity commented May 18, 2019

cockroach-teamcity commented May 19, 2019

nvanbenschoten commented May 20, 2019

cockroach-teamcity commented May 20, 2019

This comment has been minimized.

This comment has been minimized.

nvanbenschoten commented May 21, 2019

cockroach-teamcity commented May 23, 2019

cockroach-teamcity commented May 24, 2019

cockroach-teamcity commented May 24, 2019

tbg commented May 27, 2019