roachtest: cancel/tpcc/local/w=10,nodes=3 failed #35029

cockroach-teamcity · 2019-02-17T07:07:36Z

SHA: https://github.com/cockroachdb/cockroach/commits/8e9a1e310e3e8e37f091b7ca8bd204084ad9e2e5

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cancel/tpcc/local/w=10,nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1142461&tab=buildLog

The test failed on master:
	cluster.go:1226,cancel.go:87,cluster.go:1564,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1142461-cancel-tpcc-local-w-10-nodes-3:1 -- ./cockroach sql --insecure -e "CANCEL QUERIES
			SELECT query_id FROM [SHOW CLUSTER QUERIES] WHERE query not like '%SHOW CLUSTER QUERIES%'" returned:
		stderr:
		
		stdout:
		Error: pq: could not cancel query 158414ec4ac88ab60000000000000001: query ID 158414ec4ac88ab60000000000000001 not found
		Failed running "sql"
		Error:  exit status 1
		: exit status 1
	cluster.go:1585,cancel.go:105,cancel.go:131,test.go:1212: Goexit() was called

The text was updated successfully, but these errors were encountered:

tbg · 2019-02-19T10:15:25Z

@knz not sure who owns this, could you assign?

cockroach-teamcity · 2019-03-06T07:18:51Z

SHA: https://github.com/cockroachdb/cockroach/commits/959dcf7de0f94cfcfa0062387b109adebd1f11da

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cancel/tpcc/local/w=10,nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1163702&tab=buildLog

The test failed on master:
	cluster.go:1038,cancel.go:43,cancel.go:131,test.go:1214: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod put teamcity-1163702-cancel-tpcc-local-w-10-nodes-3:1-3 /home/agent/work/.go/src/github.com/cockroachdb/cockroach/cockroach.linux-2.6.32-gnu-amd64 ./cockroach returned:
		stderr:
		
		stdout:
		Error:  unknown cluster: teamcity-1163702-cancel-tpcc-local-w-10-nodes-3
		
		Available clusters:
		  bram-1551819947-sqlalchemy1
		  bram-1551821395-sqlalchemy5
		  bram-1551827723-sqlalchemy1
		  david-qa
		  david-stable
		  teamcity-1163702-acceptance
		  teamcity-1163702-backup2tb
		  teamcity-1163702-backuptpcc
		  teamcity-1163702-cancel-tpcc-distsql-w-10-nodes-3
		  teamcity-1163702-cdc-bank
		  teamcity-1163702-cdc-cloud-sink-rangefeed-true
		  teamcity-1163702-cdc-crdb-chaos-rangefeed-true
		
		Hint: use "roachprod sync" to update the list of available clusters.
		
		: exit status 1

cockroach-teamcity · 2019-03-24T06:17:18Z

SHA: https://github.com/cockroachdb/cockroach/commits/5a746073c3f8ede851f37dd895cf1a91d6dcc3cf

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cancel/tpcc/local/w=10,nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1195714&tab=buildLog

The test failed on master:
	cancel.go:99,cluster.go:1605,errgroup.go:57: query took too long to respond to cancellation
	cluster.go:1626,cancel.go:105,cancel.go:131,test.go:1214: Goexit() was called

cockroach-teamcity · 2019-05-17T06:17:06Z

SHA: https://github.com/cockroachdb/cockroach/commits/7b2651400b2003d0a381cba9dbfc0b7bc0dfee00

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cancel/tpcc/local/w=10,nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1293898&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,cancel.go:50,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1293898-cancel-tpcc-local-w-10-nodes-3:1 -- ./workload fixtures load tpcc --warehouses=10 {pgurl:1} returned:
		stderr:
		
		stdout:
		t-status reply 0
		debug3: receive packet: type 96
		debug2: channel 0: rcvd eof
		debug2: channel 0: output open -> drain
		debug2: channel 0: obuf empty
		debug2: channel 0: close_write
		debug2: channel 0: output drain -> closed
		debug3: receive packet: type 97
		debug2: channel 0: rcvd close
		debug3: channel 0: will not send data after close
		debug2: channel 0: almost dead
		debug2: channel 0: gc: notify user
		debug2: channel 0: gc: user detached
		debug2: channel 0: send close
		debug3: send packet: type 97
		debug2: channel 0: is dead
		debug2: channel 0: garbage collecting
		debug1: channel 0: free: client-session, nchannels 1
		debug3: channel 0: status: The following connections are open:
		  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
		
		debug3: send packet: type 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug3: fd 2 is not O_NONBLOCK
		Transferred: sent 3164, received 4600 bytes, in 54.2 seconds
		Bytes per second: sent 58.4, received 84.8
		debug1: Exit status 1
		: exit status 1
		: exit status 1
	cluster.go:1833,cancel.go:105,cancel.go:131,test.go:1251: Goexit() was called

cockroach-teamcity · 2019-05-18T06:19:20Z

SHA: https://github.com/cockroachdb/cockroach/commits/923a3b2a6f4a6492883141092280d1041de1381a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cancel/tpcc/local/w=10,nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1295056&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,cancel.go:50,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1295056-cancel-tpcc-local-w-10-nodes-3:1 -- ./workload fixtures load tpcc --warehouses=10 {pgurl:1} returned:
		stderr:
		
		stdout:
		: type 97
		debug2: channel 0: rcvd close
		debug3: channel 0: will not send data after close
		debug2: channel 0: obuf_empty delayed efd 7/(113)
		debug2: channel 0: written 113 to efd 7
		debug3: channel 0: will not send data after close
		debug2: channel 0: obuf empty
		debug2: channel 0: close_write
		debug2: channel 0: output drain -> closed
		debug2: channel 0: almost dead
		debug2: channel 0: gc: notify user
		debug2: channel 0: gc: user detached
		debug2: channel 0: send close
		debug3: send packet: type 97
		debug2: channel 0: is dead
		debug2: channel 0: garbage collecting
		debug1: channel 0: free: client-session, nchannels 1
		debug3: channel 0: status: The following connections are open:
		  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
		
		debug3: send packet: type 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug3: fd 2 is not O_NONBLOCK
		Transferred: sent 3164, received 4600 bytes, in 53.9 seconds
		Bytes per second: sent 58.7, received 85.4
		debug1: Exit status 1
		: exit status 1
		: exit status 1
	cluster.go:1833,cancel.go:105,cancel.go:131,test.go:1251: Goexit() was called

cockroach-teamcity · 2019-05-19T06:16:50Z

SHA: https://github.com/cockroachdb/cockroach/commits/cab299a0ef983f8b4ffe5d724e44587d9665d3a3

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cancel/tpcc/local/w=10,nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1295811&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,cancel.go:50,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1295811-cancel-tpcc-local-w-10-nodes-3:1 -- ./workload fixtures load tpcc --warehouses=10 {pgurl:1} returned:
		stderr:
		
		stdout:
		t-status reply 0
		debug3: receive packet: type 96
		debug2: channel 0: rcvd eof
		debug2: channel 0: output open -> drain
		debug2: channel 0: obuf empty
		debug2: channel 0: close_write
		debug2: channel 0: output drain -> closed
		debug3: receive packet: type 97
		debug2: channel 0: rcvd close
		debug3: channel 0: will not send data after close
		debug2: channel 0: almost dead
		debug2: channel 0: gc: notify user
		debug2: channel 0: gc: user detached
		debug2: channel 0: send close
		debug3: send packet: type 97
		debug2: channel 0: is dead
		debug2: channel 0: garbage collecting
		debug1: channel 0: free: client-session, nchannels 1
		debug3: channel 0: status: The following connections are open:
		  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
		
		debug3: send packet: type 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug3: fd 2 is not O_NONBLOCK
		Transferred: sent 3164, received 4596 bytes, in 57.2 seconds
		Bytes per second: sent 55.3, received 80.4
		debug1: Exit status 1
		: exit status 1
		: exit status 1
	cluster.go:1833,cancel.go:105,cancel.go:131,test.go:1251: Goexit() was called

cockroach-teamcity · 2019-05-20T06:20:01Z

SHA: https://github.com/cockroachdb/cockroach/commits/58c567a325056033b326cb9c4ed9ba490e8956da

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cancel/tpcc/local/w=10,nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1296592&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,cancel.go:50,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1296592-cancel-tpcc-local-w-10-nodes-3:1 -- ./workload fixtures load tpcc --warehouses=10 {pgurl:1} returned:
		stderr:
		
		stdout:
		-status reply 0
		debug3: receive packet: type 96
		debug2: channel 0: rcvd eof
		debug2: channel 0: output open -> drain
		debug2: channel 0: obuf empty
		debug2: channel 0: close_write
		debug2: channel 0: output drain -> closed
		debug3: receive packet: type 97
		debug2: channel 0: rcvd close
		debug3: channel 0: will not send data after close
		debug2: channel 0: almost dead
		debug2: channel 0: gc: notify user
		debug2: channel 0: gc: user detached
		debug2: channel 0: send close
		debug3: send packet: type 97
		debug2: channel 0: is dead
		debug2: channel 0: garbage collecting
		debug1: channel 0: free: client-session, nchannels 1
		debug3: channel 0: status: The following connections are open:
		  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
		
		debug3: send packet: type 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug3: fd 2 is not O_NONBLOCK
		Transferred: sent 3164, received 4592 bytes, in 37.8 seconds
		Bytes per second: sent 83.6, received 121.4
		debug1: Exit status 1
		: exit status 1
		: exit status 1
	cluster.go:1833,cancel.go:105,cancel.go:131,test.go:1251: Goexit() was called

nvanbenschoten · 2019-05-21T20:09:27Z

Previous two issues addressed by #37701.

cockroach-teamcity · 2020-02-13T08:23:06Z

(roachtest).cancel/tpcc/local/w=10,nodes=3 failed on master@2739821b911d777fa2a927295d699b559360a802:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1747374-1581581035-20-n3cpu4:1 -- ./workload fixtures load tpcc --warehouses=10 {pgurl:1} returned:
		    stderr:
		    I200213 08:22:24.649209 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		    Error: restoring fixture: sql: expected 6 destination arguments in Scan, not 7
		    Error:  exit status 1
		    
		    stdout::
		  - exit status 1

	cluster.go:2368,cancel.go:100,cancel.go:128,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.registerCancel.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cancel.go:100
		    main.registerCancel.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cancel.go:128
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.init
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2309
		    runtime.doInit
		    	/usr/local/go/src/runtime/proc.go:5222
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:190
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - Goexit() was called

More

Artifacts: /cancel/tpcc/local/w=10,nodes=3

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

irfansharif · 2020-03-04T19:26:18Z

restoring fixture: sql: expected 6 destination arguments in Scan, not 7

This was fixed in #45078.

This hasn't seen any unresolved failures since May last year. Closing due to inactivity.

cockroach-teamcity added this to the 2.2 milestone Feb 17, 2019

cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Feb 17, 2019

tbg assigned knz Feb 19, 2019

knz assigned jordanlewis and unassigned knz May 20, 2019

This comment has been minimized.

Sign in to view

tbg added the branch-master Failures and bugs on the master branch. label Jan 22, 2020

irfansharif closed this as completed Mar 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: cancel/tpcc/local/w=10,nodes=3 failed #35029

roachtest: cancel/tpcc/local/w=10,nodes=3 failed #35029

cockroach-teamcity commented Feb 17, 2019

tbg commented Feb 19, 2019

cockroach-teamcity commented Mar 6, 2019

cockroach-teamcity commented Mar 24, 2019

cockroach-teamcity commented May 17, 2019

cockroach-teamcity commented May 18, 2019

cockroach-teamcity commented May 19, 2019

cockroach-teamcity commented May 20, 2019

This comment has been minimized.

This comment has been minimized.

nvanbenschoten commented May 21, 2019

cockroach-teamcity commented Feb 13, 2020

irfansharif commented Mar 4, 2020

roachtest: cancel/tpcc/local/w=10,nodes=3 failed #35029

roachtest: cancel/tpcc/local/w=10,nodes=3 failed #35029

Comments

cockroach-teamcity commented Feb 17, 2019

tbg commented Feb 19, 2019

cockroach-teamcity commented Mar 6, 2019

cockroach-teamcity commented Mar 24, 2019

cockroach-teamcity commented May 17, 2019

cockroach-teamcity commented May 18, 2019

cockroach-teamcity commented May 19, 2019

cockroach-teamcity commented May 20, 2019

This comment has been minimized.

This comment has been minimized.

nvanbenschoten commented May 21, 2019

cockroach-teamcity commented Feb 13, 2020

irfansharif commented Mar 4, 2020