Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: tpcc-nowait/nodes=3/w=1 failed #37308

Closed
cockroach-teamcity opened this issue May 4, 2019 · 24 comments
Closed

roachtest: tpcc-nowait/nodes=3/w=1 failed #37308

cockroach-teamcity opened this issue May 4, 2019 · 24 comments
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. S-2 Medium-high impact: many users impacted, risks of availability and difficult-to-fix data errors
Milestone
19.2

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/856ba9108f112f85d406bbe88d2208651859336e

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc-nowait/nodes=3/w=1 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1274175&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,tpcc.go:168,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1274175-tpcc-nowait-nodes-3-w-1:4 -- ./workload run tpcc --warehouses=1 --histograms=logs/stats.json --wait=false --ramp=5m0s --duration=10m0s {pgurl:1-3} returned:
		stderr:
		
		stdout:
		     79.7 delivery
		   1m53s        0           34.0           27.0     37.7     71.3    113.2    113.2 newOrder
		   1m53s        0            2.0            2.7      6.0      6.8      6.8      6.8 orderStatus
		   1m53s        0           42.0           27.1     15.7    104.9    134.2    134.2 payment
		   1m53s        0            3.0            2.7     10.5     15.7     15.7     15.7 stockLevel
		   1m54s        0            4.0            2.7     52.4     54.5     54.5     54.5 delivery
		   1m54s        0           46.0           27.2     31.5     46.1     60.8     60.8 newOrder
		   1m54s        0            5.0            2.7      5.8      7.1      7.1      7.1 orderStatus
		   1m54s        0           36.0           27.1     18.9     88.1     96.5     96.5 payment
		   1m54s        0            4.0            2.7     11.5     25.2     25.2     25.2 stockLevel
		Error: error in newOrder: ERROR: request timestamp 1556982431.577537311,9 less than txn timestamp 1556982431.577703563,1 (SQLSTATE XXUUU)
		Error:  exit status 1
		: exit status 1
	cluster.go:1833,tpcc.go:178,tpcc.go:287,test.go:1251: Goexit() was called
@cockroach-teamcity cockroach-teamcity added this to the 19.2 milestone May 4, 2019
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels May 4, 2019
@andreimatei
Copy link
Contributor

@nvanbenschoten this error looks scary?

@andreimatei andreimatei added the S-1 High impact: many users impacted, serious risk of high unavailability or data loss label May 9, 2019
@nvanbenschoten
Copy link
Member

Yeah, this is coming from #36307, which made this error a lot louder. It's scary because it's pointing out a case where timestamps aren't propagated correctly. We set the batch header's timestamp here, which means that the node sending the QueryTxn request had a clock that was below the timestamp of a transaction that it was pushing.

I don't know what to suggest other than that we need to divorce causality transfer for HLC clocks from the batch header timestamp. Without that, these issues will keep popping up even as we fix individual occurrences one-by-one.

@nvanbenschoten nvanbenschoten added S-2 Medium-high impact: many users impacted, risks of availability and difficult-to-fix data errors and removed S-1 High impact: many users impacted, serious risk of high unavailability or data loss labels May 10, 2019
@nvanbenschoten nvanbenschoten removed their assignment May 10, 2019
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/c25518b4e9a723d8de0dba30a95ce0ade7963aed

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc-nowait/nodes=3/w=1 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1286188&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,tpcc.go:168,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1286188-tpcc-nowait-nodes-3-w-1:4 -- ./workload run tpcc --warehouses=1 --histograms=logs/stats.json --wait=false --ramp=5m0s --duration=10m0s {pgurl:1-3} returned:
		stderr:
		
		stdout:
		     52.4 delivery
		   2m39s        0           36.0           33.7     52.4    100.7    125.8    125.8 newOrder
		   2m39s        0            4.0            3.4      6.0      7.3      7.3      7.3 orderStatus
		   2m39s        0           30.0           33.7     46.1    117.4    151.0    151.0 payment
		   2m39s        0            4.0            3.4     10.0     25.2     25.2     25.2 stockLevel
		   2m40s        0            2.0            3.4     48.2     50.3     50.3     50.3 delivery
		   2m40s        0           26.0           33.7     37.7    104.9    251.7    251.7 newOrder
		   2m40s        0            3.0            3.4      5.2      6.8      6.8      6.8 orderStatus
		   2m40s        0           27.0           33.7     39.8    486.5    520.1    520.1 payment
		   2m40s        0            2.0            3.4      6.6      8.1      8.1      8.1 stockLevel
		Error: error in payment: ERROR: request timestamp 1557671993.466594938,11 less than txn timestamp 1557671993.467048774,1 (SQLSTATE XXUUU)
		Error:  exit status 1
		: exit status 1
	cluster.go:1833,tpcc.go:178,tpcc.go:287,test.go:1251: Goexit() was called

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/7b2651400b2003d0a381cba9dbfc0b7bc0dfee00

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc-nowait/nodes=3/w=1 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1293898&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,tpcc.go:157,tpcc.go:159,tpcc.go:287,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1293898-tpcc-nowait-nodes-3-w-1:4 -- ./workload fixtures load tpcc --warehouses=1  {pgurl:1} returned:
		stderr:
		
		stdout:
		status reply 0
		debug3: receive packet: type 96
		debug2: channel 0: rcvd eof
		debug2: channel 0: output open -> drain
		debug2: channel 0: obuf empty
		debug2: channel 0: close_write
		debug2: channel 0: output drain -> closed
		debug3: receive packet: type 97
		debug2: channel 0: rcvd close
		debug3: channel 0: will not send data after close
		debug2: channel 0: almost dead
		debug2: channel 0: gc: notify user
		debug2: channel 0: gc: user detached
		debug2: channel 0: send close
		debug3: send packet: type 97
		debug2: channel 0: is dead
		debug2: channel 0: garbage collecting
		debug1: channel 0: free: client-session, nchannels 1
		debug3: channel 0: status: The following connections are open:
		  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
		
		debug3: send packet: type 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug3: fd 2 is not O_NONBLOCK
		Transferred: sent 3164, received 4584 bytes, in 10.5 seconds
		Bytes per second: sent 301.8, received 437.2
		debug1: Exit status 1
		: exit status 1
		: exit status 1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/923a3b2a6f4a6492883141092280d1041de1381a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc-nowait/nodes=3/w=1 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1295056&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,tpcc.go:157,tpcc.go:159,tpcc.go:287,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1295056-tpcc-nowait-nodes-3-w-1:4 -- ./workload fixtures load tpcc --warehouses=1  {pgurl:1} returned:
		stderr:
		
		stdout:
		-status reply 0
		debug3: receive packet: type 96
		debug2: channel 0: rcvd eof
		debug2: channel 0: output open -> drain
		debug2: channel 0: obuf empty
		debug2: channel 0: close_write
		debug2: channel 0: output drain -> closed
		debug3: receive packet: type 97
		debug2: channel 0: rcvd close
		debug3: channel 0: will not send data after close
		debug2: channel 0: almost dead
		debug2: channel 0: gc: notify user
		debug2: channel 0: gc: user detached
		debug2: channel 0: send close
		debug3: send packet: type 97
		debug2: channel 0: is dead
		debug2: channel 0: garbage collecting
		debug1: channel 0: free: client-session, nchannels 1
		debug3: channel 0: status: The following connections are open:
		  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
		
		debug3: send packet: type 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug3: fd 2 is not O_NONBLOCK
		Transferred: sent 3164, received 4528 bytes, in 9.2 seconds
		Bytes per second: sent 343.9, received 492.2
		debug1: Exit status 1
		: exit status 1
		: exit status 1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/cab299a0ef983f8b4ffe5d724e44587d9665d3a3

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc-nowait/nodes=3/w=1 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1295811&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,tpcc.go:157,tpcc.go:159,tpcc.go:287,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1295811-tpcc-nowait-nodes-3-w-1:4 -- ./workload fixtures load tpcc --warehouses=1  {pgurl:1} returned:
		stderr:
		
		stdout:
		-status reply 0
		debug3: receive packet: type 96
		debug2: channel 0: rcvd eof
		debug2: channel 0: output open -> drain
		debug2: channel 0: obuf empty
		debug2: channel 0: close_write
		debug2: channel 0: output drain -> closed
		debug3: receive packet: type 97
		debug2: channel 0: rcvd close
		debug3: channel 0: will not send data after close
		debug2: channel 0: almost dead
		debug2: channel 0: gc: notify user
		debug2: channel 0: gc: user detached
		debug2: channel 0: send close
		debug3: send packet: type 97
		debug2: channel 0: is dead
		debug2: channel 0: garbage collecting
		debug1: channel 0: free: client-session, nchannels 1
		debug3: channel 0: status: The following connections are open:
		  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
		
		debug3: send packet: type 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug3: fd 2 is not O_NONBLOCK
		Transferred: sent 3164, received 4536 bytes, in 9.3 seconds
		Bytes per second: sent 338.8, received 485.7
		debug1: Exit status 1
		: exit status 1
		: exit status 1

@nvanbenschoten nvanbenschoten mentioned this issue May 20, 2019
Closed
@nvanbenschoten
Copy link
Member

The failures over the past 3 days are because of #37590.

@cockroach-teamcity

This comment has been minimized.

Sign in to view
@cockroach-teamcity

This comment has been minimized.

Sign in to view
@cockroach-teamcity

This comment has been minimized.

Sign in to view
@nvanbenschoten
Copy link
Member

Previous two issues addressed by #37701.

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/7009f8750d5c3af32d5c43011869048ea7a311ae

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc-nowait/nodes=3/w=1 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1300930&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1510,tpcc.go:168,cluster.go:1848,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1300930-tpcc-nowait-nodes-3-w-1:4 -- ./workload run tpcc --warehouses=1 --histograms=logs/stats.json --wait=false --ramp=5m0s --duration=10m0s {pgurl:1-3} returned:
		stderr:
		
		stdout:
		.0    302.0 newOrder
		   2m59s        0            4.0            3.2      5.2      5.8      5.8      5.8 orderStatus
		   2m59s        0           28.0           31.7     50.3    201.3    243.3    243.3 payment
		   2m59s        0            5.0            3.2     10.0     30.4     30.4     30.4 stockLevel
		    3m0s        0            4.0            3.2     50.3     56.6     56.6     56.6 delivery
		    3m0s        0           32.0           31.7     46.1    151.0    226.5    226.5 newOrder
		    3m0s        0            1.0            3.2      5.0      5.0      5.0      5.0 orderStatus
		    3m0s        0           28.0           31.6     50.3    302.0    302.0    302.0 payment
		    3m0s        0            2.0            3.2     14.7     16.8     16.8     16.8 stockLevel
		Error: error in newOrder: ERROR: request timestamp 1558541113.827713606,28 less than txn timestamp 1558541113.829743477,1 (SQLSTATE XXUUU)
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_34.74.28.113_2019-05-22T16:02:12Z: exit status 1
		: exit status 1
	cluster.go:1869,tpcc.go:178,tpcc.go:287,test.go:1251: Goexit() was called

@nvanbenschoten
Copy link
Member

Same story as #37308 (comment).

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/c9301cf71ea69da451fe5e5ba2c3074a4fe53831

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc-nowait/nodes=3/w=1 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1303699&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1516,tpcc.go:168,cluster.go:1854,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1303699-tpcc-nowait-nodes-3-w-1:4 -- ./workload run tpcc --warehouses=1 --histograms=logs/stats.json --wait=false --ramp=5m0s --duration=10m0s {pgurl:1-3} returned:
		stderr:
		
		stdout:
		09.7    209.7 newOrder
		   1m19s        0            5.0            2.8      4.5      5.2      5.2      5.2 orderStatus
		   1m19s        0           33.0           27.5     16.8    104.9    268.4    268.4 payment
		   1m19s        0            3.0            2.8     25.2     39.8     39.8     39.8 stockLevel
		   1m20s        0            3.0            2.8     52.4     54.5     54.5     54.5 delivery
		   1m20s        0           36.0           27.6     35.7     83.9    151.0    151.0 newOrder
		   1m20s        0            1.0            2.7      4.7      4.7      4.7      4.7 orderStatus
		   1m20s        0           34.0           27.6     18.9     65.0    234.9    234.9 payment
		   1m20s        0            4.0            2.8     10.0     28.3     28.3     28.3 stockLevel
		Error: error in payment: ERROR: request timestamp 1558626548.541566650,9 less than txn timestamp 1558626548.541987729,1 (SQLSTATE XXUUU)
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_34.74.116.12_2019-05-23T15:47:46Z: exit status 1
		: exit status 1
	cluster.go:1875,tpcc.go:178,tpcc.go:287,test.go:1251: Goexit() was called

@tbg
Copy link
Member

tbg commented May 27, 2019

^- ditto #37308 (comment)

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/4e1ebb25931b5d33ec5bc9686cd83fe9c135971e

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc-nowait/nodes=3/w=1 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1311055&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1516,tpcc.go:168,cluster.go:1854,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1311055-tpcc-nowait-nodes-3-w-1:4 -- ./workload run tpcc --warehouses=1 --histograms=logs/stats.json --wait=false --ramp=5m0s --duration=10m0s {pgurl:1-3} returned:
		stderr:
		
		stdout:
		3     92.3 newOrder
		   2m29s        0            4.0            3.4      5.2      6.8      6.8      6.8 orderStatus
		   2m29s        0           39.0           34.3     32.5    109.1    159.4    159.4 payment
		   2m29s        0            3.0            3.4     27.3     32.5     32.5     32.5 stockLevel
		   2m30s        0            3.0            3.4     54.5    104.9    104.9    104.9 delivery
		   2m30s        0           30.0           34.3     37.7    176.2    218.1    218.1 newOrder
		   2m30s        0            5.0            3.4      5.8      6.6      6.6      6.6 orderStatus
		   2m30s        0           33.0           34.3     22.0    184.5    352.3    352.3 payment
		   2m30s        0            3.0            3.4      9.4      9.4      9.4      9.4 stockLevel
		Error: error in payment: ERROR: request timestamp 1558974259.351167812,17 less than txn timestamp 1558974259.351716258,11 (SQLSTATE XXUUU)
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_35.237.152.64_2019-05-27T16:21:48Z: exit status 1
		: exit status 1
	cluster.go:1875,tpcc.go:178,tpcc.go:287,test.go:1251: Goexit() was called

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/8e7ef35a8e4169ec63dc5a4df963d8b31a3d5b61

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc-nowait/nodes=3/w=1 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1324151&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1513,tpcc.go:165,cluster.go:1851,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1324151-tpcc-nowait-nodes-3-w-1:4 -- ./workload run tpcc --warehouses=1 --histograms=logs/stats.json --wait=false --ramp=5m0s --duration=10m0s {pgurl:1-3} returned:
		stderr:
		
		stdout:
		5.0      5.0      5.0 orderStatus
		      4s        0           37.0           38.0     12.6     67.1    159.4    159.4 payment
		      4s        0            4.0            3.8      8.4     28.3     28.3     28.3 stockLevel
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      5s        0            4.0            4.0     46.1     56.6     56.6     56.6 delivery
		      5s        0           36.0           37.4     32.5     65.0     79.7     79.7 newOrder
		      5s        0            4.0            3.8      5.0      5.5      5.5      5.5 orderStatus
		      5s        0           40.0           38.4     15.2    117.4    159.4    159.4 payment
		      5s        0            4.0            3.8      7.3      8.4      8.4      8.4 stockLevel
		Error: error in payment: ERROR: request timestamp 1559751852.057185285,9 less than txn timestamp 1559751852.057217625,1 (SQLSTATE XXUUU)
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_35.196.64.101_2019-06-05T16:19:05Z: exit status 1
		: exit status 1
	cluster.go:1872,tpcc.go:175,tpcc.go:284,test.go:1248: Goexit() was called

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/3f2470f358e638f526a07495bfd11f1805c135a8

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc-nowait/nodes=3/w=1 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1325775&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1513,tpcc.go:169,cluster.go:1851,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1325775-tpcc-nowait-nodes-3-w-1:4 -- ./workload run tpcc --warehouses=1 --histograms=logs/stats.json --wait=false --ramp=5m0s --duration=10m0s {pgurl:1-3} returned:
		stderr:
		
		stdout:
		   3623.9 newOrder
		   4m26s        0            4.0            3.5      5.5      6.6      6.6      6.6 orderStatus
		   4m26s        0           39.0           35.4     14.7    113.2    142.6    142.6 payment
		   4m26s        0            4.0            3.5      8.9     21.0     21.0     21.0 stockLevel
		   4m27s        0            4.0            3.5     46.1     58.7     58.7     58.7 delivery
		   4m27s        0           35.0           35.4     35.7    125.8    151.0    151.0 newOrder
		   4m27s        0            3.0            3.5      5.8      6.0      6.0      6.0 orderStatus
		   4m27s        0           38.0           35.4     44.0    151.0    218.1    218.1 payment
		   4m27s        0            4.0            3.5      9.4     46.1     46.1     46.1 stockLevel
		Error: error in payment: ERROR: request timestamp 1559834224.856721668,23 less than txn timestamp 1559834224.856721668,30 (SQLSTATE XXUUU)
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_35.231.166.145_2019-06-06T15:12:36Z: exit status 1
		: exit status 1
	cluster.go:1872,tpcc.go:179,tpcc.go:288,test.go:1248: Goexit() was called

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/a581c20147539c74a8c72f8e6b4a7a4206a50853

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc-nowait/nodes=3/w=1 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1334794&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1513,tpcc.go:169,cluster.go:1851,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1334794-tpcc-nowait-nodes-3-w-1:4 -- ./workload run tpcc --warehouses=1 --histograms=logs/stats.json --wait=false --ramp=5m0s --duration=10m0s {pgurl:1-3} returned:
		stderr:
		
		stdout:
		2.3    352.3 newOrder
		   1m26s        0            2.0            2.7      5.8      6.0      6.0      6.0 orderStatus
		   1m26s        0           27.0           27.2     17.8    176.2    192.9    192.9 payment
		   1m26s        0            3.0            2.7     10.0     16.3     16.3     16.3 stockLevel
		   1m27s        0            3.0            2.7     50.3     83.9     83.9     83.9 delivery
		   1m27s        0           30.0           27.3     35.7     75.5     92.3     92.3 newOrder
		   1m27s        0            4.0            2.7      5.8      6.0      6.0      6.0 orderStatus
		   1m27s        0           35.0           27.3     21.0    113.2    176.2    176.2 payment
		   1m27s        0            3.0            2.7     21.0     22.0     22.0     22.0 stockLevel
		Error: error in payment: ERROR: request timestamp 1560265671.707002633,9 less than txn timestamp 1560265671.707120496,1 (SQLSTATE XXUUU)
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_34.74.193.209_2019-06-11T15:06:23Z: exit status 1
		: exit status 1
	cluster.go:1872,tpcc.go:179,tpcc.go:288,test.go:1248: Goexit() was called

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/5c0b1644f9f9fe65bfb8cf3f7a5af2595bd859a8

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc-nowait/nodes=3/w=1 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1341218&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1513,tpcc.go:169,cluster.go:1851,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1341218-tpcc-nowait-nodes-3-w-1:4 -- ./workload run tpcc --warehouses=1 --histograms=logs/stats.json --wait=false --ramp=5m0s --duration=10m0s {pgurl:1-3} returned:
		stderr:
		
		stdout:
		.1    218.1 newOrder
		   4m21s        0            5.0            3.0      5.8      6.0      6.0      6.0 orderStatus
		   4m21s        0           33.0           29.6     35.7    130.0    209.7    209.7 payment
		   4m21s        0            4.0            3.0     14.7     22.0     22.0     22.0 stockLevel
		   4m22s        0            2.0            3.0     52.4     54.5     54.5     54.5 delivery
		   4m22s        0           31.0           29.6     39.8     75.5     92.3     92.3 newOrder
		   4m22s        0            2.0            3.0      5.8      6.3      6.3      6.3 orderStatus
		   4m22s        0           41.0           29.6     41.9    167.8    234.9    234.9 payment
		   4m22s        0            2.0            3.0     10.0     17.8     17.8     17.8 stockLevel
		Error: error in payment: ERROR: request timestamp 1560696995.580485400,15 less than txn timestamp 1560696995.581214064,11 (SQLSTATE XXUUU)
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_35.231.64.19_2019-06-16T14:52:11Z: exit status 1
		: exit status 1
	cluster.go:1872,tpcc.go:179,tpcc.go:288,test.go:1248: Goexit() was called

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/5f358ed804af05f8c4b404efc4d8a282d8e0916c

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc-nowait/nodes=3/w=1 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1360435&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1511,tpcc.go:167,cluster.go:1849,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1360435-tpcc-nowait-nodes-3-w-1:4 -- ./workload run tpcc --warehouses=1 --histograms=logs/stats.json --wait=false --ramp=5m0s --duration=10m0s {pgurl:1-3} returned:
		stderr:
		
		stdout:
		 234.9    234.9 newOrder
		   1m35s        0            2.0            2.3      5.5      5.8      5.8      5.8 orderStatus
		   1m35s        0           32.0           23.1     24.1    151.0    209.7    209.7 payment
		   1m35s        0            5.0            2.3     23.1     46.1     46.1     46.1 stockLevel
		   1m36s        0            2.0            2.3     58.7     83.9     83.9     83.9 delivery
		   1m36s        0           24.0           23.1     54.5    218.1    268.4    268.4 newOrder
		   1m36s        0            3.0            2.3      5.2      6.6      6.6      6.6 orderStatus
		   1m36s        0           29.0           23.1     27.3    192.9    302.0    302.0 payment
		   1m36s        0            1.0            2.3     21.0     21.0     21.0     21.0 stockLevel
		Error: error in payment: ERROR: request timestamp 1561560178.791677254,9 less than txn timestamp 1561560178.794447862,1 (SQLSTATE XXUUU)
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_34.74.1.50_2019-06-26T14:41:21Z: exit status 1
		: exit status 1
	cluster.go:1870,tpcc.go:177,tpcc.go:286,test.go:1249: Goexit() was called

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/36b74f0373b040aa873d4c23b415d94deba88378

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc-nowait/nodes=3/w=1 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1376442&tab=buildLog

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190708-1376442/tpcc-nowait/nodes=3/w=1/run_1
	cluster.go:1724,tpcc.go:173,cluster.go:2067,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1562566386-14-n4cpu16:4 -- ./workload run tpcc --warehouses=1 --histograms=logs/stats.json --wait=false --ramp=5m0s --duration=10m0s {pgurl:1-3} returned:
		stderr:
		
		stdout:
		2      5.2      5.2 orderStatus
		   2m12s        0           33.0           29.6     35.7    184.5    251.7    251.7 payment
		   2m12s        0            1.0            3.0      7.9      7.9      7.9      7.9 stockLevel
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   2m13s        0            4.0            3.0     41.9     48.2     48.2     48.2 delivery
		   2m13s        0           33.0           29.7     41.9    109.1    302.0    302.0 newOrder
		   2m13s        0            3.0            3.0      5.0      5.8      5.8      5.8 orderStatus
		   2m13s        0           34.0           29.7     17.8    167.8    226.5    226.5 payment
		   2m13s        0            4.0            3.0      8.9     16.8     16.8     16.8 stockLevel
		Error: error in newOrder: ERROR: request timestamp 1562576743.619883053,43 less than txn timestamp 1562576743.620238197,1 (SQLSTATE XXUUU)
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_35.229.77.249_2019-07-08T09:03:28Z: exit status 1
		: exit status 1
	cluster.go:2088,tpcc.go:183,tpcc.go:292,test_runner.go:681: Goexit() was called

@johnrk-zz
Copy link

@nvanbenschoten , can we close this? or is this being re-opened by teamcity noise?

@nvanbenschoten
Copy link
Member

Yes, let's close this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. S-2 Medium-high impact: many users impacted, risks of availability and difficult-to-fix data errors
Projects
None yet
Milestone
19.2
Development

No branches or pull requests
5 participants
@andreimatei @johnrk-zz @tbg @nvanbenschoten @cockroach-teamcity