Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: tpcc/mixed-headroom/n5cpu16 failed #42081

Closed
cockroach-teamcity opened this issue Oct 31, 2019 · 16 comments · Fixed by #43001
Closed

roachtest: tpcc/mixed-headroom/n5cpu16 failed #42081

cockroach-teamcity opened this issue Oct 31, 2019 · 16 comments · Fixed by #43001
Assignees
@dt @itsbilal
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone
19.2

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/0f473848083559c8a98be032949df9428068c223

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1565997&tab=artifacts#/tpcc/mixed-headroom/n5cpu16

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191031-1565997/tpcc/mixed-headroom/n5cpu16/run_1
	test_runner.go:712: test timed out (10h0m0s)
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Oct 31, 2019
@cockroach-teamcity cockroach-teamcity added this to the 19.2 milestone Oct 31, 2019
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/b9dc16b9d6cbd4627bac4b6329e87e29e52401b9

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1571295&tab=artifacts#/tpcc/mixed-headroom/n5cpu16

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191103-1571295/tpcc/mixed-headroom/n5cpu16/run_1
	test_runner.go:712: test timed out (10h0m0s)

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/8b9f54761adc58eb9aecbf9b26f1a7987d8a01e5

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1573251&tab=artifacts#/tpcc/mixed-headroom/n5cpu16

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191105-1573251/tpcc/mixed-headroom/n5cpu16/run_1
	test_runner.go:712: test timed out (10h0m0s)

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/33b96613ae532b25a1b6b716453bece9b60ba2d6

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1583742&tab=artifacts#/tpcc/mixed-headroom/n5cpu16

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191109-1583742/tpcc/mixed-headroom/n5cpu16/run_1
	tpcc.go:256,test_runner.go:697: prev version not set for version: 20.1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/35e138aa3c2be545fb4e17a85ea6f1b8d6525e53

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1584763&tab=artifacts#/tpcc/mixed-headroom/n5cpu16

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191110-1584763/tpcc/mixed-headroom/n5cpu16/run_1
	tpcc.go:256,test_runner.go:697: prev version not set for version: 20.1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/0e9dd73f803247cdcfd06f51ce6b23396af1b9f5

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1587121&tab=artifacts#/tpcc/mixed-headroom/n5cpu16

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191112-1587121/tpcc/mixed-headroom/n5cpu16/run_1
	tpcc.go:256,test_runner.go:697: prev version not set for version: 20.1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/35e138aa3c2be545fb4e17a85ea6f1b8d6525e53

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1587967&tab=artifacts#/tpcc/mixed-headroom/n5cpu16

The test failed on branch=provisional_201911111508_v20.1.0-alpha.20191118, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191112-1587967/tpcc/mixed-headroom/n5cpu16/run_1
	tpcc.go:256,test_runner.go:697: prev version not set for version: 20.1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/f97dc13163020a032b098ef3eb88e4d9f54a04ba

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1613952&tab=artifacts#/tpcc/mixed-headroom/n5cpu16

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191127-1613952/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1783,tpcc.go:136,tpcc.go:137,tpcc.go:148,tpcc.go:261,test_runner.go:697: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1613952-1574840180-83-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned:
		stderr:
		
		stdout:
		I191127 12:50:32.218100 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		I191127 12:50:36.611033 114 ccl/workloadccl/fixture.go:547  loaded 7.8 MiB table item in 4.385207592s (100000 rows, 0 index entries, 1.8 MiB)
		I191127 12:50:36.638060 108 ccl/workloadccl/fixture.go:547  loaded 53 KiB table warehouse in 4.412168632s (1000 rows, 0 index entries, 12 KiB)
		I191127 12:50:40.038085 109 ccl/workloadccl/fixture.go:547  loaded 1006 KiB table district in 7.811985002s (10000 rows, 0 index entries, 129 KiB)
		Error: restoring fixture: pq: importing 48 ranges: importing span /Table/56/1/{491/5/-522-528/5/-2907}: adding to batch: /Table/54/1/509/10/-1688/0/1558478069.788044469,0 -> /TUPLE/4:4:Int/2502/1:5:Time/2006-01-02T15:04:05Z/1:6:Int/3/1:7:Int/12/1:8:Int/1: addsstable [/Table/54/1/491/5/-522/0,/Table/54/1/509/10/-1689/0/NULL): closing sstable iterator: leveldb/table: invalid table (bad magic number)
		Error:  exit status 1
		: exit status 1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/581a2bff2f50d5626f68bc1d51af0cf05b455dfc

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1618924&tab=artifacts#/tpcc/mixed-headroom/n5cpu16

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191130-1618924/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1783,tpcc.go:136,tpcc.go:137,tpcc.go:148,tpcc.go:261,test_runner.go:697: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1618924-1575099071-78-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned:
		stderr:
		
		stdout:
		.go:547  loaded 7.8 MiB table item in 1.053288326s (100000 rows, 0 index entries, 7.4 MiB)
		I191130 12:36:05.353443 41 ccl/workloadccl/fixture.go:547  loaded 1006 KiB table district in 6.670834414s (10000 rows, 0 index entries, 151 KiB)
		Error: restoring fixture: pq: importing 1101 ranges: importing span /Table/59/1/{9/12612-10/17214}: adding to batch: /Table/55/1/9/64968/0/1558478069.788080604,0 -> /TUPLE/3:3:Int/38/1:4:Bytes/5sOwsQAp97wRmprMRUOrR0Pz/1:5:Bytes/9SqSNE0OfEFYXxr1xZjUtAyk/1:6:Bytes/j74LyZZm219q1W7uHxB1velF/1:7:Bytes/xaj3oEIrdSD0PHgI0TLe2WEG/1:8:Bytes/p9oREhYxUlsvrl6HW5yLnRwy/1:9:Bytes/hNgSeHcuJDBsft2DuvtOWxct/1:10:Bytes/8Vpg5sN9h7MSe8HYWxs95M4F/1:11:Bytes/btM5yYZqojxriAEkt6Mwn1wY/1:12:Bytes/qllswemRSy4LNcl1EZVgtInh/1:13:Bytes/vLr1aF6o5SalC1ePRecOZnpd/1:14:Int/0/1:15:Int/0/1:16:Int/0/1:17:Bytes/GFuhIxB0W32l4MZVHvNBORIGINALeYf1EXbESVoqh2BGWr: addsstable [/Table/55/1/9/12612/0,/Table/55/1/9/64967/0/NULL): closing sstable iterator: leveldb/table: invalid table (bad magic number)
		Error:  exit status 1
		: exit status 1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/6c13f01ef0d999095a16345b21fc455648796e0c

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1619815&tab=artifacts#/tpcc/mixed-headroom/n5cpu16

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191202-1619815/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1783,tpcc.go:136,tpcc.go:137,tpcc.go:148,tpcc.go:261,test_runner.go:697: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1619815-1575271421-82-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned:
		stderr:
		
		stdout:
		I191202 12:28:51.743537 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		I191202 12:28:55.043127 134 ccl/workloadccl/fixture.go:547  loaded 7.8 MiB table item in 3.287539934s (100000 rows, 0 index entries, 2.4 MiB)
		I191202 12:29:02.546961 80 ccl/workloadccl/fixture.go:547  loaded 53 KiB table warehouse in 10.79160894s (1000 rows, 0 index entries, 4.9 KiB)
		Error: restoring fixture: pq: importing 1 ranges: importing span /Table/54/{1-2}: addsstable [/Table/53/1/0/1/0,/Table/53/1/999/10/0/NULL): closing sstable iterator: leveldb/table: invalid table (bad magic number)
		Error:  exit status 1
		: exit status 1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/d3574ad671bd3631e780510235485681720c2b8f

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1622074&tab=artifacts#/tpcc/mixed-headroom/n5cpu16

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191203-1622074/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1783,tpcc.go:136,tpcc.go:137,tpcc.go:148,tpcc.go:261,test_runner.go:697: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1622074-1575358204-80-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned:
		stderr:
		
		stdout:
		91203 12:32:57.191784 119 ccl/workloadccl/fixture.go:547  loaded 1006 KiB table district in 4.098385366s (10000 rows, 0 index entries, 246 KiB)
		I191203 12:32:59.689556 124 ccl/workloadccl/fixture.go:547  loaded 7.8 MiB table item in 6.596135869s (100000 rows, 0 index entries, 1.2 MiB)
		I191203 12:33:19.400416 118 ccl/workloadccl/fixture.go:547  loaded 53 KiB table warehouse in 26.307123676s (1000 rows, 0 index entries, 2.0 KiB)
		I191203 12:33:21.444818 123 ccl/workloadccl/fixture.go:547  loaded 126 MiB table new_order in 28.351420628s (9000000 rows, 0 index entries, 4.4 MiB)
		Error: restoring fixture: pq: importing 794 ranges: importing span /Table/58/1/5{0/6/-1798/2-2/5/-551/6}: adding to batch: /Table/55/1/51/6/-2720/4/0/1558478069.788046902,0 -> /TUPLE/5:5:Int/90847/1:6:Int/51/2:8:Int/5/1:9:Decimal/7696.07/1:10:Bytes/dakmpv4DlL4V24e0jBcxKZuL: addsstable [/Table/55/1/50/6/-1798/2/0,/Table/55/1/51/6/-2720/3/0/NULL): closing sstable iterator: leveldb/table: invalid table (bad magic number)
		Error:  exit status 1
		: exit status 1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/1da69d917105a0280aad10e86a7ee8eb2059cc92

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1623285&tab=artifacts#/tpcc/mixed-headroom/n5cpu16

The test failed on branch=provisional_201912031738_v20.1.0-alpha20191209, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191203-1623285/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1783,tpcc.go:136,tpcc.go:137,tpcc.go:148,tpcc.go:261,test_runner.go:697: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1623285-1575402335-82-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned:
		stderr:
		
		stdout:
		I191204 01:12:36.919846 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		I191204 01:12:37.474691 108 ccl/workloadccl/fixture.go:547  loaded 53 KiB table warehouse in 545.164956ms (1000 rows, 0 index entries, 97 KiB)
		I191204 01:12:37.586959 109 ccl/workloadccl/fixture.go:547  loaded 1006 KiB table district in 657.497388ms (10000 rows, 0 index entries, 1.5 MiB)
		I191204 01:12:41.577462 146 ccl/workloadccl/fixture.go:547  loaded 7.8 MiB table item in 4.647853946s (100000 rows, 0 index entries, 1.7 MiB)
		Error: restoring fixture: pq: importing 48 ranges: importing span /Table/56/1/6{39/2/-1052-76/1/-451}: adding to batch: /Table/55/1/657/7/-2226/0/1558478069.788044469,0 -> /TUPLE/4:4:Int/1207/1:5:Time/2006-01-02T15:04:05Z/2:7:Int/14/1:8:Int/1: addsstable [/Table/55/1/639/2/-1052/0,/Table/55/1/657/7/-2227/0/NULL): closing sstable iterator: leveldb/table: invalid table (bad magic number)
		Error:  exit status 1
		: exit status 1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/ed717cbaf741e3a32c76db25b16a59dc2a8221d7

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1624103&tab=artifacts#/tpcc/mixed-headroom/n5cpu16

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191204-1624103/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1783,tpcc.go:136,tpcc.go:137,tpcc.go:148,tpcc.go:261,test_runner.go:697: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1624103-1575445328-82-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned:
		stderr:
		
		stdout:
		9000000 rows, 0 index entries, 4.9 MiB)
		I191204 12:55:30.369398 124 ccl/workloadccl/fixture.go:547  loaded 1006 KiB table district in 44.011785948s (10000 rows, 0 index entries, 23 KiB)
		I191204 12:55:30.459876 123 ccl/workloadccl/fixture.go:547  loaded 53 KiB table warehouse in 44.102183552s (1000 rows, 0 index entries, 1.2 KiB)
		Error: restoring fixture: pq: importing 155 ranges: importing span /Table/61/1/{15/"\x03\xd8\a\x04G\xd4B\x80\x80\x00\x00\x00\x00\x06ߔ"/PrefixEnd-30/"\a\xb3I&r\xe9J\xc0\x80\x00\x00\x00\x00\r\xc4\xef"/PrefixEnd}: adding to batch: /Table/57/1/22/"\x05ŵ9\xc1{H@\x80\x00\x00\x00\x00\nRY"/0/1558478069.788578325,0 -> /TUPLE/2:2:Int/1442/1:3:Int/6/1:4:Int/22/1:5:Int/6/2:7:Time/2006-01-02T15:04:05Z/1:8:Decimal/10.00/1:9:Bytes/qF9Xwn1ay3R4SEGEWjN9: addsstable [/Table/57/1/15/"\x03\xd8\a\x93r8K\xc0\x80\x00\x00\x00\x00\x06ߕ"/0,/Table/57/1/22/"\x05Ŵ\xaa\x97\x17O\x00\x80\x00\x00\x00\x00\nRX"/0/NULL): closing sstable iterator: leveldb/table: invalid table (bad magic number)
		Error:  exit status 1
		: exit status 1

@tbg
Copy link
Member

tbg commented Dec 4, 2019

cc @dt and perhaps @petermattis

@tbg tbg assigned dt and unassigned andreimatei Dec 4, 2019
@dt
Copy link
Member

dt commented Dec 4, 2019

@itsbilal has been poking at how we write our SSTables recently, including the use (or not) of the leveldb format. Any chance this is related?

@dt
Copy link
Member

dt commented Dec 4, 2019 •
edited
Loading

looks like the first occurrence was 7 days ago, on a SHA just a couple commits after #42763 merged so the timing certainly lines up

@itsbilal itsbilal mentioned this issue Dec 5, 2019
Merged
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/28f216e1bd53da872a759a98779144c7f70f33a3

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1629573&tab=artifacts#/tpcc/mixed-headroom/n5cpu16

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191206-1629573/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1783,tpcc.go:136,tpcc.go:137,tpcc.go:148,tpcc.go:261,test_runner.go:697: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1629573-1575617269-87-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned:
		stderr:
		
		stdout:
		le district in 10.166447982s (10000 rows, 0 index entries, 99 KiB)
		I191206 12:35:01.357217 154 ccl/workloadccl/fixture.go:547  loaded 53 KiB table warehouse in 10.602915286s (1000 rows, 0 index entries, 5.0 KiB)
		I191206 12:35:03.515335 159 ccl/workloadccl/fixture.go:547  loaded 126 MiB table new_order in 12.761224791s (9000000 rows, 0 index entries, 9.8 MiB)
		Error: restoring fixture: pq: importing 155 ranges: importing span /Table/61/1/1{57/"(Qi7\x8aiD\x00\x80\x00\x00\x00\x00H\x18\x1f"-72/",\t_\xb3}(H\x00\x80\x00\x00\x00\x00N\xbe]"/PrefixEnd}: adding to batch: /Table/54/1/164/"*-k+\x80oF\x00\x80\x00\x00\x00\x00KkJ"/0/1558478069.788578325,0 -> /TUPLE/2:2:Int/1667/1:3:Int/8/1:4:Int/164/1:5:Int/8/2:7:Time/2006-01-02T15:04:05Z/1:8:Decimal/10.00/1:9:Bytes/1WZMlSrj45wZJRAMaaHIe4J: addsstable [/Table/54/1/157/"(Qi7\x8aiD\x00\x80\x00\x00\x00\x00H\x18\x1f"/0,/Table/54/1/164/"*-j\x9cV\fL\x00\x80\x00\x00\x00\x00KkI"/0/NULL): closing sstable iterator: leveldb/table: invalid table (bad magic number)
		Error:  exit status 1
		: exit status 1

craig bot pushed a commit that referenced this issue Dec 6, 2019
@itsbilal
Merge #43001
762d0b4 
43001: storage, ccl: Gate AddSSTable()ing RocksDBv2 format SSTs on cluster version r=itsbilal a=itsbilal

Change #42763 caused all SSTs written for ingestion to
be written in the RocksDBv2 format as opposed to the leveldb format.
This turned out to be an issue in mixed-version clusters where not
all nodes can iterate over and ingest RocksDBv2 SSTs; nodes without
commit 2beab58  (so 19.2.* and below) cannot iterate over
these SSTs.

This change reverts back to creating LevelDB SSTs for ingestion
in the SSTBatcher only, unless the minimum cluster version is a 20.1 commit.
Other cases where we make RocksDBv2 SSTs (eg. in replica_raftstorage)
are okay and do not require this check, since those SSTs are ingested
by the same node where they're written.

Fixes #42081 .

Release note: None.

Co-authored-by: Bilal Akhtar <[email protected]>
@craig craig bot closed this as completed in 3b349d5 Dec 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dt dt

@itsbilal itsbilal

Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Milestone
19.2
Development

Successfully merging a pull request may close this issue.

storage, ccl: Gate AddSSTable()ing RocksDBv2 format SSTs on cluster version itsbilal/cockroach
5 participants
@dt @itsbilal @andreimatei @tbg @cockroach-teamcity