roachtest: kv/gracefuldraining/nodes=3 failed [skipped] #33501

cockroach-teamcity · 2019-01-04T10:15:00Z

SHA: https://github.com/cockroachdb/cockroach/commits/d6adf24cae788d7cd967feadae8e9c0388ce5273

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1079910&tab=buildLog

The test failed on master:
	test.go:701,kv.go:267: QPS of 337.90 at time 2019-01-04 10:08:30 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1546596500000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596510000000000 Value:337.9 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596520000000000 Value:999.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596530000000000 Value:999.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596540000000000 Value:998.1000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596550000000000 Value:999.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596560000000000 Value:999.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596570000000000 Value:995.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596580000000000 Value:993.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596590000000000 Value:994.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596600000000000 Value:996 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596610000000000 Value:997.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596620000000000 Value:996.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596630000000000 Value:994.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596640000000000 Value:998.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596650000000000 Value:998.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596660000000000 Value:999.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596670000000000 Value:998.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596680000000000 Value:1000.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596690000000000 Value:999.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596700000000000 Value:997.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596710000000000 Value:995.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596720000000000 Value:998.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596730000000000 Value:999.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596740000000000 Value:997.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596750000000000 Value:997.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546596760000000000 Value:997.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2019-01-05T09:57:48Z

SHA: https://github.com/cockroachdb/cockroach/commits/6a61a7a44171580de390edf98e33156e351ccac7

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1081629&tab=buildLog

The test failed on master:
	test.go:701,kv.go:267: QPS of 659.90 at time 2019-01-05 09:50:20 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1546681810000000000 Value:0.9 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546681820000000000 Value:659.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546681830000000000 Value:997.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546681840000000000 Value:999 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546681850000000000 Value:997.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546681860000000000 Value:997.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546681870000000000 Value:996.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546681880000000000 Value:989 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546681890000000000 Value:986.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546681900000000000 Value:986.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546681910000000000 Value:982.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546681920000000000 Value:990.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546681930000000000 Value:998.1000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546681940000000000 Value:998.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546681950000000000 Value:999.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546681960000000000 Value:1000 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546681970000000000 Value:999.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546681980000000000 Value:1000.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546681990000000000 Value:999.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546682000000000000 Value:998.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546682010000000000 Value:999.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546682020000000000 Value:998.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546682030000000000 Value:999.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546682040000000000 Value:997.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546682050000000000 Value:998.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546682060000000000 Value:996.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546682070000000000 Value:1000 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-01-06T09:52:28Z

SHA: https://github.com/cockroachdb/cockroach/commits/99064bf2797cce49e2d56d0324a3344cf2312aca

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1082456&tab=buildLog

The test failed on master:
	test.go:701,kv.go:267: QPS of 543.30 at time 2019-01-06 09:45:20 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1546767910000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546767920000000000 Value:543.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546767930000000000 Value:998.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546767940000000000 Value:998.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546767950000000000 Value:997.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546767960000000000 Value:999.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546767970000000000 Value:999.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546767980000000000 Value:995.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546767990000000000 Value:984.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546768000000000000 Value:991.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546768010000000000 Value:995 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546768020000000000 Value:995.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546768030000000000 Value:999.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546768040000000000 Value:999.1000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546768050000000000 Value:998.1000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546768060000000000 Value:1000.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546768070000000000 Value:999.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546768080000000000 Value:999.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546768090000000000 Value:999.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546768100000000000 Value:999.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546768110000000000 Value:999.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546768120000000000 Value:1000.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546768130000000000 Value:999 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546768140000000000 Value:998.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546768150000000000 Value:998 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546768160000000000 Value:998.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546768170000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

tbg · 2019-01-07T10:50:20Z

@a-robinson are you going to take a look at this?

cockroach-teamcity · 2019-01-08T10:35:19Z

SHA: https://github.com/cockroachdb/cockroach/commits/dadcaa493bca02996ebd87c0c85106e566893ce2

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1083956&tab=buildLog

The test failed on master:
	test.go:701,kv.go:267: QPS of 544.40 at time 2019-01-08 10:28:50 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1546943320000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943330000000000 Value:544.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943340000000000 Value:998.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943350000000000 Value:997.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943360000000000 Value:998.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943370000000000 Value:999.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943380000000000 Value:998.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943390000000000 Value:997.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943400000000000 Value:993.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943410000000000 Value:996.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943420000000000 Value:992.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943430000000000 Value:995.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943440000000000 Value:997.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943450000000000 Value:996.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943460000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943470000000000 Value:1000.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943480000000000 Value:999.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943490000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943500000000000 Value:999.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943510000000000 Value:999.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943520000000000 Value:999.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943530000000000 Value:997.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943540000000000 Value:999.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943550000000000 Value:997.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943560000000000 Value:999.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943570000000000 Value:994 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546943580000000000 Value:993.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-01-09T01:43:11Z

SHA: https://github.com/cockroachdb/cockroach/commits/12e28159b1d8b63b56d6a48f22ebbb5c75e8ee5c

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1085451&tab=buildLog

The test failed on provisional_201901081731_v2.2.0-alpha-20190114:
	test.go:701,kv.go:267: QPS of 558.20 at time 2019-01-09 01:36:00 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1546997750000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997760000000000 Value:558.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997770000000000 Value:998.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997780000000000 Value:997.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997790000000000 Value:996.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997800000000000 Value:998.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997810000000000 Value:1000.1000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997820000000000 Value:993.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997830000000000 Value:985.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997840000000000 Value:985.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997850000000000 Value:984.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997860000000000 Value:986.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997870000000000 Value:996.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997880000000000 Value:993.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997890000000000 Value:1000.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997900000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997910000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997920000000000 Value:1000.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997930000000000 Value:1000 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997940000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997950000000000 Value:996.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997960000000000 Value:998 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997970000000000 Value:998.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997980000000000 Value:998.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546997990000000000 Value:997.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546998000000000000 Value:995.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1546998010000000000 Value:998.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-01-09T09:56:56Z

SHA: https://github.com/cockroachdb/cockroach/commits/cff3e8870a9f2f84feb37415f223ce6922487504

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1085650&tab=buildLog

The test failed on master:
	test.go:701,kv.go:267: QPS of 450.10 at time 2019-01-09 09:50:30 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1547027420000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027430000000000 Value:450.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027440000000000 Value:999.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027450000000000 Value:999.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027460000000000 Value:998 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027470000000000 Value:999.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027480000000000 Value:1000.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027490000000000 Value:992.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027500000000000 Value:985.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027510000000000 Value:988.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027520000000000 Value:991.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027530000000000 Value:994.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027540000000000 Value:996.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027550000000000 Value:998.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027560000000000 Value:1000.1000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027570000000000 Value:997.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027580000000000 Value:999.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027590000000000 Value:998.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027600000000000 Value:999.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027610000000000 Value:1000 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027620000000000 Value:998.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027630000000000 Value:996.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027640000000000 Value:999.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027650000000000 Value:998.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027660000000000 Value:997.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027670000000000 Value:998.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547027680000000000 Value:997.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-01-11T09:53:41Z

SHA: https://github.com/cockroachdb/cockroach/commits/f5e3c29b2eed92868cf3d449575283e2e383f199

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1088848&tab=buildLog

The test failed on master:
	test.go:701,kv.go:267: QPS of 686.50 at time 2019-01-11 09:47:10 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1547200020000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200030000000000 Value:686.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200040000000000 Value:999.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200050000000000 Value:999.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200060000000000 Value:998 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200070000000000 Value:999.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200080000000000 Value:999.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200090000000000 Value:993.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200100000000000 Value:991.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200110000000000 Value:990.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200120000000000 Value:991 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200130000000000 Value:989 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200140000000000 Value:989 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200150000000000 Value:991.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200160000000000 Value:998.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200170000000000 Value:998.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200180000000000 Value:999.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200190000000000 Value:999.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200200000000000 Value:1000 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200210000000000 Value:998.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200220000000000 Value:989.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200230000000000 Value:993.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200240000000000 Value:993.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200250000000000 Value:998.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200260000000000 Value:999.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200270000000000 Value:998.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547200280000000000 Value:998.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-01-12T01:23:26Z

SHA: https://github.com/cockroachdb/cockroach/commits/5058e4a6b1a20ec208505029eddb5b5e25cb7d65

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1089946&tab=buildLog

The test failed on provisional_201901110710_v2.2.0-alpha-20190114:
	test.go:701,kv.go:267: QPS of 333.90 at time 2019-01-12 01:15:50 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1547255740000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255750000000000 Value:333.9 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255760000000000 Value:999.1000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255770000000000 Value:998.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255780000000000 Value:999.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255790000000000 Value:1000.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255800000000000 Value:998.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255810000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255820000000000 Value:995.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255830000000000 Value:996.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255840000000000 Value:996.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255850000000000 Value:998.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255860000000000 Value:1000 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255870000000000 Value:999.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255880000000000 Value:998.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255890000000000 Value:999.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255900000000000 Value:999.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255910000000000 Value:999.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255920000000000 Value:999.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255930000000000 Value:999.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255940000000000 Value:999.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255950000000000 Value:999.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255960000000000 Value:998.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255970000000000 Value:999.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255980000000000 Value:999.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547255990000000000 Value:998.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547256000000000000 Value:1000 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-01-19T09:50:38Z

SHA: https://github.com/cockroachdb/cockroach/commits/0e848c9ef932f8f2a2b016580feedf2275c8e505

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1100545&tab=buildLog

The test failed on master:
	test.go:732,kv.go:255: QPS of 430.30 at time 2019-01-19 09:43:40 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1547891010000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891020000000000 Value:430.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891030000000000 Value:998.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891040000000000 Value:999.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891050000000000 Value:998 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891060000000000 Value:999.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891070000000000 Value:997.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891080000000000 Value:998.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891090000000000 Value:996.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891100000000000 Value:991.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891110000000000 Value:996.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891120000000000 Value:997 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891130000000000 Value:996.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891140000000000 Value:992 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891150000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891160000000000 Value:998.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891170000000000 Value:998.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891180000000000 Value:1000.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891190000000000 Value:999.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891200000000000 Value:998.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891210000000000 Value:999.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891220000000000 Value:999.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891230000000000 Value:995.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891240000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891250000000000 Value:999 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891260000000000 Value:997.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1547891270000000000 Value:999 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-01-21T09:52:37Z

SHA: https://github.com/cockroachdb/cockroach/commits/792e95bf111fc544b06b471d6ed5b0d8fe3acf5e

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1102546&tab=buildLog

The test failed on master:
	test.go:732,kv.go:255: QPS of 646.90 at time 2019-01-21 09:45:30 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1548063920000000000 Value:126.60000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548063930000000000 Value:646.9 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548063940000000000 Value:999.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548063950000000000 Value:998.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548063960000000000 Value:998.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548063970000000000 Value:998 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548063980000000000 Value:998 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548063990000000000 Value:992 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548064000000000000 Value:983.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548064010000000000 Value:987 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548064020000000000 Value:991.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548064030000000000 Value:998.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548064040000000000 Value:998.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548064050000000000 Value:997.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548064060000000000 Value:997.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548064070000000000 Value:998.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548064080000000000 Value:998.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548064090000000000 Value:998.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548064100000000000 Value:999.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548064110000000000 Value:999.1000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548064120000000000 Value:998.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548064130000000000 Value:998.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548064140000000000 Value:997.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548064150000000000 Value:997.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548064160000000000 Value:999 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548064170000000000 Value:999.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548064180000000000 Value:999.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-01-23T09:58:59Z

SHA: https://github.com/cockroachdb/cockroach/commits/f37d09cd3cdd32f4d4894611cfd60caf25c10fff

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1105096&tab=buildLog

The test failed on master:
	test.go:748,kv.go:255: QPS of 38.00 at time 2019-01-23 09:52:00 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1548237110000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237120000000000 Value:38 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237130000000000 Value:998 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237140000000000 Value:998.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237150000000000 Value:998.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237160000000000 Value:1000.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237170000000000 Value:999.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237180000000000 Value:997.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237190000000000 Value:991.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237200000000000 Value:990.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237210000000000 Value:987.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237220000000000 Value:993.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237230000000000 Value:996.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237240000000000 Value:996.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237250000000000 Value:995.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237260000000000 Value:998.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237270000000000 Value:999.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237280000000000 Value:999.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237290000000000 Value:999.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237300000000000 Value:999.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237310000000000 Value:997.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237320000000000 Value:992.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237330000000000 Value:994.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237340000000000 Value:997.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237350000000000 Value:997.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237360000000000 Value:997.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548237370000000000 Value:998.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

tbg · 2019-01-23T12:30:32Z

@petermattis can you look at this at some point? I'll snatch it from you if I clear out my queue before you, but this is somewhat lower priority than what I have right now (same might be true for you).

cockroach-teamcity · 2019-01-24T10:04:02Z

SHA: https://github.com/cockroachdb/cockroach/commits/295b6ae3142518f04a5771c79ce171043697ee1f

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1107301&tab=buildLog

The test failed on master:
	test.go:748,kv.go:255: QPS of 844.10 at time 2019-01-24 09:56:50 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1548323800000000000 Value:223.4 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548323810000000000 Value:844.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548323820000000000 Value:998.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548323830000000000 Value:997.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548323840000000000 Value:998.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548323850000000000 Value:999.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548323860000000000 Value:997.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548323870000000000 Value:991.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548323880000000000 Value:990.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548323890000000000 Value:992.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548323900000000000 Value:993 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548323910000000000 Value:992.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548323920000000000 Value:992.1000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548323930000000000 Value:996.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548323940000000000 Value:999.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548323950000000000 Value:999.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548323960000000000 Value:998.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548323970000000000 Value:997.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548323980000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548323990000000000 Value:997.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548324000000000000 Value:992.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548324010000000000 Value:991.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548324020000000000 Value:993.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548324030000000000 Value:998.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548324040000000000 Value:999.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548324050000000000 Value:998.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548324060000000000 Value:996.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-01-25T09:51:14Z

SHA: https://github.com/cockroachdb/cockroach/commits/2952f08ba7260967d7dfd10addbfe80b51d2b8ed

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1109027&tab=buildLog

The test failed on master:
	test.go:748,kv.go:255: QPS of 436.20 at time 2019-01-25 09:44:40 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1548409470000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409480000000000 Value:436.20000000000005 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409490000000000 Value:999.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409500000000000 Value:997.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409510000000000 Value:997.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409520000000000 Value:999.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409530000000000 Value:997.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409540000000000 Value:992.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409550000000000 Value:989.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409560000000000 Value:994.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409570000000000 Value:996.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409580000000000 Value:993.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409590000000000 Value:995.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409600000000000 Value:995 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409610000000000 Value:999.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409620000000000 Value:999.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409630000000000 Value:998.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409640000000000 Value:998.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409650000000000 Value:1000.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409660000000000 Value:999.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409670000000000 Value:995.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409680000000000 Value:994.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409690000000000 Value:998.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409700000000000 Value:998.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409710000000000 Value:1000.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409720000000000 Value:997.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548409730000000000 Value:998 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-01-26T10:06:05Z

SHA: https://github.com/cockroachdb/cockroach/commits/dc2fbcdc0dccb8cc676fc67370375bab36b3cff0

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1110068&tab=buildLog

The test failed on master:
	test.go:748,kv.go:255: QPS of 236.30 at time 2019-01-26 09:58:50 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1548496720000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496730000000000 Value:236.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496740000000000 Value:997.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496750000000000 Value:996.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496760000000000 Value:994.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496770000000000 Value:998.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496780000000000 Value:908.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496790000000000 Value:995.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496800000000000 Value:977.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496810000000000 Value:985.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496820000000000 Value:987.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496830000000000 Value:992.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496840000000000 Value:994.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496850000000000 Value:992.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496860000000000 Value:998.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496870000000000 Value:997.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496880000000000 Value:998.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496890000000000 Value:999.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496900000000000 Value:996.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496910000000000 Value:998.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496920000000000 Value:992.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496930000000000 Value:993.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496940000000000 Value:989.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496950000000000 Value:994.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496960000000000 Value:997.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496970000000000 Value:995.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548496980000000000 Value:993.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-01-27T09:51:05Z

SHA: https://github.com/cockroachdb/cockroach/commits/8cbeb534432b81c57564956ed7d645b854b426be

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1111300&tab=buildLog

The test failed on master:
	test.go:748,kv.go:255: QPS of 562.00 at time 2019-01-27 09:44:20 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1548582250000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582260000000000 Value:562 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582270000000000 Value:998.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582280000000000 Value:1000 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582290000000000 Value:998.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582300000000000 Value:997.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582310000000000 Value:997.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582320000000000 Value:992 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582330000000000 Value:990.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582340000000000 Value:991.1000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582350000000000 Value:992.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582360000000000 Value:997.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582370000000000 Value:995.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582380000000000 Value:995.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582390000000000 Value:996.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582400000000000 Value:999.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582410000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582420000000000 Value:998.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582430000000000 Value:999.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582440000000000 Value:997.1000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582450000000000 Value:993.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582460000000000 Value:995.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582470000000000 Value:996.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582480000000000 Value:999.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582490000000000 Value:997.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582500000000000 Value:999.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548582510000000000 Value:1000.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-01-28T09:59:28Z

SHA: https://github.com/cockroachdb/cockroach/commits/9f084ad576e85756c5c5a7e41335d9aa2d3eee30

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1112101&tab=buildLog

The test failed on master:
	test.go:748,kv.go:255: QPS of 443.60 at time 2019-01-28 09:52:20 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1548669130000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669140000000000 Value:443.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669150000000000 Value:997.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669160000000000 Value:997.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669170000000000 Value:998 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669180000000000 Value:999.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669190000000000 Value:999.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669200000000000 Value:993.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669210000000000 Value:981.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669220000000000 Value:985.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669230000000000 Value:985.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669240000000000 Value:997.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669250000000000 Value:998.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669260000000000 Value:998.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669270000000000 Value:994.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669280000000000 Value:999.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669290000000000 Value:999.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669300000000000 Value:999.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669310000000000 Value:999.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669320000000000 Value:997.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669330000000000 Value:995.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669340000000000 Value:995.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669350000000000 Value:998.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669360000000000 Value:996.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669370000000000 Value:997.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669380000000000 Value:997.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548669390000000000 Value:999.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-01-29T09:53:46Z

SHA: https://github.com/cockroachdb/cockroach/commits/e10fb557b11b5ff1b8609aa963da23c37a1143c8

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1113854&tab=buildLog

The test failed on master:
	test.go:748,kv.go:255: QPS of 186.30 at time 2019-01-29 09:46:10 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1548755160000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755170000000000 Value:186.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755180000000000 Value:997.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755190000000000 Value:999.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755200000000000 Value:998.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755210000000000 Value:999.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755220000000000 Value:998.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755230000000000 Value:997.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755240000000000 Value:992.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755250000000000 Value:990.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755260000000000 Value:995.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755270000000000 Value:996.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755280000000000 Value:993.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755290000000000 Value:993.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755300000000000 Value:999.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755310000000000 Value:999.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755320000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755330000000000 Value:999.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755340000000000 Value:998.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755350000000000 Value:999.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755360000000000 Value:994.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755370000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755380000000000 Value:996.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755390000000000 Value:998.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755400000000000 Value:999.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755410000000000 Value:994.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548755420000000000 Value:995.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-01-31T09:58:02Z

SHA: https://github.com/cockroachdb/cockroach/commits/f1221451eb0aaa3e7b7d27eefeaff79bd39642ad

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1117793&tab=buildLog

The test failed on release-2.1:
	test.go:748,kv.go:255: QPS of 641.50 at time 2019-01-31 09:51:20 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1548928270000000000 Value:25.700000000000003 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928280000000000 Value:641.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928290000000000 Value:992.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928300000000000 Value:996.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928310000000000 Value:994.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928320000000000 Value:993.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928330000000000 Value:996.1000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928340000000000 Value:980.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928350000000000 Value:969.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928360000000000 Value:973 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928370000000000 Value:970.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928380000000000 Value:963 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928390000000000 Value:957.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928400000000000 Value:968.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928410000000000 Value:981 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928420000000000 Value:978.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928430000000000 Value:986.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928440000000000 Value:995.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928450000000000 Value:998.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928460000000000 Value:995 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928470000000000 Value:991.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928480000000000 Value:979.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928490000000000 Value:965.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928500000000000 Value:967.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928510000000000 Value:973.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928520000000000 Value:966.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1548928530000000000 Value:976.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-05-14T10:38:19Z

SHA: https://github.com/cockroachdb/cockroach/commits/ba5c092a726134b73e789c2047f7ec151be7c1a1

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1288263&tab=buildLog

The test failed on branch=master, cloud=gce:
	kv.go:374,test.go:1251: QPS of 576.40 at time 2019-05-14 10:31:20 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1557829870000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557829880000000000 Value:576.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557829890000000000 Value:996.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557829900000000000 Value:998.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557829910000000000 Value:998.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557829920000000000 Value:999.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557829930000000000 Value:999.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557829940000000000 Value:992.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557829950000000000 Value:984.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557829960000000000 Value:989.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557829970000000000 Value:987.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557829980000000000 Value:995 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557829990000000000 Value:998.1000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557830000000000000 Value:999 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557830010000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557830020000000000 Value:999.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557830030000000000 Value:999.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557830040000000000 Value:1000 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557830050000000000 Value:999.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557830060000000000 Value:998.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557830070000000000 Value:999 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557830080000000000 Value:997.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557830090000000000 Value:998 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557830100000000000 Value:999.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557830110000000000 Value:999.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557830120000000000 Value:998.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557830130000000000 Value:1000 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-05-14T11:10:45Z

SHA: https://github.com/cockroachdb/cockroach/commits/dcd4cc5e37ebbcebbbf0f01670811b176a58bf89

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1288281&tab=buildLog

The test failed on branch=release-19.1, cloud=gce:
	kv.go:374,test.go:1251: QPS of 669.40 at time 2019-05-14 11:03:40 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1557831810000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557831820000000000 Value:669.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557831830000000000 Value:997 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557831840000000000 Value:998.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557831850000000000 Value:998.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557831860000000000 Value:997.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557831870000000000 Value:998.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557831880000000000 Value:995.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557831890000000000 Value:991.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557831900000000000 Value:991.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557831910000000000 Value:987.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557831920000000000 Value:995 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557831930000000000 Value:996.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557831940000000000 Value:997.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557831950000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557831960000000000 Value:999.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557831970000000000 Value:999.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557831980000000000 Value:1000.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557831990000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557832000000000000 Value:998.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557832010000000000 Value:996.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557832020000000000 Value:996 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557832030000000000 Value:994.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557832040000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557832050000000000 Value:999.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557832060000000000 Value:998.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557832070000000000 Value:1000.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-05-15T09:37:55Z

SHA: https://github.com/cockroachdb/cockroach/commits/46b82eb637a3f6a1ae563fcad7448fa5458f8298

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1290161&tab=buildLog

The test failed on branch=release-2.1, cloud=gce:
	kv.go:374,test.go:1251: QPS of 696.20 at time 2019-05-15 09:31:50 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1557912700000000000 Value:172.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912710000000000 Value:696.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912720000000000 Value:993.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912730000000000 Value:994.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912740000000000 Value:995.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912750000000000 Value:991 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912760000000000 Value:977.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912770000000000 Value:985.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912780000000000 Value:975.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912790000000000 Value:968.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912800000000000 Value:962.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912810000000000 Value:967.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912820000000000 Value:982 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912830000000000 Value:993.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912840000000000 Value:999.1000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912850000000000 Value:990.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912860000000000 Value:989.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912870000000000 Value:999.1000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912880000000000 Value:999.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912890000000000 Value:998.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912900000000000 Value:996.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912910000000000 Value:995.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912920000000000 Value:996.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912930000000000 Value:987.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912940000000000 Value:987.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912950000000000 Value:976.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1557912960000000000 Value:998.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-05-16T11:21:46Z

SHA: https://github.com/cockroachdb/cockroach/commits/c8bda1de440cfe90cf23a433119d77795cfa0047

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1292152&tab=buildLog

The test failed on branch=release-19.1, cloud=gce:
	kv.go:374,test.go:1251: QPS of 664.50 at time 2019-05-16 11:13:30 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1558005200000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005210000000000 Value:664.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005220000000000 Value:998.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005230000000000 Value:997.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005240000000000 Value:995.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005250000000000 Value:997.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005260000000000 Value:999.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005270000000000 Value:993.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005280000000000 Value:989.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005290000000000 Value:989.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005300000000000 Value:990.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005310000000000 Value:992 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005320000000000 Value:995 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005330000000000 Value:997 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005340000000000 Value:999.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005350000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005360000000000 Value:999 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005370000000000 Value:1000.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005380000000000 Value:999.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005390000000000 Value:996.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005400000000000 Value:996 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005410000000000 Value:993.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005420000000000 Value:992.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005430000000000 Value:996.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005440000000000 Value:999.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005450000000000 Value:998.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558005460000000000 Value:1000 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-05-17T10:25:40Z

SHA: https://github.com/cockroachdb/cockroach/commits/7b2651400b2003d0a381cba9dbfc0b7bc0dfee00

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1293898&tab=buildLog

The test failed on branch=master, cloud=gce:
	kv.go:374,test.go:1251: QPS of 893.40 at time 2019-05-17 10:18:40 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1558088310000000000 Value:1.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088320000000000 Value:893.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088330000000000 Value:999.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088340000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088350000000000 Value:998.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088360000000000 Value:1000.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088370000000000 Value:999.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088380000000000 Value:997.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088390000000000 Value:998.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088400000000000 Value:996.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088410000000000 Value:998.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088420000000000 Value:994.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088430000000000 Value:997.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088440000000000 Value:997.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088450000000000 Value:999.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088460000000000 Value:999.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088470000000000 Value:999.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088480000000000 Value:1000.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088490000000000 Value:1000 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088500000000000 Value:998.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088510000000000 Value:997.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088520000000000 Value:998.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088530000000000 Value:996.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088540000000000 Value:998.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088550000000000 Value:998.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088560000000000 Value:996.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558088570000000000 Value:999.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-05-17T12:08:22Z

SHA: https://github.com/cockroachdb/cockroach/commits/c8bda1de440cfe90cf23a433119d77795cfa0047

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1294577&tab=buildLog

The test failed on branch=release-19.1, cloud=gce:
	kv.go:374,test.go:1251: QPS of 515.50 at time 2019-05-17 12:01:10 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1558094460000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094470000000000 Value:515.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094480000000000 Value:998.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094490000000000 Value:997.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094500000000000 Value:997.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094510000000000 Value:998.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094520000000000 Value:997.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094530000000000 Value:988.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094540000000000 Value:994.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094550000000000 Value:995.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094560000000000 Value:991.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094570000000000 Value:985.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094580000000000 Value:986.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094590000000000 Value:987.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094600000000000 Value:996.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094610000000000 Value:998 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094620000000000 Value:998 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094630000000000 Value:998.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094640000000000 Value:998 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094650000000000 Value:996.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094660000000000 Value:968.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094670000000000 Value:994.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094680000000000 Value:993.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094690000000000 Value:994.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094700000000000 Value:995 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094710000000000 Value:994.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558094720000000000 Value:996.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-05-18T10:17:23Z

SHA: https://github.com/cockroachdb/cockroach/commits/923a3b2a6f4a6492883141092280d1041de1381a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1295056&tab=buildLog

The test failed on branch=master, cloud=gce:
	kv.go:374,test.go:1251: QPS of 541.50 at time 2019-05-18 10:10:40 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1558174230000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174240000000000 Value:541.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174250000000000 Value:999.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174260000000000 Value:998.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174270000000000 Value:998.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174280000000000 Value:999.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174290000000000 Value:999.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174300000000000 Value:987 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174310000000000 Value:987.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174320000000000 Value:984.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174330000000000 Value:987.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174340000000000 Value:995.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174350000000000 Value:999.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174360000000000 Value:998.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174370000000000 Value:1000.1000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174380000000000 Value:999.8 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174390000000000 Value:999.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174400000000000 Value:1000.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174410000000000 Value:1000 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174420000000000 Value:998.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174430000000000 Value:998.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174440000000000 Value:996.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174450000000000 Value:997.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174460000000000 Value:998.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174470000000000 Value:998.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174480000000000 Value:999.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558174490000000000 Value:998.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-05-19T10:23:53Z

SHA: https://github.com/cockroachdb/cockroach/commits/cab299a0ef983f8b4ffe5d724e44587d9665d3a3

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1295811&tab=buildLog

The test failed on branch=master, cloud=gce:
	kv.go:374,test.go:1251: QPS of 841.30 at time 2019-05-19 10:17:20 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1558261030000000000 Value:315 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261040000000000 Value:841.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261050000000000 Value:999.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261060000000000 Value:997.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261070000000000 Value:999.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261080000000000 Value:999.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261090000000000 Value:996.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261100000000000 Value:988.8000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261110000000000 Value:987.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261120000000000 Value:987.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261130000000000 Value:988.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261140000000000 Value:992.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261150000000000 Value:993.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261160000000000 Value:997.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261170000000000 Value:998.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261180000000000 Value:998.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261190000000000 Value:999.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261200000000000 Value:999.5 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261210000000000 Value:998.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261220000000000 Value:994.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261230000000000 Value:991.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261240000000000 Value:990.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261250000000000 Value:992.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261260000000000 Value:997.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261270000000000 Value:998 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261280000000000 Value:998.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558261290000000000 Value:998.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

cockroach-teamcity · 2019-05-21T10:42:20Z

SHA: https://github.com/cockroachdb/cockroach/commits/9671342fead0509bec0913bae4ae1f244660788e

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1298500&tab=buildLog

The test failed on branch=release-19.1, cloud=gce:
	cluster.go:1474,kv.go:294,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1298500-kv-gracefuldraining-nodes-3:4 -- ./workload run kv --init --max-ops=1 --splits 100 {pgurl:1} returned:
		stderr:
		
		stdout:
		I190521 10:40:08.164839 1 workload/workload.go:562  starting 100 splits
		Error: ALTER TABLE kv SPLIT AT VALUES (91320515216384000): pq: splits would be immediately discarded by merge queue; disable the merge queue first by running 'SET CLUSTER SETTING kv.range_merge.queue_enabled = false'
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_34.74.109.229_2019-05-21T10:40:07Z: exit status 1
		: exit status 1

cockroach-teamcity · 2019-05-21T12:39:27Z

SHA: https://github.com/cockroachdb/cockroach/commits/5542939c07947546136d5ae895b7349751534595

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv/gracefuldraining/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1298482&tab=buildLog

The test failed on branch=master, cloud=gce:
	kv.go:374,test.go:1251: QPS of 221.90 at time 2019-05-21 12:33:30 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1558442000000000000 Value:1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442010000000000 Value:221.9 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442020000000000 Value:1000.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442030000000000 Value:1000 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442040000000000 Value:996.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442050000000000 Value:998 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442060000000000 Value:999.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442070000000000 Value:998.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442080000000000 Value:994 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442090000000000 Value:997.3 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442100000000000 Value:993.6000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442110000000000 Value:997.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442120000000000 Value:996.6 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442130000000000 Value:995.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442140000000000 Value:996.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442150000000000 Value:1000.1 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442160000000000 Value:999 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442170000000000 Value:999.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442180000000000 Value:1000.2 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442190000000000 Value:1000.4000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442200000000000 Value:999.3000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442210000000000 Value:998.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442220000000000 Value:998 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442230000000000 Value:999 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442240000000000 Value:997.7 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442250000000000 Value:998.9000000000001 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} {TimestampNanos:1558442260000000000 Value:1000 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}]

tbg · 2019-05-21T13:17:34Z

This test is now skipped (should've done this a lot sooner). It's marked as a starter debug issue.

Quoting from my 4/1/19 post:

I took a quick look at this from the timeseries it does appear that there's a problem - the 550.00 datapoint is surrounded by datapoints that show healthy qps. On the other hand, the workload log shows no moment in time at which the workload thought it had done less than 900+ qps. Something here doesn't square up. I generally would trust the time series more than the built-in computation, but then that also has a lot more moving parts and it may just be an error of arithmetic (perhaps related to the node going down since we use an AVG downsampler?).

I should've done this a long time ago, it's been flaky for five months and failed approximately 120 times (i.e. most of the time). See cockroachdb#33501 (comment) Release note: None

ajwerner · 2019-05-21T13:20:40Z

Thanks! It’s been at the bottom of my todo list for a number of days now and I just didn’t get there.

tbg · 2019-05-21T13:24:46Z

Yep, and before that on my list, and before that on someone else's. One of these days we'll get to it.

37673: roachtest: skip kv/gracefuldraining r=nvanbenschoten a=tbg I should've done this a long time ago, it's been flaky for five months and failed approximately 120 times (i.e. most of the time). See #33501 (comment) Release note: None 37689: partitionccl: error instead of panic for an invalid partitioning r=justinj,jordanlewis a=danhhz If a PARTITION BY LIST both didn't match the column names in the index being partitioned and also tried to partition by more columns than were in the index, we previously panic'd when constructing the error. Closes #37682 Release note (bug fix): Fixed a panic when construting the error message for an invalid partitioning. Co-authored-by: Tobias Schottdorf <[email protected]> Co-authored-by: Daniel Harrison <[email protected]>

cockroach-teamcity added this to the 2.2 milestone Jan 4, 2019

cockroach-teamcity assigned a-robinson Jan 4, 2019

cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Jan 4, 2019

tbg unassigned a-robinson Jan 11, 2019

tbg assigned petermattis Jan 23, 2019

ajkr mentioned this issue May 20, 2019

release: v19.1.1 #37535

Closed

18 tasks

tbg added the E-starter Might be suitable for a starter project for new employees or team members. label May 21, 2019

tbg changed the title ~~roachtest: kv/gracefuldraining/nodes=3 failed~~ roachtest: kv/gracefuldraining/nodes=3 failed [skipped] May 21, 2019

tbg mentioned this issue May 21, 2019

roachtest: skip kv/gracefuldraining #37673

Merged

tbg added the branch-master Failures and bugs on the master branch. label Jan 22, 2020

tbg assigned knz and unassigned ajwerner Aug 27, 2020

knz mentioned this issue Aug 31, 2020

roachtests: re-enable election-after-restart and kv/gracefuldraining/nodes=3 #53689

Merged

craig bot closed this as completed in f2056ba Sep 1, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: kv/gracefuldraining/nodes=3 failed [skipped] #33501

roachtest: kv/gracefuldraining/nodes=3 failed [skipped] #33501

cockroach-teamcity commented Jan 4, 2019

cockroach-teamcity commented Jan 5, 2019

cockroach-teamcity commented Jan 6, 2019

tbg commented Jan 7, 2019

cockroach-teamcity commented Jan 8, 2019

cockroach-teamcity commented Jan 9, 2019

cockroach-teamcity commented Jan 9, 2019

cockroach-teamcity commented Jan 11, 2019

cockroach-teamcity commented Jan 12, 2019

cockroach-teamcity commented Jan 19, 2019

cockroach-teamcity commented Jan 21, 2019

cockroach-teamcity commented Jan 23, 2019

tbg commented Jan 23, 2019

cockroach-teamcity commented Jan 24, 2019

cockroach-teamcity commented Jan 25, 2019

cockroach-teamcity commented Jan 26, 2019

cockroach-teamcity commented Jan 27, 2019

cockroach-teamcity commented Jan 28, 2019

cockroach-teamcity commented Jan 29, 2019

cockroach-teamcity commented Jan 31, 2019

cockroach-teamcity commented May 14, 2019

cockroach-teamcity commented May 14, 2019

cockroach-teamcity commented May 15, 2019

cockroach-teamcity commented May 16, 2019

cockroach-teamcity commented May 17, 2019

cockroach-teamcity commented May 17, 2019

cockroach-teamcity commented May 18, 2019

cockroach-teamcity commented May 19, 2019

cockroach-teamcity commented May 21, 2019

cockroach-teamcity commented May 21, 2019

tbg commented May 21, 2019

ajwerner commented May 21, 2019

tbg commented May 21, 2019

roachtest: kv/gracefuldraining/nodes=3 failed [skipped] #33501

roachtest: kv/gracefuldraining/nodes=3 failed [skipped] #33501

Comments

cockroach-teamcity commented Jan 4, 2019

cockroach-teamcity commented Jan 5, 2019

cockroach-teamcity commented Jan 6, 2019

tbg commented Jan 7, 2019

cockroach-teamcity commented Jan 8, 2019

cockroach-teamcity commented Jan 9, 2019

cockroach-teamcity commented Jan 9, 2019

cockroach-teamcity commented Jan 11, 2019

cockroach-teamcity commented Jan 12, 2019

cockroach-teamcity commented Jan 19, 2019

cockroach-teamcity commented Jan 21, 2019

cockroach-teamcity commented Jan 23, 2019

tbg commented Jan 23, 2019

cockroach-teamcity commented Jan 24, 2019

cockroach-teamcity commented Jan 25, 2019

cockroach-teamcity commented Jan 26, 2019

cockroach-teamcity commented Jan 27, 2019

cockroach-teamcity commented Jan 28, 2019

cockroach-teamcity commented Jan 29, 2019

cockroach-teamcity commented Jan 31, 2019

cockroach-teamcity commented May 14, 2019

cockroach-teamcity commented May 14, 2019

cockroach-teamcity commented May 15, 2019

cockroach-teamcity commented May 16, 2019

cockroach-teamcity commented May 17, 2019

cockroach-teamcity commented May 17, 2019

cockroach-teamcity commented May 18, 2019

cockroach-teamcity commented May 19, 2019

cockroach-teamcity commented May 21, 2019

cockroach-teamcity commented May 21, 2019

tbg commented May 21, 2019

ajwerner commented May 21, 2019

tbg commented May 21, 2019