Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: tpccbench/nodes=3/cpu=16 failed #37393

Closed
cockroach-teamcity opened this issue May 8, 2019 · 1 comment · Fixed by #37401
Closed

roachtest: tpccbench/nodes=3/cpu=16 failed #37393

cockroach-teamcity opened this issue May 8, 2019 · 1 comment · Fixed by #37401
Assignees
@nvanbenschoten
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone
19.2

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/8abb47a1c9795c1463183bc44e776b054bece682

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1279687&tab=buildLog

The test failed on branch=master, cloud=aws:
	cluster.go:1833,tpcc.go:853,tpcc.go:554,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1279687-tpccbench-nodes-3-cpu-16:4 -- ./workload fixtures import tpcc --warehouses=2500 --split --scatter --checks=false {pgurl:1} returned:
		stderr:
		
		stdout:
		0x1 fp=0xc0009577e0 sp=0xc0009577d8 pc=0x466f61
		created by runtime.(*timersBucket).addtimerLocked
			/usr/local/go/src/runtime/time.go:170 +0x114
		
		goroutine 899 [timer goroutine (idle)]:
		runtime.gopark(0x15ed1a8, 0x28cb0c0, 0x3906e301414, 0x1)
			/usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000a3e728 sp=0xc000a3e708 pc=0x438f8b
		runtime.goparkunlock(0x28cb0c0, 0xc0009a1414, 0x1)
			/usr/local/go/src/runtime/proc.go:308 +0x53 fp=0xc000a3e758 sp=0xc000a3e728 pc=0x439033
		runtime.timerproc(0x28cb0c0)
			/usr/local/go/src/runtime/time.go:280 +0x288 fp=0xc000a3e7d8 sp=0xc000a3e758 pc=0x456168
		runtime.goexit()
			/usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000a3e7e0 sp=0xc000a3e7d8 pc=0x466f61
		created by runtime.(*timersBucket).addtimerLocked
			/usr/local/go/src/runtime/time.go:170 +0x114
		bash: line 1:  4273 Aborted                 (core dumped) bash -c "./workload fixtures import tpcc --warehouses=2500 --split --scatter --checks=false 'postgres://[email protected]:26257?sslmode=disable'"
		Error:  exit status 134
		: exit status 1
@cockroach-teamcity cockroach-teamcity added this to the 19.2 milestone May 8, 2019
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels May 8, 2019
@danhhz danhhz mentioned this issue May 8, 2019
Merged
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/8abb47a1c9795c1463183bc44e776b054bece682

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1279683&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1833,tpcc.go:853,tpcc.go:554,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1279683-tpccbench-nodes-3-cpu-16:4 -- ./workload fixtures load tpcc --warehouses=2000 --split --scatter --checks=false {pgurl:1} returned:
		stderr:
		
		stdout:
		3 +0x1 fp=0xc000a64fe0 sp=0xc000a64fd8 pc=0x466f61
		created by runtime.(*timersBucket).addtimerLocked
			/usr/local/go/src/runtime/time.go:170 +0x114
		
		goroutine 43 [timer goroutine (idle)]:
		runtime.gopark(0x15ed1a8, 0x28caf40, 0x437ac7b1414, 0x1)
			/usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000bdcf28 sp=0xc000bdcf08 pc=0x438f8b
		runtime.goparkunlock(0x28caf40, 0xc000671414, 0x1)
			/usr/local/go/src/runtime/proc.go:308 +0x53 fp=0xc000bdcf58 sp=0xc000bdcf28 pc=0x439033
		runtime.timerproc(0x28caf40)
			/usr/local/go/src/runtime/time.go:280 +0x288 fp=0xc000bdcfd8 sp=0xc000bdcf58 pc=0x456168
		runtime.goexit()
			/usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000bdcfe0 sp=0xc000bdcfd8 pc=0x466f61
		created by runtime.(*timersBucket).addtimerLocked
			/usr/local/go/src/runtime/time.go:170 +0x114
		bash: line 1:  4068 Aborted                 (core dumped) bash -c "./workload fixtures load tpcc --warehouses=2000 --split --scatter --checks=false 'postgres://[email protected]:26257?sslmode=disable'"
		Error:  exit status 134
		: exit status 1

danhhz added a commit to danhhz/cockroach that referenced this issue May 8, 2019
@danhhz
workload: fix --splits regression introduced in cockroachdb#35349
b0607c0 
workload's Table schemas are SQL schemas, but cockroachdb#35349 switched the
initial data to be returned as a coldata.Batch, which has a more limited
set of types. (Or, in the case of simple workloads that return a
[]interface{}, it's roundtripped through coldata.Batch by the `Tuples`
helper.) Notably, this means a SQL STRING column is represented the same
as a BYTES column (ditto UUID, etc).

This caused a regression in splits, which received some []byte data for
a column tried to hand it to SPLIT as a SQL BYTES datum. This didn't
work for the UUID column in tpcc's history table nor the VARCHAR in
ycsb's usertable. Happily, a STRING works for both of these. It also
seems to work for BYTES columns, so it seems like the ambiguity is fine
in this case. When/if someone wants to add a workload that splits a
BYTES primary key column containing non-utf8 data, we'll may need to
revisit.

A more principled fix would be to get the fidelity back by parsing the
SQL schema, which in fact we do in `importccl.makeDatumFromColOffset`.
However, at the moment, this hack works and avoids the complexity and
the undesirable pkg/sql/parser dep.

Closes cockroachdb#37383
Closes cockroachdb#37382
Closes cockroachdb#37381
Closes cockroachdb#37380
Closes cockroachdb#37379
Closes cockroachdb#37378
Closes cockroachdb#37377
Closes cockroachdb#37393

Release note: None
craig bot pushed a commit that referenced this issue May 8, 2019
@danhhz
Merge #37401
61851c7 
37401: workload: fix --splits regression introduced in #35349 r=tbg a=danhhz

workload's Table schemas are SQL schemas, but #35349 switched the
initial data to be returned as a coldata.Batch, which has a more limited
set of types. (Or, in the case of simple workloads that return a
[]interface{}, it's roundtripped through coldata.Batch by the `Tuples`
helper.) Notably, this means a SQL STRING column is represented the same
as a BYTES column (ditto UUID, etc).

This caused a regression in splits, which received some []byte data for
a column tried to hand it to SPLIT as a SQL BYTES datum. This didn't
work for the UUID column in tpcc's history table nor the VARCHAR in
ycsb's usertable. Happily, a STRING works for both of these. It also
seems to work for BYTES columns, so it seems like the ambiguity is fine
in this case. When/if someone wants to add a workload that splits a
BYTES primary key column containing non-utf8 data, we'll may need to
revisit.

A more principled fix would be to get the fidelity back by parsing the
SQL schema, which in fact we do in `importccl.makeDatumFromColOffset`.
However, at the moment, this hack works and avoids the complexity and
the undesirable pkg/sql/parser dep.

Closes #37383
Closes #37382
Closes #37381
Closes #37380
Closes #37379
Closes #37378
Closes #37377
Closes #37393

Release note: None

Co-authored-by: Daniel Harrison <[email protected]>
@craig craig bot closed this as completed in #37401 May 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nvanbenschoten nvanbenschoten

Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Milestone
19.2
Development

Successfully merging a pull request may close this issue.

workload: fix --splits regression introduced in #35349 danhhz/cockroach
2 participants
@nvanbenschoten @cockroach-teamcity