Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: sqlsmith failed #37549

Closed
cockroach-teamcity opened this issue May 16, 2019 · 9 comments · Fixed by #37710
Closed

roachtest: sqlsmith failed #37549

cockroach-teamcity opened this issue May 16, 2019 · 9 comments · Fixed by #37710
Assignees
@maddyblue
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone
19.2

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/e4806ba5d174c7aeb88ebbefb64d29747803cefa

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=sqlsmith PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1292134&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,sqlsmith.go:101,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1292134-sqlsmith:1 -- cd /mnt/data1/sqlsmith/ && ./sqlsmith --target="postgres://root@localhost:26257/tpcc?sslmode=disable" --max-queries=1000 --verbose > --dump-all-queries returned:
		stderr:
		
		stdout:
		lif: 159/205 (bad/ok)
		  truth_value: 162/272 (bad/ok)
		  null_predicate: 201/374 (bad/ok)
		  exists_predicate: 158/243 (bad/ok)
		  bool_term: 215/363 (bad/ok)
		  comparison_op: 259/495 (bad/ok)
		  window_function: 41/65 (bad/ok)
		  table_or_query_name: 233/511 (bad/ok)
		  target_table: 15/0 (bad/ok)
		  table_subquery: 161/350 (bad/ok)
		  simple_join_cond: 69/91 (bad/ok)
		  expr_join_cond: 131/239 (bad/ok)
		  joined_table: 133/253 (bad/ok)
		  from_clause: 228/511 (bad/ok)
		  select_list: 243/519 (bad/ok)
		  query_spec: 216/475 (bad/ok)
		  select_for_update: 57/170 (bad/ok)
		  delete_returning: 10/11 (bad/ok)
		  insert_stmt: 18/6 (bad/ok)
		  set_list: 38/1 (bad/ok)
		  upsert_stmt: 18/0 (bad/ok)
		  when_clause: 5/0 (bad/ok)
		  when_clause_update: 8/0 (bad/ok)
		  when_clause_insert: 8/0 (bad/ok)
		  merge_stmt: 15/0 (bad/ok)
		  update_returning: 12/1 (bad/ok)
		  common_table_expression: 49/40 (bad/ok)
		
		..e.e.e.......e..eeee.e....e........ee..ee..e.e.e........e......ee.......e.e.ee.
		e.....e..e.eee.e..eeee....e...ee.e..e....e......CCCCCCCCC: signal: killed
	cluster.go:1833,sqlsmith.go:106,sqlsmith.go:114,test.go:1251: unexpected node event: 1: dead
	cluster.go:1038,context.go:89,cluster.go:1027,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1833,sqlsmith.go:106,sqlsmith.go:114,test.go:1251: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1292134-sqlsmith --oneshot --ignore-empty-nodes: exit status 1 1: dead
		Error:  1: dead
@cockroach-teamcity cockroach-teamcity added this to the 19.2 milestone May 16, 2019
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels May 16, 2019
@nvanbenschoten
Copy link
Member 

panic: panic while executing 1 statements: SELECT (SELECT _ FROM _._ LIMIT _ OFFSET _) AS _ FROM _._ AS _ WHERE crdb_internal.force_panic(CAST(CAST(NULLIF(crdb_internal.node_executable_version(), current_schema()) AS STRING) AS STRING)) >= (SELECT _ FROM _._ LIMIT _ OFFSET _) LIMIT _; caused by 19.1-2

@mjibson sqlsmith is cheating. It learned about crdb_internal.force_panic.

@jordanlewis
Copy link
Member

This is the C sqlsmith. We should delete this roachtest, as it's no longer developed.

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/7b2651400b2003d0a381cba9dbfc0b7bc0dfee00

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=sqlsmith PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1293898&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,sqlsmith.go:100,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1293898-sqlsmith:1 -- ./workload fixtures load tpcc --warehouses=10 {pgurl:1} returned:
		stderr:
		
		stdout:
		t-status reply 0
		debug3: receive packet: type 96
		debug2: channel 0: rcvd eof
		debug2: channel 0: output open -> drain
		debug2: channel 0: obuf empty
		debug2: channel 0: close_write
		debug2: channel 0: output drain -> closed
		debug3: receive packet: type 97
		debug2: channel 0: rcvd close
		debug3: channel 0: will not send data after close
		debug2: channel 0: almost dead
		debug2: channel 0: gc: notify user
		debug2: channel 0: gc: user detached
		debug2: channel 0: send close
		debug3: send packet: type 97
		debug2: channel 0: is dead
		debug2: channel 0: garbage collecting
		debug1: channel 0: free: client-session, nchannels 1
		debug3: channel 0: status: The following connections are open:
		  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
		
		debug3: send packet: type 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug3: fd 2 is not O_NONBLOCK
		Transferred: sent 3164, received 4600 bytes, in 64.0 seconds
		Bytes per second: sent 49.5, received 71.9
		debug1: Exit status 1
		: exit status 1
		: exit status 1
	cluster.go:1833,sqlsmith.go:106,sqlsmith.go:114,test.go:1251: Goexit() was called

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/923a3b2a6f4a6492883141092280d1041de1381a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=sqlsmith PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1295056&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,sqlsmith.go:100,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1295056-sqlsmith:1 -- ./workload fixtures load tpcc --warehouses=10 {pgurl:1} returned:
		stderr:
		
		stdout:
		t-status reply 0
		debug3: receive packet: type 96
		debug2: channel 0: rcvd eof
		debug2: channel 0: output open -> drain
		debug2: channel 0: obuf empty
		debug2: channel 0: close_write
		debug2: channel 0: output drain -> closed
		debug3: receive packet: type 97
		debug2: channel 0: rcvd close
		debug3: channel 0: will not send data after close
		debug2: channel 0: almost dead
		debug2: channel 0: gc: notify user
		debug2: channel 0: gc: user detached
		debug2: channel 0: send close
		debug3: send packet: type 97
		debug2: channel 0: is dead
		debug2: channel 0: garbage collecting
		debug1: channel 0: free: client-session, nchannels 1
		debug3: channel 0: status: The following connections are open:
		  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
		
		debug3: send packet: type 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug3: fd 2 is not O_NONBLOCK
		Transferred: sent 3164, received 4600 bytes, in 57.0 seconds
		Bytes per second: sent 55.5, received 80.7
		debug1: Exit status 1
		: exit status 1
		: exit status 1
	cluster.go:1833,sqlsmith.go:106,sqlsmith.go:114,test.go:1251: Goexit() was called

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/cab299a0ef983f8b4ffe5d724e44587d9665d3a3

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=sqlsmith PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1295811&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,sqlsmith.go:100,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1295811-sqlsmith:1 -- ./workload fixtures load tpcc --warehouses=10 {pgurl:1} returned:
		stderr:
		
		stdout:
		t-status reply 0
		debug3: receive packet: type 96
		debug2: channel 0: rcvd eof
		debug2: channel 0: output open -> drain
		debug2: channel 0: obuf empty
		debug2: channel 0: close_write
		debug2: channel 0: output drain -> closed
		debug3: receive packet: type 97
		debug2: channel 0: rcvd close
		debug3: channel 0: will not send data after close
		debug2: channel 0: almost dead
		debug2: channel 0: gc: notify user
		debug2: channel 0: gc: user detached
		debug2: channel 0: send close
		debug3: send packet: type 97
		debug2: channel 0: is dead
		debug2: channel 0: garbage collecting
		debug1: channel 0: free: client-session, nchannels 1
		debug3: channel 0: status: The following connections are open:
		  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
		
		debug3: send packet: type 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug3: fd 2 is not O_NONBLOCK
		Transferred: sent 3164, received 4600 bytes, in 56.4 seconds
		Bytes per second: sent 56.1, received 81.6
		debug1: Exit status 1
		: exit status 1
		: exit status 1
	cluster.go:1833,sqlsmith.go:106,sqlsmith.go:114,test.go:1251: Goexit() was called

@nvanbenschoten nvanbenschoten mentioned this issue May 20, 2019
Closed
@nvanbenschoten
Copy link
Member

The failures over the past 3 days are because of #37590.

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/58c567a325056033b326cb9c4ed9ba490e8956da

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=sqlsmith PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1296592&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,sqlsmith.go:100,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1296592-sqlsmith:1 -- ./workload fixtures load tpcc --warehouses=10 {pgurl:1} returned:
		stderr:
		
		stdout:
		t open -> drain
		debug2: channel 0: obuf empty
		debug2: channel 0: close_write
		debug2: channel 0: output drain -> closed
		debug3: receive packet: type 98
		debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
		debug3: receive packet: type 97
		debug2: channel 0: rcvd close
		debug3: channel 0: will not send data after close
		debug2: channel 0: almost dead
		debug2: channel 0: gc: notify user
		debug2: channel 0: gc: user detached
		debug2: channel 0: send close
		debug3: send packet: type 97
		debug2: channel 0: is dead
		debug2: channel 0: garbage collecting
		debug1: channel 0: free: client-session, nchannels 1
		debug3: channel 0: status: The following connections are open:
		  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
		
		debug3: send packet: type 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug3: fd 2 is not O_NONBLOCK
		Transferred: sent 3164, received 4592 bytes, in 58.7 seconds
		Bytes per second: sent 53.9, received 78.2
		debug1: Exit status 1
		: exit status 1
		: exit status 1
	cluster.go:1833,sqlsmith.go:106,sqlsmith.go:114,test.go:1251: Goexit() was called

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/9671342fead0509bec0913bae4ae1f244660788e

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=sqlsmith PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1298500&tab=buildLog

The test failed on branch=release-19.1, cloud=gce:
	cluster.go:1474,sqlsmith.go:100,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1298500-sqlsmith:1 -- ./workload fixtures load tpcc --warehouses=10 {pgurl:1} returned:
		stderr:
		
		stdout:
		I190521 12:47:44.335289 1 ccl/workloadccl/cliccl/fixtures.go:300  starting load of 9 tables
		Error: restoring fixture: pq: table "csv.stock" does not exist
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_35.196.89.193_2019-05-21T12:47:42Z: exit status 1
		: exit status 1
	cluster.go:1833,sqlsmith.go:106,sqlsmith.go:114,test.go:1251: Goexit() was called

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/5542939c07947546136d5ae895b7349751534595

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=sqlsmith PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1298482&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,sqlsmith.go:100,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1298482-sqlsmith:1 -- ./workload fixtures load tpcc --warehouses=10 {pgurl:1} returned:
		stderr:
		
		stdout:
		I190521 14:58:21.610424 1 ccl/workloadccl/cliccl/fixtures.go:300  starting load of 9 tables
		Error: restoring fixture: pq: table "csv.district" does not exist
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_34.73.229.13_2019-05-21T14:58:20Z: exit status 1
		: exit status 1
	cluster.go:1833,sqlsmith.go:106,sqlsmith.go:114,test.go:1251: Goexit() was called

@maddyblue maddyblue mentioned this issue May 21, 2019
Merged
craig bot pushed a commit that referenced this issue May 22, 2019
@maddyblue
Merge #37710
7009f87 
37710: cmd/roachtest: remove C sqlsmith test r=mjibson a=mjibson

Fixes #37549

Release note: None

Co-authored-by: Matt Jibson <[email protected]>
@craig craig bot closed this as completed in #37710 May 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@maddyblue maddyblue

Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Milestone
19.2
Development

Successfully merging a pull request may close this issue.

cmd/roachtest: remove C sqlsmith test
4 participants
@maddyblue @jordanlewis @nvanbenschoten @cockroach-teamcity