Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: out-of-bounds error in rangeStatsOperator #87249

Closed
cockroach-teamcity opened this issue Sep 1, 2022 · 5 comments · Fixed by #87273
Closed

roachtest: out-of-bounds error in rangeStatsOperator #87249

cockroach-teamcity opened this issue Sep 1, 2022 · 5 comments · Fixed by #87273
Assignees
@DrewKimball
Labels
branch-master Failures and bugs on the master branch. branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-queries SQL Queries Team
Milestone
22.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Sep 1, 2022 •
edited by cockroach-jira-scripts
Loading

roachtest.sqlsmith/setup=seed/setting=no-mutations failed with artifacts on master @ b316a5ed5fe7253d113174d9d95ddebf1143b4e4:

test artifacts and logs in: /artifacts/sqlsmith/setup=seed/setting=no-mutations/run_1
	sqlsmith.go:264,sqlsmith.go:324,test_runner.go:897: error: pq: internal error: unexpected error from the vectorized engine: runtime error: index out of range [5] with length 5
		stmt:
		SELECT
			'2018-12-27 04:49:18.000437+00:00':::TIMESTAMPTZ AS col_283066,
			tab_115474.crdb_internal_mvcc_timestamp AS col_283067,
			crdb_internal.range_stats(tab_115474._bytes::BYTES)::JSONB AS col_283068,
			true AS col_283069,
			tab_115474.crdb_internal_mvcc_timestamp AS col_283070
		FROM
			defaultdb.public.seed@seed__int8__float8__date_idx AS tab_115474
		WHERE
			tab_115474._bool
		LIMIT
			5:::INT8;

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-19234
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Sep 1, 2022
@cockroach-teamcity cockroach-teamcity added this to the 22.2 milestone Sep 1, 2022
@blathers-crl blathers-crl bot added the T-sql-queries SQL Queries Team label Sep 1, 2022
@DrewKimball
Copy link
Collaborator

Stack trace:

E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +unexpected error from the vectorized engine: runtime error: index out of range [5] with length 5
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +(1) assertion failure
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +Wraps: (2) attached stack trace
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  -- stack trace:
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:89
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | runtime.gopanic
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	GOROOT/src/runtime/panic.go:838
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | runtime.goPanicIndex
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	GOROOT/src/runtime/panic.go:89
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*rangeStatsOperator).Next.func1.1
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/range_stats.go:122
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*rangeStatsOperator).Next.func1
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/range_stats.go:134
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colmem.(*Allocator).PerformOperation
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colmem/allocator.go:380
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*rangeStatsOperator).Next
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/range_stats.go:90
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/operator.go:138
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.constBoolOp.Next
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/const.eg.go:172
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase/simple_project.go:124
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*limitOp).Next
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/limit.go:47
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*panicInjector).Next
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/panic_injector.go:70
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:99
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:107
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colexecop.(*noopOperator).Next
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecop/operator.go:391
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*panicInjector).Next
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/panic_injector.go:70
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:99
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:107
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:252
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).next
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:256
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:299
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:320
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:695
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1611
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1334
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1537
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1173
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:686
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
E220901 05:46:39.475550 2657 sql/sqltelemetry/report.go:57 ⋮ [n1,client=35.243.230.84:49032,user=root] 1835 +  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:128

@DrewKimball
Copy link
Collaborator

I'm guessing the problem is that Fetcher.RangeStats uses the length of the keys slice to decide how large the response should be, but the keys slice may be smaller than the batch size because nulls are skipped.

@DrewKimball DrewKimball self-assigned this Sep 1, 2022
@DrewKimball DrewKimball changed the title roachtest: sqlsmith/setup=seed/setting=no-mutations failed roachtest: out-of-bounds error in rangeStatsOperator Sep 1, 2022
@DrewKimball
Copy link
Collaborator

Can also get a nil pointer dereference panic with this:

statement ok
CREATE TABLE t (a BYTES);
INSERT INTO t VALUES (NULL);
SELECT crdb_internal.range_stats(a) FROM t;

@ajwerner (since it seems like you've interacted with this code recently) should this be fixed, or is it expected that crdb_internal.range_stats will not be used by customers, and will not be given invalid/null inputs? If the latter, we could just skip this function during sqlsmith query generation.

@ajwerner
Copy link
Contributor

ajwerner commented Sep 1, 2022

I had hoped #86715 would have fixed all the issues. For now let's mark this as a release blocker

@ajwerner ajwerner added release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 labels Sep 1, 2022
@cucaroach
Copy link
Contributor

cucaroach commented Sep 1, 2022 •
edited
Loading

Another issue or dupe? #86930

I think its a dupe but feel free to re-open if not.

@cucaroach cucaroach mentioned this issue Sep 1, 2022
Closed
@yuzefovich yuzefovich mentioned this issue Sep 1, 2022
Merged
@craig craig bot closed this as completed in 25eac36 Sep 1, 2022
@mgartner mgartner moved this to Done in SQL Queries Jul 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DrewKimball DrewKimball

Labels
branch-master Failures and bugs on the master branch. branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-queries SQL Queries Team
Projects
SQL Queries
Archived in project
Milestone
22.2
Development

Successfully merging a pull request may close this issue.

colexec: fix range stats operator yuzefovich/cockroach
4 participants
@ajwerner @cockroach-teamcity @DrewKimball @cucaroach