sql: CockroachDB crashes when running a certain query through drivers {pgx, npgsql} #94356

dikshant · 2022-12-28T03:50:13Z

Describe the problem

This query:

SELECT (1,'foo'::text)::record, ARRAY[(1,'foo'::text)::record, (1,'foo'::text)::record];

runs fine through the CockroachDB CLI:

root@localhost:26257/defaultdb> SELECT (1,'foo'::text)::record, ARRAY[(1,'foo'::text)::record, (1,'foo'::text)::record];
  record  |         array
----------+------------------------
  (1,foo) | {"(1,foo)","(1,foo)"}
(1 row)


Time: 1ms total (execution 0ms / network 0ms)

To Reproduce

I ran that same query through pgx and npgsql. Both resulted in CockroachDB crashing. Originally I stumbled on this while running npgsql tests against CockroachDB, but when I tried it with pgx I was able to reproduce it there as well.

For reproducing via pgx see this snippet and run it locally:

https://go.dev/play/p/4wmQvk8d0AJ

For reproducing via npgsql follow these steps:

Set up CockroachDB 22.2.1. I tried docker image for 22.2.1 as well as the binary.
Download and install DotNet 7.0 from here.
Untar the repro.tar.gz attached below. cd into the repro directory and with CRDB running locally and execute dotnet run.
See error and stacktrace.

Expected behavior
The database should not crash, and I should get the same result I get via the cockroach SQL Shell.

Stacktrace
This stack trace is from the official docker container for 22.2.1, but it happens on master as of 21226cf as well.

certificate dir "certs" is successfully set up
starting node for the initialization process. This could take a couple of seconds...
*
* WARNING: ALL SECURITY CONTROLS HAVE BEEN DISABLED!
*
* This mode is intended for non-production testing only.
*
* In this mode:
* - Your cluster is open to any client that can access any of your IP addresses.
* - Intruders with access to your machine or network can observe client-server traffic.
* - Intruders can log in without password and read or write any data in the cluster.
* - Intruders can consume all your server's resources and cause unavailability.
*
*
* INFO: To start a secure server without mandating TLS for clients,
* consider --accept-sql-without-tls instead. For other options, see:
*
* - https://go.crdb.dev/issue-v/53404/v22.2
* - https://www.cockroachlabs.com/docs/v22.2/secure-a-cluster.html
*
*
* WARNING: Running a server without --sql-addr, with a combined RPC/SQL listener, is deprecated.
* This feature will be removed in the next version of CockroachDB.
*
init node successfully started
CockroachDB node starting at 2022-12-28 01:52:25.803797386 +0000 UTC m=+0.744080585 (took 0.4s)
build:               CCL v22.2.1 @ 2022/12/22 15:32:21 (go1.19.1)
webui:               http://127.0.0.1:8080
sql:                 postgresql://[email protected]:26257/defaultdb?sslmode=disable
sql (JDBC):          jdbc:postgresql://127.0.0.1:26257/defaultdb?sslmode=disable&user=root
RPC client flags:    /cockroach/cockroach <client cmd> --host=127.0.0.1:26257 --insecure
logs:                /cockroach/cockroach-data/logs
temp dir:            /cockroach/cockroach-data/cockroach-temp3179500688
external I/O path:   /cockroach/cockroach-data/extern
store[0]:            path=/cockroach/cockroach-data
storage engine:      pebble
clusterID:           335cb3e4-6b9a-4f3c-84d6-7401d5421921
status:              initialized new cluster
nodeID:              1
database "defaultdb" already exists
start running init files from /docker-entrypoint-initdb.d
/cockroach/cockroach.sh: ignoring /docker-entrypoint-initdb.d/*

end running init files from /docker-entrypoint-initdb.d
"${start_node_query[@]}"
*
* ERROR: Queued as error 2b83c5b01d0342f18b69d288c1a5e670
*
*
* ERROR: Queued as error a05f4f29f7894f988a509d03628f7ba1
*
*
* ERROR: a SQL panic has occurred while executing the following statement:
* SELECT (1, 'foo')::RECORD, ARRAY[(1, 'foo')::RECORD, (1, 'foo')::RECORD]
*
*
* ERROR: a panic has occurred!
* runtime error: index out of range [1] with length 1
* (1) attached stack trace
*   -- stack trace:
*   | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:830
*   | [...repeated from below...]
* Wraps: (2) while executing: SELECT (_, _)::RECORD, ARRAY[(_, _)::RECORD, (_, _)::RECORD]
* Wraps: (3) attached stack trace
*   -- stack trace:
*   | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:830
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:884
*   | runtime.goPanicIndex
*   | 	GOROOT/src/runtime/panic.go:113
*   | github.com/cockroachdb/cockroach/pkg/sql/pgwire.writeBinaryDatumNotNull
*   | 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/types.go:688
*   | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*writeBuffer).writeBinaryColumnarElement
*   | 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/types.go:818
*   | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).bufferBatch
*   | 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:1393
*   | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*commandResult).AddBatch
*   | 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/command_result.go:230
*   | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLReceiver).PushBatch
*   | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1357
*   | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
*   | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:311
*   | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
*   | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:320
*   | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
*   | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:719
*   | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
*   | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1686
*   | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
*   | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1409
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1571
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1186
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:689
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:131
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2412
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:130
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:220
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2006
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2008
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1854
*   | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:832
*   | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
*   | 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:728
*   | runtime.goexit
*   | 	src/runtime/asm_arm64.s:1172
* Wraps: (4) runtime error: index out of range [1] with length 1
* Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *withstack.withStack (4) runtime.boundsError
*
*
* ERROR: Queued as error 57bb8a7d1d6e4e828cbe643639a6b1d0
*

The stacktrace on master points to this:
https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/pgwire/types.go#L705

Environment:

CockroachDB CCL v22.2.1 (aarch64-apple-darwin21.2, built 2022/12/22 15:20:25, go1.19.1) as well as master on 21226cf
Server OS: macOS Darwin ARM64
npgsql 7.0, pgx

Jira issue: CRDB-22845

Epic CRDB-17785

The text was updated successfully, but these errors were encountered:

dikshant · 2022-12-28T16:24:47Z

Also crashes on v22.2.0

certificate dir "certs" is successfully set up
starting node for the initialization process. This could take a couple seconds...
*
* WARNING: ALL SECURITY CONTROLS HAVE BEEN DISABLED!
*
* This mode is intended for non-production testing only.
*
* In this mode:
* - Your cluster is open to any client that can access any of your IP addresses.
* - Intruders with access to your machine or network can observe client-server traffic.
* - Intruders can log in without password and read or write any data in the cluster.
* - Intruders can consume all your server's resources and cause unavailability.
*
*
* INFO: To start a secure server without mandating TLS for clients,
* consider --accept-sql-without-tls instead. For other options, see:
*
* - https://go.crdb.dev/issue-v/53404/v22.2
* - https://www.cockroachlabs.com/docs/v22.2/secure-a-cluster.html
*
*
* WARNING: Running a server without --sql-addr, with a combined RPC/SQL listener, is deprecated.
* This feature will be removed in the next version of CockroachDB.
*
init node successfully started
CockroachDB node starting at 2022-12-28 16:23:12.161094211 +0000 UTC m=+0.909055293 (took 0.4s)
build:               CCL v22.2.0 @ 2022/12/05 17:11:12 (go1.19.1)
webui:               http://127.0.0.1:8080
sql:                 postgresql://[email protected]:26257/defaultdb?sslmode=disable
sql (JDBC):          jdbc:postgresql://127.0.0.1:26257/defaultdb?sslmode=disable&user=root
RPC client flags:    /cockroach/cockroach <client cmd> --host=127.0.0.1:26257 --insecure
logs:                /cockroach/cockroach-data/logs
temp dir:            /cockroach/cockroach-data/cockroach-temp874425105
external I/O path:   /cockroach/cockroach-data/extern
store[0]:            path=/cockroach/cockroach-data
storage engine:      pebble
clusterID:           3ca10728-6030-4ac1-899a-3b80ece9fe21
status:              initialized new cluster
nodeID:              1
database "defaultdb" already exists
start running init files from /docker-entrypoint-initdb.d
/cockroach/cockroach.sh: ignoring /docker-entrypoint-initdb.d/*

end running init files from /docker-entrypoint-initdb.d
"${start_node_query[@]}"
*
* ERROR: a SQL panic has occurred while executing the following statement:
* SELECT (1, 'foo')::RECORD, ARRAY[(1, 'foo')::RECORD, (1, 'foo')::RECORD]
*
*
* ERROR: a panic has occurred!
* runtime error: index out of range [1] with length 1
* (1) attached stack trace
*   -- stack trace:
*   | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:826
*   | [...repeated from below...]
* Wraps: (2) while executing: SELECT (_, _)::RECORD, ARRAY[(_, _)::RECORD, (_, _)::RECORD]
* Wraps: (3) attached stack trace
*   -- stack trace:
*   | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:826
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:884
*   | runtime.goPanicIndex
*   | 	GOROOT/src/runtime/panic.go:113
*   | github.com/cockroachdb/cockroach/pkg/sql/pgwire.writeBinaryDatumNotNull
*   | 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/types.go:683
*   | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*writeBuffer).writeBinaryColumnarElement
*   | 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/types.go:813
*   | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).bufferBatch
*   | 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:1393
*   | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*commandResult).AddBatch
*   | 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/command_result.go:230
*   | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLReceiver).PushBatch
*   | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1282
*   | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
*   | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:311
*   | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
*   | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:320
*   | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
*   | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:695
*   | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
*   | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1611
*   | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
*   | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1334
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1541
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1177
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:687
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:129
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2382
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:128
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:218
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1998
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2000
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1846
*   | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
*   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:828
*   | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
*   | 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:728
*   | runtime.goexit
*   | 	src/runtime/asm_arm64.s:1172
* Wraps: (4) runtime error: index out of range [1] with length 1
* Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *withstack.withStack (4) runtime.boundsError
*
*
* ERROR: Queued as error 41c0fbffc19544c19ce664a020c07aee
*

fixes #94356

dikshant added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Dec 28, 2022

dikshant changed the title ~~sql: CockroachDB crashes when running a certain query through pgwire {pgx, npgsql}~~ sql: CockroachDB crashes when running a certain query through drivers {pgx, npgsql} Dec 28, 2022

dikshant mentioned this issue Dec 28, 2022

sql: update support for npgsql #92623

Closed

yuzefovich mentioned this issue Dec 28, 2022

sql: v22.2.1: index out of range when writing a tuple into pgwire message #94354

Closed

rafiss mentioned this issue Dec 29, 2022

pgwire: support binary encoding of generic tuples #94405

Merged

craig bot closed this as completed in 606313d Dec 29, 2022

This was referenced Dec 29, 2022

release-22.2: pgwire: support binary encoding of generic tuples #94419

Merged

release-22.1: pgwire: support binary encoding of generic tuples #94420

Merged

rafiss added a commit that referenced this issue Dec 29, 2022

Merge pull request #94419 from rafiss/backport22.2-94405

46e1556

fixes #94356

rafiss added a commit that referenced this issue Dec 29, 2022

Merge pull request #94420 from rafiss/backport22.1-94405

1f1f576

fixes #94356

yuzefovich mentioned this issue Jan 10, 2023

sentry: conn_executor.go:826: runtime error: index out of range [1] with length 1 (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1 | github.... #94375

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sql: CockroachDB crashes when running a certain query through drivers {pgx, npgsql} #94356

sql: CockroachDB crashes when running a certain query through drivers {pgx, npgsql} #94356

dikshant commented Dec 28, 2022 •

edited by exalate-issue-sync bot

Loading

dikshant commented Dec 28, 2022

sql: CockroachDB crashes when running a certain query through drivers {pgx, npgsql} #94356

sql: CockroachDB crashes when running a certain query through drivers {pgx, npgsql} #94356

Comments

dikshant commented Dec 28, 2022 • edited by exalate-issue-sync bot Loading

dikshant commented Dec 28, 2022

dikshant commented Dec 28, 2022 •

edited by exalate-issue-sync bot

Loading