Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sentry: conn_executor.go:890: panic: × (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1 | github.com/cockroachdb/cockroach/pkg/sql/conn_exec... #105056

Closed
cockroach-teamcity opened this issue Jun 16, 2023 · 1 comment
Closed

sentry: conn_executor.go:890: panic: × (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1 | github.com/cockroachdb/cockroach/pkg/sql/conn_exec... #105056

cockroach-teamcity opened this issue Jun 16, 2023 · 1 comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jun 16, 2023 •
edited by cockroach-jira-scripts
Loading

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://cockroach-labs.sentry.io/issues/4255862934/?referrer=webhooks_plugin

Panic message:

conn_executor.go:890: panic: ×
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:890
| [...repeated from below...]
Wraps: (2) while executing: SELECT crdb_internal.pretty_key(_, _) AS , crdb_internal.pretty_key(, ) AS , , _ FROM _ WHERE ('' <= ) AND ( <= '')
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:890
| runtime.gopanic
| GOROOT/src/runtime/panic.go:884
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:58
| runtime.gopanic
| GOROOT/src/runtime/panic.go:884
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:58
| runtime.gopanic
| GOROOT/src/runtime/panic.go:884
| github.com/cockroachdb/cockroach/pkg/roachpb.assertValid
| github.com/cockroachdb/cockroach/pkg/roachpb/pkg/roachpb/tenant.go:99
| github.com/cockroachdb/cockroach/pkg/roachpb.MustMakeTenantID
| github.com/cockroachdb/cockroach/pkg/roachpb/pkg/roachpb/tenant.go:52
| github.com/cockroachdb/cockroach/pkg/keys.DecodeTenantPrefix
| github.com/cockroachdb/cockroach/pkg/keys/sql.go:43
| github.com/cockroachdb/cockroach/pkg/keys.formatTenantKey
| github.com/cockroachdb/cockroach/pkg/keys/printer.go:652
| github.com/cockroachdb/cockroach/pkg/keys.safeFormatInternal.func1
| github.com/cockroachdb/cockroach/pkg/keys/printer.go:693
| github.com/cockroachdb/cockroach/pkg/keys.safeFormatInternal
| github.com/cockroachdb/cockroach/pkg/keys/printer.go:749
| github.com/cockroachdb/cockroach/pkg/keys.PrettyPrint
| github.com/cockroachdb/cockroach/pkg/keys/printer.go:599
| github.com/cockroachdb/cockroach/pkg/roachpb.Key.StringWithDirs
| github.com/cockroachdb/cockroach/pkg/roachpb/pkg/roachpb/data.go:244
| github.com/cockroachdb/cockroach/pkg/sql/catalog/catalogkeys.PrettyKey
| github.com/cockroachdb/cockroach/pkg/sql/catalog/catalogkeys/keys.go:171
| github.com/cockroachdb/cockroach/pkg/sql/sem/builtins.glob..func258
| github.com/cockroachdb/cockroach/pkg/sql/sem/builtins/builtins.go:5518
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*defaultBuiltinFuncOperator).Next.func1
| github.com/cockroachdb/cockroach/pkg/sql/colexec/builtin_funcs.go:77
| github.com/cockroachdb/cockroach/pkg/sql/colmem.(*Allocator).PerformOperation
| github.com/cockroachdb/cockroach/pkg/sql/colmem/allocator.go:460
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*defaultBuiltinFuncOperator).Next
| github.com/cockroachdb/cockroach/pkg/sql/colexec/builtin_funcs.go:54
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase/simple_project.go:124
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:118
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
| github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:126
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter
| github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:251
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).next
| github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:255
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
| github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:293
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
| github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:326
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:902
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1913
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1645
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1648
Wraps: (4) panic: ×
Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *withstack.withStack (4) *errutil.leafError
-- report composition:
*errutil.leafError: panic: ×
conn_executor.go:890: *withstack.withStack (top exception)
*safedetails.withSafeDetails: while executing: SELECT crdb_internal.pretty_key(, _) AS , crdb_internal.pretty_key(, _) AS , , _ FROM _ WHERE ('' <= ) AND ( <= '')
conn_executor.go:890: *withstack.withStack (1)
(check the extra data payloads)

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/conn_executor.go

Lines 889 to 891 in ef8d5da

r := recover()
h.ex.closeWrapper(ctx, r)
}(ctx, h)
in pkg/sql.(*Server).ServeConn.func1
GOROOT/src/runtime/panic.go#L883-L885 in runtime.gopanic

cockroach/pkg/sql/colexecerror/error.go

Lines 57 to 59 in ef8d5da

if !shouldCatchPanic(panicEmittedFrom) {
panic(panicObj)
}
in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
GOROOT/src/runtime/panic.go#L883-L885 in runtime.gopanic

cockroach/pkg/sql/colexecerror/error.go

Lines 57 to 59 in ef8d5da

if !shouldCatchPanic(panicEmittedFrom) {
panic(panicObj)
}
in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
GOROOT/src/runtime/panic.go#L883-L885 in runtime.gopanic
https://github.com/cockroachdb/cockroach/blob/ef8d5da59d22e333087d51c8b34bfaba39b5b7a4/pkg/roachpb/pkg/roachpb/tenant.go#L98-L100 in pkg/roachpb.assertValid
https://github.com/cockroachdb/cockroach/blob/ef8d5da59d22e333087d51c8b34bfaba39b5b7a4/pkg/roachpb/pkg/roachpb/tenant.go#L51-L53 in pkg/roachpb.MustMakeTenantID

cockroach/pkg/keys/sql.go

Lines 42 to 44 in ef8d5da

}
return rem, roachpb.MustMakeTenantID(tenID), nil
}
in pkg/keys.DecodeTenantPrefix

cockroach/pkg/keys/printer.go

Lines 651 to 653 in ef8d5da

) {
key, tID, err := DecodeTenantPrefix(key)
if err != nil {
in pkg/keys.formatTenantKey

cockroach/pkg/keys/printer.go

Lines 692 to 694 in ef8d5da

b.Print(e.Name)
e.sfFunc(&buf, valDirs, key, quoteRawKeys)
b.Print(buf.RedactableString())
in pkg/keys.safeFormatInternal.func1

cockroach/pkg/keys/printer.go

Lines 748 to 750 in ef8d5da

}
helper(&b, key)
return b.RedactableString()
in pkg/keys.safeFormatInternal

cockroach/pkg/keys/printer.go

Lines 598 to 600 in ef8d5da

func PrettyPrint(valDirs []encoding.Direction, key roachpb.Key) string {
return safeFormatInternal(valDirs, key, QuoteRaw).StripMarkers()
}
in pkg/keys.PrettyPrint
https://github.com/cockroachdb/cockroach/blob/ef8d5da59d22e333087d51c8b34bfaba39b5b7a4/pkg/roachpb/pkg/roachpb/data.go#L243-L245 in pkg/roachpb.Key.StringWithDirs

cockroach/pkg/sql/catalog/catalogkeys/keys.go

Lines 170 to 172 in ef8d5da

var buf redact.StringBuilder
key.StringWithDirs(&buf, valDirs)
p := buf.String()
in pkg/sql/catalog/catalogkeys.PrettyKey

cockroach/pkg/sql/sem/builtins/builtins.go

Lines 5517 to 5519 in ef8d5da

Fn: func(_ context.Context, _ *eval.Context, args tree.Datums) (tree.Datum, error) {
return tree.NewDString(catalogkeys.PrettyKey(
nil, /* valDirs */
in pkg/sql/sem/builtins.glob..func258

cockroach/pkg/sql/colexec/builtin_funcs.go

Lines 76 to 78 in ef8d5da

res, err = b.funcExpr.ResolvedOverload().
Fn.(eval.FnOverload)(b.Ctx, b.evalCtx, b.row)
if err != nil {
in pkg/sql/colexec.(*defaultBuiltinFuncOperator).Next.func1

cockroach/pkg/sql/colmem/allocator.go

Lines 459 to 461 in ef8d5da

// caused by this approach is ok.
operation()
after := getVecsMemoryFootprint(destVecs)
in pkg/sql/colmem.(*Allocator).PerformOperation

cockroach/pkg/sql/colexec/builtin_funcs.go

Lines 53 to 55 in ef8d5da

output := batch.ColVec(b.outputIdx)
b.allocator.PerformOperation(
[]coldata.Vec{output},
in pkg/sql/colexec.(*defaultBuiltinFuncOperator).Next

cockroach/pkg/sql/colexec/colexecbase/simple_project.go

Lines 123 to 125 in ef8d5da

func (d *simpleProjectOp) Next() coldata.Batch {
batch := d.Input.Next()
if batch.Length() == 0 {
in pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next

cockroach/pkg/sql/colflow/stats.go

Lines 117 to 119 in ef8d5da

func (bic *batchInfoCollector) next() {
bic.batch = bic.Input.Next()
}
in pkg/sql/colflow.(*batchInfoCollector).next

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in ef8d5da

}()
operation()
return retErr
in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colflow/stats.go

Lines 125 to 127 in ef8d5da

// execution time (e.g. in the statement bundle).
err := colexecerror.CatchVectorizedRuntimeError(bic.next)
bic.stopwatch.Stop()
in pkg/sql/colflow.(*batchInfoCollector).Next

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 250 to 252 in ef8d5da

func (f *BatchFlowCoordinator) nextAdapter() {
f.batch = f.input.Root.Next()
}
in pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in ef8d5da

}()
operation()
return retErr
in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 254 to 256 in ef8d5da

func (f *BatchFlowCoordinator) next() error {
return colexecerror.CatchVectorizedRuntimeError(f.nextAdapter)
}
in pkg/sql/colflow.(*BatchFlowCoordinator).next

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 292 to 294 in ef8d5da

for status == execinfra.NeedMoreRows {
err := f.next()
if err != nil {
in pkg/sql/colflow.(*BatchFlowCoordinator).Run

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 325 to 327 in ef8d5da

log.VEvent(ctx, 1, "running the batch flow coordinator in the flow's goroutine")
f.batchFlowCoordinator.Run(ctx)
}
in pkg/sql/colflow.(*vectorizedFlow).Run

cockroach/pkg/sql/distsql_running.go

Lines 901 to 903 in ef8d5da

noWait := planCtx.getPortalPauseInfo() != nil
flow.Run(ctx, noWait)
}
in pkg/sql.(*DistSQLPlanner).Run

cockroach/pkg/sql/distsql_running.go

Lines 1912 to 1914 in ef8d5da

recv.expectedRowsRead = int64(physPlan.TotalEstimatedScannedRows)
dsp.Run(ctx, planCtx, txn, physPlan, recv, evalCtx, finishedSetupFn)
}
in pkg/sql.(*DistSQLPlanner).PlanAndRun

cockroach/pkg/sql/distsql_running.go

Lines 1644 to 1646 in ef8d5da

defer cleanup()
dsp.PlanAndRun(
ctx, evalCtx, planCtx, planner.txn, planner.curPlan.main, recv, finishedSetupFn,
in pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3

cockroach/pkg/sql/distsql_running.go

Lines 1647 to 1649 in ef8d5da

)
}()
in pkg/sql.(*DistSQLPlanner).PlanAndRunAll

pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn.func1 at line 890
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 58
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 58
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
pkg/roachpb/pkg/roachpb/tenant.go in pkg/roachpb.assertValid at line 99
pkg/roachpb/pkg/roachpb/tenant.go in pkg/roachpb.MustMakeTenantID at line 52
pkg/keys/sql.go in pkg/keys.DecodeTenantPrefix at line 43
pkg/keys/printer.go in pkg/keys.formatTenantKey at line 652
pkg/keys/printer.go in pkg/keys.safeFormatInternal.func1 at line 693
pkg/keys/printer.go in pkg/keys.safeFormatInternal at line 749
pkg/keys/printer.go in pkg/keys.PrettyPrint at line 599
pkg/roachpb/pkg/roachpb/data.go in pkg/roachpb.Key.StringWithDirs at line 244
pkg/sql/catalog/catalogkeys/keys.go in pkg/sql/catalog/catalogkeys.PrettyKey at line 171
pkg/sql/sem/builtins/builtins.go in pkg/sql/sem/builtins.glob..func258 at line 5518
pkg/sql/colexec/builtin_funcs.go in pkg/sql/colexec.(*defaultBuiltinFuncOperator).Next.func1 at line 77
pkg/sql/colmem/allocator.go in pkg/sql/colmem.(*Allocator).PerformOperation at line 460
pkg/sql/colexec/builtin_funcs.go in pkg/sql/colexec.(*defaultBuiltinFuncOperator).Next at line 54
pkg/sql/colexec/colexecbase/simple_project.go in pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next at line 124
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 118
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 126
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter at line 251
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).next at line 255
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).Run at line 293
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 326
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 902
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1913
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3 at line 1645
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll at line 1648
Tag Value
Cockroach Release v23.1.2-1-gef8d5da59d2
Cockroach SHA: ef8d5da
Platform linux amd64
Distribution CCL
Environment v23.1.2
Command server
Go Version ``
# of CPUs
# of Goroutines

Jira issue: CRDB-28841
@cockroach-teamcity cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Jun 16, 2023
@yuzefovich
Copy link
Member

dup of #104606

@exalate-issue-sync exalate-issue-sync bot changed the title sentry: conn_executor.go:890: panic: × (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1 | github.com/cockroachdb/cockroach/pkg/sql/conn_exec... sentry: conn_executor.go:890: panic: × (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1 | github.com/cockroachdb/cockroach/pkg/sql/conn_exec... Jun 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yuzefovich @cockroach-teamcity