Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sentry: alter_table_drop_column.go:481: building declarative schema change targets for ALTER TABLE: failed to drop all of the relevant elements: [× ×] (1) attached stack trace -- stack trace: | ... #110953

Closed
cockroach-sentry opened this issue Sep 20, 2023 · 1 comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented Sep 20, 2023

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/4490601424/?referrer=webhooks_plugin

Panic Message:

alter_table_drop_column.go:481: building declarative schema change targets for ALTER TABLE: failed to drop all of the relevant elements: [× ×]
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scerrors.EventLogger.HandlePanicAndLogError
  | 	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scerrors/errors.go:77
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:884
  | [...repeated from below...]
Wraps: (2) building declarative schema change targets for ALTER TABLE
Wraps: (3) assertion failure
Wraps: (4) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scbuild/internal/scbuildstmt.assertAllColumnElementsAreDropped
  | 	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scbuild/internal/scbuildstmt/alter_table_drop_column.go:481
  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scbuild/internal/scbuildstmt.dropColumn
  | 	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scbuild/internal/scbuildstmt/alter_table_drop_column.go:286
  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scbuild/internal/scbuildstmt.alterTableDropColumn
  | 	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scbuild/internal/scbuildstmt/alter_table_drop_column.go:43
  | reflect.Value.call
  | 	GOROOT/src/reflect/value.go:584
  | reflect.Value.Call
  | 	GOROOT/src/reflect/value.go:368
  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scbuild/internal/scbuildstmt.AlterTable
  | 	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scbuild/internal/scbuildstmt/alter_table.go:174
  | reflect.Value.call
  | 	GOROOT/src/reflect/value.go:584
  | reflect.Value.Call
  | 	GOROOT/src/reflect/value.go:368
  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scbuild/internal/scbuildstmt.Process
  | 	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scbuild/internal/scbuildstmt/process.go:169
  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scbuild.Build
  | 	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scbuild/build.go:62
  | github.com/cockroachdb/cockroach/pkg/sql.(*planner).SchemaChange
  | 	github.com/cockroachdb/cockroach/pkg/sql/schema_change_plan_node.go:80
  | github.com/cockroachdb/cockroach/pkg/sql.buildOpaque
  | 	github.com/cockroachdb/cockroach/pkg/sql/opaque.go:56
  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).tryBuildOpaque
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/opaque.go:70
  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmt
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:404
  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmtAtRoot
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:252
  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).Build
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:226
  | github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).buildExecMemo
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:560
  | github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:231
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).makeExecPlan
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1432
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1058
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:687
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:129
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2382
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:128
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1920
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1924
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1846
  | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:828
  | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:728
  | runtime.goexit
  | 	GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (5) failed to drop all of the relevant elements: [× ×]
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *assert.withAssertionFailure (4) *withstack.withStack (5) *errutil.leafError
-- report composition:
*errutil.leafError: failed to drop all of the relevant elements: [× ×]
alter_table_drop_column.go:481: *withstack.withStack (top exception)
*assert.withAssertionFailure
*errutil.withPrefix: building declarative schema change targets for ALTER TABLE
errors.go:77: *withstack.withStack (1)
(check the extra data payloads)
Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/asm_amd64.s#L1593-L1595

reservedOwned = false // We're about to pass ownership away.
retErr = sqlServer.ServeConn(ctx, connHandler, reserved, cancelConn)
}()

}(ctx, h)
return h.ex.run(ctx, s.pool, reserved, cancel)
}

var err error
if err = ex.execCmd(); err != nil {
if errors.IsAny(err, io.EOF, errDrainingComplete) {

return err
}()
// Note: we write to ex.statsCollector.PhaseTimes, instead of ex.phaseTimes,

canAutoCommit := ex.implicitTxn() && (tcmd.LastInBatch || !implicitTxnForBatch)
ev, payload, err = ex.execStmt(
ctx, tcmd.Statement, nil /* prepared */, nil /* pinfo */, stmtRes, canAutoCommit,

case stateOpen:
err = ex.execWithProfiling(ctx, ast, prepared, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res, canAutoCommit)

} else {
err = op(ctx)
}

err = ex.execWithProfiling(ctx, ast, prepared, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res, canAutoCommit)
return err

if err := ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil {
stmtThresholdSpan.Finish()

// between here and there needs to happen even if there's an error.
err := ex.makeExecPlan(ctx, planner)
// We'll be closing the plan manually below after execution; this

func (ex *connExecutor) makeExecPlan(ctx context.Context, planner *planner) error {
if err := planner.makeOptimizerPlan(ctx); err != nil {
log.VEventf(ctx, 1, "optimizer plan failed: %v", err)

execMemo, err := opc.buildExecMemo(ctx)
if err != nil {

bld := optbuilder.New(ctx, &p.semaCtx, p.EvalContext(), &opc.catalog, f, opc.p.stmt.AST)
if err := bld.Build(); err != nil {
return nil, err

// and physical properties.
outScope := b.buildStmtAtRoot(b.stmt, nil /* desiredTypes */)

b.ctes = nil
outScope = b.buildStmt(stmt, desiredTypes, inScope)
// Build With operators for any CTEs hoisted to the top level.

// See if we have an opaque handler registered for this statement type.
if outScope := b.tryBuildOpaque(stmt, inScope); outScope != nil {
// The opaque handler may resolve objects; we don't care about caching

}
obj, err := info.buildFn(b.ctx, b.semaCtx, b.evalCtx, stmt)
if err != nil {

var err error
plan, err = p.SchemaChange(ctx, stmt)
if err != nil {

)
state, err := scbuild.Build(ctx, deps, scs.state, stmt)
if scerrors.HasNotImplemented(err) &&

}
scbuildstmt.Process(b, an.GetStatement())
an.ValidateAnnotations()


GOROOT/src/reflect/value.go#L367-L369
GOROOT/src/reflect/value.go#L583-L585
fn := reflect.ValueOf(info.fn)
fn.Call([]reflect.Value{
reflect.ValueOf(b),

GOROOT/src/reflect/value.go#L367-L369
GOROOT/src/reflect/value.go#L583-L585
checkColumnNotInaccessible(col, n)
dropColumn(b, tn, tbl, n, col, elts, n.DropBehavior)
}

}
assertAllColumnElementsAreDropped(colElts)
}

})
panic(errors.AssertionFailedf("failed to drop all of the relevant elements: %v", elements))
}

GOROOT/src/runtime/panic.go#L883-L885
if errors.HasAssertionFailure(*err) {
*err = errors.Wrapf(*err, "%s", el.msg)
}

GOROOT/src/runtime/asm_amd64.s#L1593-L1595
reservedOwned = false // We're about to pass ownership away.
retErr = sqlServer.ServeConn(ctx, connHandler, reserved, cancelConn)
}()

}(ctx, h)
return h.ex.run(ctx, s.pool, reserved, cancel)
}

var err error
if err = ex.execCmd(); err != nil {
if errors.IsAny(err, io.EOF, errDrainingComplete) {

return err
}()
// Note: we write to ex.statsCollector.PhaseTimes, instead of ex.phaseTimes,

canAutoCommit := ex.implicitTxn() && (tcmd.LastInBatch || !implicitTxnForBatch)
ev, payload, err = ex.execStmt(
ctx, tcmd.Statement, nil /* prepared */, nil /* pinfo */, stmtRes, canAutoCommit,

case stateOpen:
err = ex.execWithProfiling(ctx, ast, prepared, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res, canAutoCommit)

} else {
err = op(ctx)
}

err = ex.execWithProfiling(ctx, ast, prepared, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res, canAutoCommit)
return err

if err := ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil {
stmtThresholdSpan.Finish()

// between here and there needs to happen even if there's an error.
err := ex.makeExecPlan(ctx, planner)
// We'll be closing the plan manually below after execution; this

func (ex *connExecutor) makeExecPlan(ctx context.Context, planner *planner) error {
if err := planner.makeOptimizerPlan(ctx); err != nil {
log.VEventf(ctx, 1, "optimizer plan failed: %v", err)

execMemo, err := opc.buildExecMemo(ctx)
if err != nil {

bld := optbuilder.New(ctx, &p.semaCtx, p.EvalContext(), &opc.catalog, f, opc.p.stmt.AST)
if err := bld.Build(); err != nil {
return nil, err

// and physical properties.
outScope := b.buildStmtAtRoot(b.stmt, nil /* desiredTypes */)

b.ctes = nil
outScope = b.buildStmt(stmt, desiredTypes, inScope)
// Build With operators for any CTEs hoisted to the top level.

// See if we have an opaque handler registered for this statement type.
if outScope := b.tryBuildOpaque(stmt, inScope); outScope != nil {
// The opaque handler may resolve objects; we don't care about caching

}
obj, err := info.buildFn(b.ctx, b.semaCtx, b.evalCtx, stmt)
if err != nil {

var err error
plan, err = p.SchemaChange(ctx, stmt)
if err != nil {

)
state, err := scbuild.Build(ctx, deps, scs.state, stmt)
if scerrors.HasNotImplemented(err) &&

}
scbuildstmt.Process(b, an.GetStatement())
an.ValidateAnnotations()


GOROOT/src/reflect/value.go#L367-L369
GOROOT/src/reflect/value.go#L583-L585
fn := reflect.ValueOf(info.fn)
fn.Call([]reflect.Value{
reflect.ValueOf(b),

GOROOT/src/reflect/value.go#L367-L369
GOROOT/src/reflect/value.go#L583-L585
checkColumnNotInaccessible(col, n)
dropColumn(b, tn, tbl, n, col, elts, n.DropBehavior)
}

}
assertAllColumnElementsAreDropped(colElts)
}

})
panic(errors.AssertionFailedf("failed to drop all of the relevant elements: %v", elements))
}

GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 728
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 828
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 1846
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 1924
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func1 at line 1920
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 128
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 2382
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func1 at line 129
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 687
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1058
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).makeExecPlan at line 1432
pkg/sql/plan_opt.go in pkg/sql.(*planner).makeOptimizerPlan at line 231
pkg/sql/plan_opt.go in pkg/sql.(*optPlanningCtx).buildExecMemo at line 560
pkg/sql/opt/optbuilder/builder.go in pkg/sql/opt/optbuilder.(*Builder).Build at line 226
pkg/sql/opt/optbuilder/builder.go in pkg/sql/opt/optbuilder.(*Builder).buildStmtAtRoot at line 252
pkg/sql/opt/optbuilder/builder.go in pkg/sql/opt/optbuilder.(*Builder).buildStmt at line 404
pkg/sql/opt/optbuilder/opaque.go in pkg/sql/opt/optbuilder.(*Builder).tryBuildOpaque at line 70
pkg/sql/opaque.go in pkg/sql.buildOpaque at line 56
pkg/sql/schema_change_plan_node.go in pkg/sql.(*planner).SchemaChange at line 80
pkg/sql/schemachanger/scbuild/build.go in pkg/sql/schemachanger/scbuild.Build at line 62
pkg/sql/schemachanger/scbuild/internal/scbuildstmt/process.go in pkg/sql/schemachanger/scbuild/internal/scbuildstmt.Process at line 169
GOROOT/src/reflect/value.go in reflect.Value.Call at line 368
GOROOT/src/reflect/value.go in reflect.Value.call at line 584
pkg/sql/schemachanger/scbuild/internal/scbuildstmt/alter_table.go in pkg/sql/schemachanger/scbuild/internal/scbuildstmt.AlterTable at line 174
GOROOT/src/reflect/value.go in reflect.Value.Call at line 368
GOROOT/src/reflect/value.go in reflect.Value.call at line 584
pkg/sql/schemachanger/scbuild/internal/scbuildstmt/alter_table_drop_column.go in pkg/sql/schemachanger/scbuild/internal/scbuildstmt.alterTableDropColumn at line 43
pkg/sql/schemachanger/scbuild/internal/scbuildstmt/alter_table_drop_column.go in pkg/sql/schemachanger/scbuild/internal/scbuildstmt.dropColumn at line 286
pkg/sql/schemachanger/scbuild/internal/scbuildstmt/alter_table_drop_column.go in pkg/sql/schemachanger/scbuild/internal/scbuildstmt.assertAllColumnElementsAreDropped at line 481
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
pkg/sql/schemachanger/scerrors/errors.go in pkg/sql/schemachanger/scerrors.EventLogger.HandlePanicAndLogError at line 77
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 728
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 828
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 1846
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 1924
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func1 at line 1920
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 128
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 2382
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func1 at line 129
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 687
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1058
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).makeExecPlan at line 1432
pkg/sql/plan_opt.go in pkg/sql.(*planner).makeOptimizerPlan at line 231
pkg/sql/plan_opt.go in pkg/sql.(*optPlanningCtx).buildExecMemo at line 560
pkg/sql/opt/optbuilder/builder.go in pkg/sql/opt/optbuilder.(*Builder).Build at line 226
pkg/sql/opt/optbuilder/builder.go in pkg/sql/opt/optbuilder.(*Builder).buildStmtAtRoot at line 252
pkg/sql/opt/optbuilder/builder.go in pkg/sql/opt/optbuilder.(*Builder).buildStmt at line 404
pkg/sql/opt/optbuilder/opaque.go in pkg/sql/opt/optbuilder.(*Builder).tryBuildOpaque at line 70
pkg/sql/opaque.go in pkg/sql.buildOpaque at line 56
pkg/sql/schema_change_plan_node.go in pkg/sql.(*planner).SchemaChange at line 80
pkg/sql/schemachanger/scbuild/build.go in pkg/sql/schemachanger/scbuild.Build at line 62
pkg/sql/schemachanger/scbuild/internal/scbuildstmt/process.go in pkg/sql/schemachanger/scbuild/internal/scbuildstmt.Process at line 169
GOROOT/src/reflect/value.go in reflect.Value.Call at line 368
GOROOT/src/reflect/value.go in reflect.Value.call at line 584
pkg/sql/schemachanger/scbuild/internal/scbuildstmt/alter_table.go in pkg/sql/schemachanger/scbuild/internal/scbuildstmt.AlterTable at line 174
GOROOT/src/reflect/value.go in reflect.Value.Call at line 368
GOROOT/src/reflect/value.go in reflect.Value.call at line 584
pkg/sql/schemachanger/scbuild/internal/scbuildstmt/alter_table_drop_column.go in pkg/sql/schemachanger/scbuild/internal/scbuildstmt.alterTableDropColumn at line 43
pkg/sql/schemachanger/scbuild/internal/scbuildstmt/alter_table_drop_column.go in pkg/sql/schemachanger/scbuild/internal/scbuildstmt.dropColumn at line 286
pkg/sql/schemachanger/scbuild/internal/scbuildstmt/alter_table_drop_column.go in pkg/sql/schemachanger/scbuild/internal/scbuildstmt.assertAllColumnElementsAreDropped at line 481

Tags

Tag Value
Command server
Environment v22.2.0
Go Version go1.19.1
Platform linux amd64
Distribution CCL
Cockroach Release v22.2.0
Cockroach SHA 77667a1
# of CPUs 4
# of Goroutines 977

Jira issue: CRDB-31684

@cockroach-sentry cockroach-sentry added O-sentry Originated from an in-the-wild panic report. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. labels Sep 20, 2023
@yuzefovich
Copy link
Member

dup of #105754

@exalate-issue-sync exalate-issue-sync bot changed the title Sentry: alter_table_drop_column.go:481: building declarative schema change targets for ALTER TABLE: failed to drop all of the relevant elements: [× ×] (1) attached stack trace -- stack trace: | ... Sentry: alter_table_drop_column.go:481: building declarative schema change targets for ALTER TABLE: failed to drop all of the relevant elements: [× ×] (1) attached stack trace -- stack trace: | ... Sep 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Development

No branches or pull requests

2 participants