Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sentry: table.go:202: attempted to update job for mutation 8, but job already exists with mutation 7 (1) Wraps: (2) assertion failure Wraps: (3) attached stack trace -- stack trace: | github.com/cockroach... #83997

Closed
cockroach-teamcity opened this issue Jul 7, 2022 · 1 comment
Closed

sentry: table.go:202: attempted to update job for mutation 8, but job already exists with mutation 7 (1) Wraps: (2) assertion failure Wraps: (3) attached stack trace -- stack trace: | github.com/cockroach... #83997

cockroach-teamcity opened this issue Jul 7, 2022 · 1 comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jul 7, 2022 •
edited by cockroach-jira-scripts
Loading

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3408430427/?referrer=webhooks_plugin

Panic message:

table.go:202: attempted to update job for mutation 8, but job already exists with mutation 7
(1)
Wraps: (2) assertion failure
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*planner).createOrUpdateSchemaChangeJob
| github.com/cockroachdb/cockroach/pkg/sql/table.go:202
| github.com/cockroachdb/cockroach/pkg/sql.(*planner).writeSchemaChange
| github.com/cockroachdb/cockroach/pkg/sql/table.go:248
| github.com/cockroachdb/cockroach/pkg/sql.(*planner).dropIndexByName
| github.com/cockroachdb/cockroach/pkg/sql/drop_index.go:554
| github.com/cockroachdb/cockroach/pkg/sql.(*dropIndexNode).startExec
| github.com/cockroachdb/cockroach/pkg/sql/drop_index.go:162
| github.com/cockroachdb/cockroach/pkg/sql.startExec.func2
| github.com/cockroachdb/cockroach/pkg/sql/plan.go:516
| github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1
| github.com/cockroachdb/cockroach/pkg/sql/walk.go:112
| github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal
| github.com/cockroachdb/cockroach/pkg/sql/walk.go:297
| github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit
| github.com/cockroachdb/cockroach/pkg/sql/walk.go:79
| github.com/cockroachdb/cockroach/pkg/sql.walkPlan
| github.com/cockroachdb/cockroach/pkg/sql/walk.go:43
| github.com/cockroachdb/cockroach/pkg/sql.startExec
| github.com/cockroachdb/cockroach/pkg/sql/plan.go:519
| github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start
| github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:146
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Init
| github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:158
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Init
| github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:90
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Start.func1
| github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:248
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Start
| github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:247
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Start.func1
| github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:120
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Start
| github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:119
| github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run
| github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:752
| github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run
| github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:472
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
| github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:247
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:597
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1445
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1469
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1145
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:686
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:143
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:231
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1952
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1954
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1800
Wraps: (4) attempted to update job for mutation 8, but job already exists with mutation 7
Error types: (1) *colexecerror.notInternalError (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
-- report composition:
*errutil.leafError: attempted to update job for mutation 8, but job already exists with mutation 7
table.go:202: *withstack.withStack (top exception)
*assert.withAssertionFailure
*colexecerror.notInternalError

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/table.go

Lines 201 to 203 in fc456a2

if mutationID != descpb.InvalidMutationID && mutationID != oldDetails.TableMutationID {
return errors.AssertionFailedf(
"attempted to update job for mutation %d, but job already exists with mutation %d",
in pkg/sql.(*planner).createOrUpdateSchemaChangeJob

cockroach/pkg/sql/table.go

Lines 247 to 249 in fc456a2

if !tableDesc.IsNew() {
if err := p.createOrUpdateSchemaChangeJob(ctx, tableDesc, jobDesc, mutationID); err != nil {
return err
in pkg/sql.(*planner).writeSchemaChange

cockroach/pkg/sql/drop_index.go

Lines 553 to 555 in fc456a2

mutationID := tableDesc.ClusterVersion().NextMutationID
if err := p.writeSchemaChange(ctx, tableDesc, mutationID, jobDesc); err != nil {
return err
in pkg/sql.(*planner).dropIndexByName

cockroach/pkg/sql/drop_index.go

Lines 161 to 163 in fc456a2

// different index than the one being dropped.
if err := params.p.dropIndexByName(
ctx, index.tn, index.idxName, tableDesc, n.n.IfExists, n.n.DropBehavior, checkIdxConstraint,
in pkg/sql.(*dropIndexNode).startExec

cockroach/pkg/sql/plan.go

Lines 515 to 517 in fc456a2

}
return n.startExec(params)
},
in pkg/sql.startExec.func2

cockroach/pkg/sql/walk.go

Lines 111 to 113 in fc456a2

}
v.err = v.observer.leaveNode(name, plan)
}()
in pkg/sql.(*planVisitor).visitInternal.func1

cockroach/pkg/sql/walk.go

Lines 296 to 298 in fc456a2

}
}
in pkg/sql.(*planVisitor).visitInternal

cockroach/pkg/sql/walk.go

Lines 78 to 80 in fc456a2

}
v.visitInternal(plan, name)
return plan
in pkg/sql.(*planVisitor).visit

cockroach/pkg/sql/walk.go

Lines 42 to 44 in fc456a2

v := makePlanVisitor(ctx, observer)
v.visit(plan)
return v.err
in pkg/sql.walkPlan

cockroach/pkg/sql/plan.go

Lines 518 to 520 in fc456a2

}
return walkPlan(params.ctx, plan, o)
}
in pkg/sql.startExec

cockroach/pkg/sql/plan_node_to_row_source.go

Lines 145 to 147 in fc456a2

// This starts all of the nodes below this node.
if err := startExec(p.params, p.node); err != nil {
p.MoveToDraining(err)
in pkg/sql.(*planNodeToRowSource).Start

cockroach/pkg/sql/colexec/columnarizer.go

Lines 157 to 159 in fc456a2

ctx = c.StartInternalNoSpan(ctx)
c.input.Start(ctx)
if execStatsHijacker, ok := c.input.(execinfra.ExecStatsForTraceHijacker); ok {
in pkg/sql/colexec.(*Columnarizer).Init

cockroach/pkg/sql/colflow/stats.go

Lines 89 to 91 in fc456a2

func (bic *batchInfoCollector) Init(ctx context.Context) {
bic.Operator.Init(ctx)
bic.mu.Lock()
in pkg/sql/colflow.(*batchInfoCollector).Init

cockroach/pkg/sql/colexec/materializer.go

Lines 247 to 249 in fc456a2

if err := colexecerror.CatchVectorizedRuntimeError(func() {
m.input.Init(ctx)
}); err != nil {
in pkg/sql/colexec.(*Materializer).Start.func1

cockroach/pkg/sql/colexecerror/error.go

Lines 90 to 92 in fc456a2

}()
operation()
return retErr
in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colexec/materializer.go

Lines 246 to 248 in fc456a2

// reached), so we need to wrap it with a catcher.
if err := colexecerror.CatchVectorizedRuntimeError(func() {
m.input.Init(ctx)
in pkg/sql/colexec.(*Materializer).Start

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 119 to 121 in fc456a2

if err := colexecerror.CatchVectorizedRuntimeError(func() {
f.input.Start(ctx)
}); err != nil {
in pkg/sql/colflow.(*FlowCoordinator).Start.func1

cockroach/pkg/sql/colexecerror/error.go

Lines 90 to 92 in fc456a2

}()
operation()
return retErr
in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 118 to 120 in fc456a2

ctx = f.StartInternalNoSpan(ctx)
if err := colexecerror.CatchVectorizedRuntimeError(func() {
f.input.Start(ctx)
in pkg/sql/colflow.(*FlowCoordinator).Start

cockroach/pkg/sql/execinfra/processorsbase.go

Lines 751 to 753 in fc456a2

}
pb.self.Start(ctx)
Run(pb.Ctx, pb.self, pb.Output)
in pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run

cockroach/pkg/sql/flowinfra/flow.go

Lines 471 to 473 in fc456a2

log.VEventf(ctx, 1, "running %T in the flow's goroutine", headProc)
headProc.Run(ctx)
}
in pkg/sql/flowinfra.(*FlowBase).Run

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 246 to 248 in fc456a2

// as the root, so we run this flow with the default implementation.
f.FlowBase.Run(ctx, doneFn)
return
in pkg/sql/colflow.(*vectorizedFlow).Run

cockroach/pkg/sql/distsql_running.go

Lines 596 to 598 in fc456a2

// TODO(radu): this should go through the flow scheduler.
flow.Run(ctx, func() {})
in pkg/sql.(*DistSQLPlanner).Run

cockroach/pkg/sql/distsql_running.go

Lines 1444 to 1446 in fc456a2

recv.expectedRowsRead = int64(physPlan.TotalEstimatedScannedRows)
runCleanup := dsp.Run(ctx, planCtx, txn, physPlan, recv, evalCtx, nil /* finishedSetupFn */)
return func() {
in pkg/sql.(*DistSQLPlanner).PlanAndRun

cockroach/pkg/sql/conn_executor_exec.go

Lines 1468 to 1470 in fc456a2

// the planner whether or not to plan remote table readers.
cleanup := ex.server.cfg.DistSQLPlanner.PlanAndRun(
ctx, evalCtx, planCtx, planner.txn, planner.curPlan.main, recv,
in pkg/sql.(*connExecutor).execWithDistSQLEngine

cockroach/pkg/sql/conn_executor_exec.go

Lines 1144 to 1146 in fc456a2

ex.sessionTracing.TraceExecStart(ctx, "distributed")
stats, err := ex.execWithDistSQLEngine(
ctx, planner, stmt.AST.StatementReturnType(), res, distribute, progAtomic,
in pkg/sql.(*connExecutor).dispatchToExecutionEngine

cockroach/pkg/sql/conn_executor_exec.go

Lines 685 to 687 in fc456a2

if err := ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil {
stmtThresholdSpan.Finish()
in pkg/sql.(*connExecutor).execStmtInOpenState

cockroach/pkg/sql/conn_executor_exec.go

Lines 142 to 144 in fc456a2

} else {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res, canAutoCommit)
}
in pkg/sql.(*connExecutor).execStmt

cockroach/pkg/sql/conn_executor_exec.go

Lines 230 to 232 in fc456a2

}
ev, payload, err = ex.execStmt(ctx, portal.Stmt.Statement, portal.Stmt, pinfo, stmtRes, canAutoCommit)
// Portal suspension is supported via a "side" state machine
in pkg/sql.(*connExecutor).execPortal

cockroach/pkg/sql/conn_executor.go

Lines 1951 to 1953 in fc456a2

canAutoCommit := ex.implicitTxn() && tcmd.FollowedBySync
ev, payload, err = ex.execPortal(ctx, portal, portalName, stmtRes, pinfo, canAutoCommit)
return err
in pkg/sql.(*connExecutor).execCmd.func2

cockroach/pkg/sql/conn_executor.go

Lines 1953 to 1955 in fc456a2

return err
}()
// Note: we write to ex.statsCollector.phaseTimes, instead of ex.phaseTimes,
in pkg/sql.(*connExecutor).execCmd

cockroach/pkg/sql/conn_executor.go

Lines 1799 to 1801 in fc456a2

var err error
if err = ex.execCmd(); err != nil {
if errors.IsAny(err, io.EOF, errDrainingComplete) {
in pkg/sql.(*connExecutor).run

pkg/sql/table.go in pkg/sql.(*planner).createOrUpdateSchemaChangeJob at line 202
pkg/sql/table.go in pkg/sql.(*planner).writeSchemaChange at line 248
pkg/sql/drop_index.go in pkg/sql.(*planner).dropIndexByName at line 554
pkg/sql/drop_index.go in pkg/sql.(*dropIndexNode).startExec at line 162
pkg/sql/plan.go in pkg/sql.startExec.func2 at line 516
pkg/sql/walk.go in pkg/sql.(*planVisitor).visitInternal.func1 at line 112
pkg/sql/walk.go in pkg/sql.(*planVisitor).visitInternal at line 297
pkg/sql/walk.go in pkg/sql.(*planVisitor).visit at line 79
pkg/sql/walk.go in pkg/sql.walkPlan at line 43
pkg/sql/plan.go in pkg/sql.startExec at line 519
pkg/sql/plan_node_to_row_source.go in pkg/sql.(*planNodeToRowSource).Start at line 146
pkg/sql/colexec/columnarizer.go in pkg/sql/colexec.(*Columnarizer).Init at line 158
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Init at line 90
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).Start.func1 at line 248
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).Start at line 247
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).Start.func1 at line 120
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).Start at line 119
pkg/sql/execinfra/processorsbase.go in pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run at line 752
pkg/sql/flowinfra/flow.go in pkg/sql/flowinfra.(*FlowBase).Run at line 472
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 247
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 597
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1445
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 1469
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1145
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 686
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 143
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execPortal at line 231
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func2 at line 1952
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 1954
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 1800
Tag Value
Cockroach Release v22.1.2
Cockroach SHA: fc456a2
Platform linux amd64
Distribution CCL
Environment v22.1.2
Command start-single-node
Go Version ``
# of CPUs
# of Goroutines

Jira issue: CRDB-17397
@cockroach-teamcity cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Jul 7, 2022
@yuzefovich
Copy link
Member

dup of #82921

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yuzefovich @cockroach-teamcity