Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sentry: txn_coord_sender.go:983: transaction unexpectedly committed, ba: Put [/Table/160/1/×/×,/Min), Delete [/Table/160/14/×/×/×,/Min), ConditionalPut [/Table/160/14/×/×/×,/Min), Delete [/Tab... #114220

Closed
cockroach-sentry opened this issue Nov 10, 2023 · 1 comment
Closed

Sentry: txn_coord_sender.go:983: transaction unexpectedly committed, ba: Put [/Table/160/1/×/×,/Min), Delete [/Table/160/14/×/×/×,/Min), ConditionalPut [/Table/160/14/×/×/×,/Min), Delete [/Tab... #114220

cockroach-sentry opened this issue Nov 10, 2023 · 1 comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented Nov 10, 2023 •
edited by cockroach-jira-scripts
Loading

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/4619810585/?referrer=webhooks_plugin

Panic Message:

txn_coord_sender.go:983: transaction unexpectedly committed, ba: Put [/Table/160/1/×/×,/Min), Delete [/Table/160/14/×/×/×,/Min), ConditionalPut [/Table/160/14/×/×/×,/Min), Delete [/Table/160/18/×/×/×/×,/Min), Delete [/Table/160/26/×/×/×/×/×/×/×,/Min), ConditionalPut [/Table/160/27/×/×/×,/Min), EndTxn(commit) [/Table/160/1/×/×], [txn: f730372e]. txn: "sql txn" meta={id=f730372e key=/Table/160/1/×/× pri=0.02086121 epo=0 ts=1699602929.375007546,1 min=1699602919.046599654,0 seq=7} lock=true stat=COMMITTED rts=1699602929.375007546,1 wto=false gul=1699602919.546599654,0 int=6 ifw=5: TransactionStatusError: already committed (REASON_TXN_COMMITTED)
(1)
Wraps: (2) assertion failure
Wraps: (3) issue: https://github.com/cockroachdb/cockroach/issues/103817
  | detail: you have encountered a known bug in CockroachDB, please consider reporting on the Github issue or reach out via Support.
Wraps: (4)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.sanityCheckErrWithTxn
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:983
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).updateStateLocked
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:947
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:524
  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1006
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send
  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:1060
  | github.com/cockroachdb/cockroach/pkg/kv.sendAndFill
  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:836
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Run
  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:675
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).CommitInBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:723
  | github.com/cockroachdb/cockroach/pkg/sql.(*tableWriterBase).finalize
  | 	github.com/cockroachdb/cockroach/pkg/sql/tablewriter.go:230
  | github.com/cockroachdb/cockroach/pkg/sql.(*updateNode).BatchedNext
  | 	github.com/cockroachdb/cockroach/pkg/sql/update.go:201
  | github.com/cockroachdb/cockroach/pkg/sql.(*serializeNode).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_batch.go:115
  | github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:220
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:239
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:250
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:254
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:286
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:324
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:902
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1913
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1645
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1648
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2155
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1709
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1036
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:142
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:3086
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:141
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:247
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2287
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2289
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2104
Wraps: (5) transaction unexpectedly committed, ba: Put [/Table/160/1/×/×,/Min), Delete [/Table/160/14/×/×/×,/Min), ConditionalPut [/Table/160/14/×/×/×,/Min), Delete [/Table/160/18/×/×/×/×,/Min), Delete [/Table/160/26/×/×/×/×/×/×/×,/Min), ConditionalPut [/Table/160/27/×/×/×,/Min), EndTxn(commit) [/Table/160/1/×/×], [txn: f730372e]. txn: "sql txn" meta={id=f730372e key=/Table/160/1/×/× pri=0.02086121 epo=0 ts=1699602929.375007546,1 min=1699602919.046599654,0 seq=7} lock=true stat=COMMITTED rts=1699602929.375007546,1 wto=false gul=1699602919.546599654,0 int=6 ifw=5
Wraps: (6) TransactionStatusError: already committed (REASON_TXN_COMMITTED)
Error types: (1) *colexecerror.notInternalError (2) *assert.withAssertionFailure (3) *issuelink.withIssueLink (4) *errbase.opaqueWrapper (5) *errutil.withPrefix (6) *kvpb.TransactionStatusError
-- report composition:
*kvpb.TransactionStatusError
*errutil.withPrefix: transaction unexpectedly committed, ba: Put [/Table/160/1/×/×,/Min), Delete [/Table/160/14/×/×/×,/Min), ConditionalPut [/Table/160/14/×/×/×,/Min), Delete [/Table/160/18/×/×/×/×,/Min), Delete [/Table/160/26/×/×/×/×/×/×/×,/Min), ConditionalPut [/Table/160/27/×/×/×,/Min), EndTxn(commit) [/Table/160/1/×/×], [txn: f730372e]. txn: "sql txn" meta={id=f730372e key=/Table/160/1/×/× pri=0.02086121 epo=0 ts=1699602929.375007546,1 min=1699602919.046599654,0 seq=7} lock=true stat=COMMITTED rts=1699602929.375007546,1 wto=false gul=1699602919.546599654,0 int=6 ifw=5
txn_coord_sender.go:983: *withstack.withStack (top exception)
*issuelink.withIssueLink: https://github.com/cockroachdb/cockroach/issues/103817
*assert.withAssertionFailure
*colexecerror.notInternalError
Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/conn_executor.go

Lines 2103 to 2105 in 6bd6e0c

var err error
if err = ex.execCmd(); err != nil {
// Both of these errors are normal ways for the connExecutor to exit.

cockroach/pkg/sql/conn_executor.go

Lines 2288 to 2290 in 6bd6e0c

return err
}()
// Note: we write to ex.statsCollector.phaseTimes, instead of ex.phaseTimes,

cockroach/pkg/sql/conn_executor.go

Lines 2286 to 2288 in 6bd6e0c

canAutoCommit := ex.implicitTxn() && tcmd.FollowedBySync
ev, payload, err = ex.execPortal(ctx, portal, portalName, stmtRes, pinfo, canAutoCommit)
return err

cockroach/pkg/sql/conn_executor_exec.go

Lines 246 to 248 in 6bd6e0c

}
ev, payload, retErr = ex.execStmt(ctx, portal.Stmt.Statement, &portal, pinfo, stmtRes, canAutoCommit)
// For a non-pausable portal, it is considered exhausted regardless of the

cockroach/pkg/sql/conn_executor_exec.go

Lines 140 to 142 in 6bd6e0c

}
err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit)

cockroach/pkg/sql/conn_executor_exec.go

Lines 3085 to 3087 in 6bd6e0c

} else {
err = op(ctx)
}

cockroach/pkg/sql/conn_executor_exec.go

Lines 141 to 143 in 6bd6e0c

err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit)
return err

cockroach/pkg/sql/conn_executor_exec.go

Lines 1035 to 1037 in 6bd6e0c

if err = ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil {
stmtThresholdSpan.Finish()

cockroach/pkg/sql/conn_executor_exec.go

Lines 1708 to 1710 in 6bd6e0c

ex.sessionTracing.TraceExecStart(ctx, "distributed")
stats, err = ex.execWithDistSQLEngine(
ctx, planner, stmt.AST.StatementReturnType(), res, distribute, progAtomic,

cockroach/pkg/sql/conn_executor_exec.go

Lines 2154 to 2156 in 6bd6e0c

}
err = ex.server.cfg.DistSQLPlanner.PlanAndRunAll(ctx, evalCtx, planCtx, planner, recv, evalCtxFactory)
}

cockroach/pkg/sql/distsql_running.go

Lines 1647 to 1649 in 6bd6e0c

)
}()

cockroach/pkg/sql/distsql_running.go

Lines 1644 to 1646 in 6bd6e0c

defer cleanup()
dsp.PlanAndRun(
ctx, evalCtx, planCtx, planner.txn, planner.curPlan.main, recv, finishedSetupFn,

cockroach/pkg/sql/distsql_running.go

Lines 1912 to 1914 in 6bd6e0c

recv.expectedRowsRead = int64(physPlan.TotalEstimatedScannedRows)
dsp.Run(ctx, planCtx, txn, physPlan, recv, evalCtx, finishedSetupFn)
}

cockroach/pkg/sql/distsql_running.go

Lines 901 to 903 in 6bd6e0c

noWait := planCtx.getPortalPauseInfo() != nil
flow.Run(ctx, noWait)
}

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 323 to 325 in 6bd6e0c

log.VEvent(ctx, 1, "running the batch flow coordinator in the flow's goroutine")
f.batchFlowCoordinator.Run(ctx)
}

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 285 to 287 in 6bd6e0c

for status == execinfra.NeedMoreRows {
err := f.next()
if err != nil {

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 253 to 255 in 6bd6e0c

func (f *BatchFlowCoordinator) next() error {
return colexecerror.CatchVectorizedRuntimeError(f.nextAdapter)
}

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in 6bd6e0c

}()
operation()
return retErr

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 249 to 251 in 6bd6e0c

func (f *BatchFlowCoordinator) nextAdapter() {
f.batch = f.input.Root.Next()
}

cockroach/pkg/sql/colexec/columnarizer.go

Lines 238 to 240 in 6bd6e0c

for batchDone := false; !batchDone; {
row, meta := c.input.Next()
if meta != nil {

cockroach/pkg/sql/plan_node_to_row_source.go

Lines 219 to 221 in 6bd6e0c

for p.State == execinfra.StateRunning {
valid, err := p.node.Next(p.params)
if err != nil || !valid {

cockroach/pkg/sql/plan_batch.go

Lines 114 to 116 in 6bd6e0c

// First batch, or finished previous batch; advance one.
if next, err := s.source.BatchedNext(params); !next {
return false, err

cockroach/pkg/sql/update.go

Lines 200 to 202 in 6bd6e0c

u.run.tu.setRowsWrittenLimit(params.extendedEvalCtx.SessionData())
if err := u.run.tu.finalize(params.ctx); err != nil {
return false, err

cockroach/pkg/sql/tablewriter.go

Lines 229 to 231 in 6bd6e0c

// coordinator.
err = tb.txn.CommitInBatch(ctx, tb.b)
} else {

cockroach/pkg/kv/txn.go

Lines 722 to 724 in 6bd6e0c

b.initResult(1 /* calls */, 0, b.raw, nil)
return txn.Run(ctx, b)
}

cockroach/pkg/kv/txn.go

Lines 674 to 676 in 6bd6e0c

}
return sendAndFill(ctx, txn.Send, b)
}

cockroach/pkg/kv/db.go

Lines 835 to 837 in 6bd6e0c

ba.AdmissionHeader = b.AdmissionHeader
b.response, b.pErr = send(ctx, ba)
b.fillResults(ctx)

cockroach/pkg/kv/txn.go

Lines 1059 to 1061 in 6bd6e0c

txn.mu.Unlock()
br, pErr := txn.db.sendUsingSender(ctx, ba, sender)
if pErr == nil {

cockroach/pkg/kv/db.go

Lines 1005 to 1007 in 6bd6e0c

br, pErr := sender.Send(ctx, ba)
if pErr != nil {

cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go

Lines 523 to 525 in 6bd6e0c

pErr = tc.updateStateLocked(ctx, ba, br, pErr)

cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go

Lines 946 to 948 in 6bd6e0c

if errTxn := pErr.GetTxn(); errTxn != nil {
if err := sanityCheckErrWithTxn(ctx, pErr, ba, &tc.testingKnobs); err != nil {
return kvpb.NewError(err)

cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go

Lines 982 to 984 in 6bd6e0c

// are committed.
err := errors.Wrapf(pErrWithTxn.GoError(),
"transaction unexpectedly committed, ba: %s. txn: %s", 

pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 2104
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 2289
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func2 at line 2287
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execPortal at line 247
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 141
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 3086
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func1 at line 142
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 1036
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1709
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 2155
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll at line 1648
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3 at line 1645
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1913
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 902
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 324
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).Run at line 286
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).next at line 254
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter at line 250
pkg/sql/colexec/columnarizer.go in pkg/sql/colexec.(*Columnarizer).Next at line 239
pkg/sql/plan_node_to_row_source.go in pkg/sql.(*planNodeToRowSource).Next at line 220
pkg/sql/plan_batch.go in pkg/sql.(*serializeNode).Next at line 115
pkg/sql/update.go in pkg/sql.(*updateNode).BatchedNext at line 201
pkg/sql/tablewriter.go in pkg/sql.(*tableWriterBase).finalize at line 230
pkg/kv/txn.go in pkg/kv.(*Txn).CommitInBatch at line 723
pkg/kv/txn.go in pkg/kv.(*Txn).Run at line 675
pkg/kv/db.go in pkg/kv.sendAndFill at line 836
pkg/kv/txn.go in pkg/kv.(*Txn).Send at line 1060
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 1006
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send at line 524
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).updateStateLocked at line 947
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.sanityCheckErrWithTxn at line 983

Tags

Tag Value
Command server
Environment v23.1.10
Go Version go1.19.10
Platform linux amd64
Distribution CCL
Cockroach Release v23.1.10
Cockroach SHA 6bd6e0c
# of CPUs 8
# of Goroutines 707

Jira issue: CRDB-33400
@cockroach-sentry cockroach-sentry added O-sentry Originated from an in-the-wild panic report. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. labels Nov 10, 2023
@yuzefovich
Copy link
Member

fixed later

@renatolabs renatolabs mentioned this issue Nov 17, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yuzefovich @cockroach-sentry