Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: follower-reads/mixed-version/single-region failed #83502

Closed
cockroach-teamcity opened this issue Jun 28, 2022 · 6 comments · Fixed by #83625
Closed

roachtest: follower-reads/mixed-version/single-region failed #83502

cockroach-teamcity opened this issue Jun 28, 2022 · 6 comments · Fixed by #83625
Assignees
@stevendanna
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-jobs T-kv KV Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jun 28, 2022 •
edited by cockroach-jira-scripts
Loading

roachtest.follower-reads/mixed-version/single-region failed with artifacts on master @ 7f8fef4c3f24ec2b6f9a3db1eb38756da68b3d5a:

test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/run_1
	follower_reads.go:295,follower_reads.go:901,follower_reads.go:102,test_runner.go:896: error verifying node values: dial tcp 34.148.137.234:26257: connect: connection refused

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=2 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-17109
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Jun 28, 2022
@blathers-crl blathers-crl bot added the T-kv KV Team label Jun 28, 2022
@cockroach-teamcity
Copy link
Member Author

roachtest.follower-reads/mixed-version/single-region failed with artifacts on master @ ac11a55499120ffd33016feebc47b4ed43a21007:

test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/run_1
	follower_reads.go:295,follower_reads.go:901,follower_reads.go:102,test_runner.go:896: error verifying node values: dial tcp 35.227.105.156:26257: connect: connection refused

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=2 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@tbg
Copy link
Member

tbg commented Jun 29, 2022

https://teamcityartifacts.crdb.io/presignedTokenAuth/a04e0433-e0f3-4d1a-8de3-b959241077df/repository/download/Cockroach_Nightlies_RoachtestNightlyGceBazel/5594177:id/follower-reads/mixed-version/single-region/run_1/artifacts.zip!/logs/1.unredacted/cockroach-stderr.log

I would hazard a guess that this is invalid span usage (vs a bug in tracing), and it seems to be coming from a trace span thati s set up and finished as part of the jobs infrastructure. cc @cockroachdb/jobs

I220629 09:55:46.609531 1 util/log/flags.go:198  [-] 1  stderr capture started
panic: RecordingStructured has 12 recordings; expected 1 [recovered]
	panic: RecordingStructured has 12 recordings; expected 1 [recovered]
	panic: RecordingStructured has 12 recordings; expected 1

goroutine 831 [running]:
panic({0x46f6f80, 0xc003530960})
	GOROOT/src/runtime/panic.go:1147 +0x3a8 fp=0xc001d5ff88 sp=0xc001d5fec8 pc=0x48a828
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1()
	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:58 +0x3bb fp=0xc001d60120 sp=0xc001d5ff88 pc=0x1ffd71b
runtime.deferCallSave(0xc001d60228, 0xc001d61dc8)
	GOROOT/src/runtime/panic.go:950 +0x82 fp=0xc001d60130 sp=0xc001d60120 pc=0x48a422
runtime.runOpenDeferFrame(0xc0029ac500, 0xc0029ac550)
	GOROOT/src/runtime/panic.go:889 +0x27b fp=0xc001d601b0 sp=0xc001d60130 pc=0x489dbb
panic({0x46f6f80, 0xc003530960})
	GOROOT/src/runtime/panic.go:1038 +0x215 fp=0xc001d60270 sp=0xc001d601b0 pc=0x48a695
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1()
	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:58 +0x3bb fp=0xc001d60408 sp=0xc001d60270 pc=0x1ffd71b
runtime.deferCallSave(0xc001d60510, 0xc001d61cf0)
	GOROOT/src/runtime/panic.go:950 +0x82 fp=0xc001d60418 sp=0xc001d60408 pc=0x48a422
runtime.runOpenDeferFrame(0xc0029ac550, 0xc0029ac500)
	GOROOT/src/runtime/panic.go:889 +0x27b fp=0xc001d60498 sp=0xc001d60418 pc=0x489dbb
panic({0x46f6f80, 0xc003530960})
	GOROOT/src/runtime/panic.go:1038 +0x215 fp=0xc001d60558 sp=0xc001d60498 pc=0x48a695
github.com/cockroachdb/cockroach/pkg/util/tracing.(*crdbSpan).recordFinishedChildrenLocked(0xc001f43558, {0xc001ef6900, 0x69708b8c99, 0xc001d60648})
	github.com/cockroachdb/cockroach/pkg/util/tracing/crdbspan.go:550 +0x325 fp=0xc001d605d8 sp=0xc001d60558 pc=0xbf3c25
github.com/cockroachdb/cockroach/pkg/util/tracing.(*crdbSpan).childFinished(0xc001f43558, 0xc001262e58)
	github.com/cockroachdb/cockroach/pkg/util/tracing/crdbspan.go:1069 +0x251 fp=0xc001d60658 sp=0xc001d605d8 pc=0xbf6bd1
github.com/cockroachdb/cockroach/pkg/util/tracing.(*crdbSpan).finish(0xc001262e58)
	github.com/cockroachdb/cockroach/pkg/util/tracing/crdbspan.go:279 +0x153 fp=0xc001d606e0 sp=0xc001d60658 pc=0xbf2513
github.com/cockroachdb/cockroach/pkg/util/tracing.(*spanInner).Finish(0xc001262e08)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span_inner.go:101 +0x45 fp=0xc001d60710 sp=0xc001d606e0 pc=0xbfa505
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).finishInternal(0xc001262e00)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:263 +0xb7 fp=0xc001d60738 sp=0xc001d60710 pc=0xbf7937
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).Finish(0xc001f43500)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:251 +0x19 fp=0xc001d60750 sp=0xc001d60738 pc=0xbf7859
github.com/cockroachdb/cockroach/pkg/jobs.(*Job).load·dwrap·12()
	github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:803 +0x26 fp=0xc001d60768 sp=0xc001d60750 pc=0x21735a6
github.com/cockroachdb/cockroach/pkg/jobs.(*Job).load(0xc00297b860, {0x6704458, 0xc004bb2b10}, 0xc0048e2480)
	github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:853 +0x271 fp=0xc001d60830 sp=0xc001d60768 pc=0x2172eb1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).LoadJobWithTxn(0xc000ada280, {0x6704458, 0xc004bb2b10}, 0xac05ce538c38001, 0x3ff8000000000000)
	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:647 +0x7f fp=0xc001d60870 sp=0xc001d60830 pc=0x217b63f
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).LoadJob(...)
	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:620
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).waitForJobs(0xc000ada280, {0x6704458, 0xc004bb2b10}, {0x67a4998, 0xc000f02ba0}, {0xc00505c4f0, 0x1, 0x1}, 0xc003c62b40)
	github.com/cockroachdb/cockroach/pkg/jobs/wait.go:121 +0x5d3 fp=0xc001d60cc0 sp=0xc001d60870 pc=0x218be73
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Run(0xc002395c00, {0x6704458, 0xc004bb2b10}, {0x67a4998, 0xc000f02ba0}, {0xc00505c4f0, 0x0, 0x1})
	github.com/cockroachdb/cockroach/pkg/jobs/wait.go:174 +0x14e fp=0xc001d60d50 sp=0xc001d60cc0 pc=0x218cbce
github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).runMigration(0xc0027e1100, {0x6704458, 0xc004bb2b10}, {{0x4fabac1, 0xc004bb2b10}}, {{0x2, 0x0, 0x1, 0x0}})
	github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:304 +0x154 fp=0xc001d60dd0 sp=0xc001d60d50 pc=0x38166d4
github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).Migrate(0xc0027e1100, {0x67043b0, 0xc0048494c0}, {{0x4fabac1, 0xc004bb02d0}}, {{0xa, 0x0, 0x10, 0x0}}, {{0x16, ...}}, ...)
	github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:137 +0x505 fp=0xc001d60f08 sp=0xc001d60dd0 pc=0x3815c85
github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).Migrate-fm({0x67043b0, 0xc0048494c0}, {{0x4fabac1, 0x9f26c40}}, {{0x4bb2ae0, 0xc0, 0x0, 0x0}}, {{0x16, 0x1, ...}}, ...)
	github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:99 +0x8b fp=0xc001d60f68 sp=0xc001d60f08 pc=0x38c924b
github.com/cockroachdb/cockroach/pkg/sql.runMigrationsAndUpgradeVersion({0x67043b0, 0xc0048494c0}, 0xc0004e2480, {{0x4fabac1, 0x0}}, {0x67d04a8, 0xc004a57540}, {0x67d04a8, 0xc004975b30}, 0xc004bb2ae0)
	github.com/cockroachdb/cockroach/pkg/sql/set_cluster_setting.go:585 +0x156 fp=0xc001d60fe8 sp=0xc001d60f68 pc=0x3369dd6
github.com/cockroachdb/cockroach/pkg/sql.setVersionSetting({0x67043b0, 0xc0048494c0}, 0xc0004e2480, 0xc000104a50, {0x513000f, 0xc0007160c0}, 0x0, {{0x4fabac1, 0x0}}, 0xc000e0a000, ...)
	github.com/cockroachdb/cockroach/pkg/sql/set_cluster_setting.go:521 +0x505 fp=0xc001d61170 sp=0xc001d60fe8 pc=0x3369025
github.com/cockroachdb/cockroach/pkg/sql.writeNonDefaultSettingValue({0x67043b0, 0xc0048494c0}, 0xc0004e2480, {0x67a9e08, 0xc000104a50}, {0x513000f, 0x7}, 0xc00451fad0, {{0x4fabac1, 0x4}}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/set_cluster_setting.go:384 +0x1db fp=0xc001d612e0 sp=0xc001d61170 pc=0x336873b
github.com/cockroachdb/cockroach/pkg/sql.writeSettingInternal.func1({0x67043b0, 0xc0048494c0}, 0x6782f68)
	github.com/cockroachdb/cockroach/pkg/sql/set_cluster_setting.go:320 +0x169 fp=0xc001d613d0 sp=0xc001d612e0 pc=0x3368189
github.com/cockroachdb/cockroach/pkg/kv.runTxn.func1({0x67043b0, 0xc0048494c0}, 0x4c1af3)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:966 +0x27 fp=0xc001d613f8 sp=0xc001d613d0 pc=0x12f1887
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec(0xc004531c20, {0x67043b0, 0xc0048494c0}, 0xc001d61738)
	github.com/cockroachdb/cockroach/pkg/kv/txn.go:959 +0xae fp=0xc001d616e0 sp=0xc001d613f8 pc=0x12f9ece
github.com/cockroachdb/cockroach/pkg/kv.runTxn({0x67043b0, 0xc0048494c0}, 0xc000714bd0, 0xddbb10)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:965 +0x5a fp=0xc001d61758 sp=0xc001d616e0 pc=0x12f179a
github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl(0x4ecca9, {0x67043b0, 0xc0048494c0}, 0x9f808d8, 0x0, 0xc000ddbad0)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:928 +0x89 fp=0xc001d617a0 sp=0xc001d61758 pc=0x12f1529
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn(...)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:907
github.com/cockroachdb/cockroach/pkg/sql.writeSettingInternal({0x67043b0, 0xc0048494c0}, 0x10, {0x67a9e08, 0xc000104a50}, {0x513000f, 0x45df95}, {{0x4fabac1, 0x49eb2e0}}, 0xc000e0a000, ...)
	github.com/cockroachdb/cockroach/pkg/sql/set_cluster_setting.go:305 +0x145 fp=0xc001d61860 sp=0xc001d617a0 pc=0x3367f85
github.com/cockroachdb/cockroach/pkg/sql.(*setClusterSettingNode).startExec(0xc0048491c0, {{0x67043b0, 0xc0048494c0}, 0xc004a1a380, 0xc004936d08})
	github.com/cockroachdb/cockroach/pkg/sql/set_cluster_setting.go:215 +0x3df fp=0xc001d61990 sp=0xc001d61860 pc=0x336765f
github.com/cockroachdb/cockroach/pkg/sql.startExec.func2({0xc004a29a80, 0x4625b0}, {0x67068a8, 0xc0048491c0})
	github.com/cockroachdb/cockroach/pkg/sql/plan.go:515 +0x147 fp=0xc001d61a40 sp=0xc001d61990 pc=0x3302887
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1()
	github.com/cockroachdb/cockroach/pkg/sql/walk.go:112 +0x3e fp=0xc001d61a78 sp=0xc001d61a40 pc=0x33bd75e
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal(0xc004a29b90, {0x67068a8, 0xc0048491c0}, {0x4fee9c9, 0x13})
	github.com/cockroachdb/cockroach/pkg/sql/walk.go:297 +0x14c9 fp=0xc001d61b08 sp=0xc001d61a78 pc=0x33bd669
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit(0xc004a29b90, {0x67068a8, 0xc0048491c0})
	github.com/cockroachdb/cockroach/pkg/sql/walk.go:79 +0xf7 fp=0xc001d61b58 sp=0xc001d61b08 pc=0x33bc097
github.com/cockroachdb/cockroach/pkg/sql.walkPlan(...)
	github.com/cockroachdb/cockroach/pkg/sql/walk.go:43
github.com/cockroachdb/cockroach/pkg/sql.startExec({{0x67043b0, 0xc0048494c0}, 0xc004a1a380, 0xc004936d08}, {0x67068a8, 0xc0048491c0})
	github.com/cockroachdb/cockroach/pkg/sql/plan.go:518 +0x125 fp=0xc001d61bd8 sp=0xc001d61b58 pc=0x33026e5
github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start(0xc004a20000, {0x67043b0, 0xc0048494c0})
	github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:147 +0x87 fp=0xc001d61c18 sp=0xc001d61bd8 pc=0x33057c7
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Init(0xc004a10c00, {0x67043b0, 0xc0048494c0})
	github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:159 +0xd4 fp=0xc001d61c58 sp=0xc001d61c18 pc=0x2800c94
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Init(0xc004a22380, {0x67043b0, 0xc0048494c0})
	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:90 +0x37 fp=0xc001d61c88 sp=0xc001d61c58 pc=0x3101577
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedStatsCollectorImpl).Init(0xc004a22380, {0x67043b0, 0xc0048494c0})
	<autogenerated>:1 +0x33 fp=0xc001d61cb0 sp=0xc001d61c88 pc=0x310cfb3
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Start.func1()
	github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:250 +0x34 fp=0xc001d61cd8 sp=0xc001d61cb0 pc=0x2803ff4
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError(0x0)
	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91 +0x62 fp=0xc001d61d18 sp=0xc001d61cd8 pc=0x1ffd302
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Start(0xc00452b680, {0x67043b0, 0xc0048494c0})
	github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:249 +0x78 fp=0xc001d61d88 sp=0xc001d61d18 pc=0x2803f38
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Start.func1()
	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:122 +0x34 fp=0xc001d61db0 sp=0xc001d61d88 pc=0x30fd734
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError(0x0)
	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91 +0x62 fp=0xc001d61df0 sp=0xc001d61db0 pc=0x1ffd302
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Start(0xc00175e360, {0x67043b0, 0xc0048494c0})
	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:121 +0x6b fp=0xc001d61e50 sp=0xc001d61df0 pc=0x30fd6ab
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run(0xc00175e360, {0x67043b0, 0xc0048494c0})
	github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:728 +0x3a fp=0xc001d61e90 sp=0xc001d61e50 pc=0x21d18ba
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Run(0xc00175e360, {0x67043b0, 0xc0048494c0})
	<autogenerated>:1 +0x33 fp=0xc001d61eb8 sp=0xc001d61e90 pc=0x310ad13
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run(0xc00452b4a0, {0x67043b0, 0xc0048494c0}, 0xc00175e360)
	github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:475 +0x258 fp=0xc001d61f78 sp=0xc001d61eb8 pc=0x26b2d18
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run(0xc0048ddb30, {0x67043b0, 0xc0048494c0}, 0x1)
	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:249 +0x205 fp=0xc001d62018 sp=0xc001d61f78 pc=0x3102b65
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc0025bea00, {0x6704458, 0xc0048ddad0}, 0xc00493bce0, 0xc004531b80, 0xc004940b80, 0xc002de0000, 0xc004936fc8, 0x0)
	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:607 +0xb04 fp=0xc001d62768 sp=0xc001d62018 pc=0x327c884
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0x6704458, {0x6704458, 0xc0048ddad0}, 0x1, 0xc00493bce0, 0xc0048dd860, {{0x67068a8, 0xc0048491c0}, 0x0}, 0xc002de0000)
	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1455 +0x1b2 fp=0xc001d62820 sp=0xc001d62768 pc=0x3280672
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0xc004936600, {0x6704458, 0xc0048ddad0}, 0xc004936d08, 0xc0048ddad0, {0x67ac818, 0xc0048dd860}, 0x45, 0xc004977c18)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1485 +0x614 fp=0xc001d62998 sp=0xc001d62820 pc=0x320a614
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc004936600, {0x6704458, 0xc0048dd9b0}, 0xc004936d08, {0x67ac818, 0xc0048dd860})
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1159 +0xb87 fp=0xc001d62d98 sp=0xc001d62998 pc=0x32081a7
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc004936600, {0x67043b0, 0xc004849140}, {{0x6734e78, 0xc00497bda0}, {0x512fffb, 0x45}, 0x0, 0x0}, 0x0, ...)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:690 +0x2091 fp=0xc001d63670 sp=0xc001d62d98 pc=0x3204071
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc004936600, {0x67043b0, 0xc004849140}, {{0x6734e78, 0xc00497bda0}, {0x512fffb, 0x45}, 0x0, 0x0}, 0x0, ...)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:145 +0x59e fp=0xc001d638e0 sp=0xc001d63670 pc=0x320113e
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1({{{0x6734e78, 0xc00497bda0}, {0x512fffb, 0x45}, 0x0, 0x0}, {0xc0a7264a74b61db1, 0x1c5707f19, 0x0}, {0xc0a7264a74b61db1, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1892 +0x2f6 fp=0xc001d63998 sp=0xc001d638e0 pc=0x31f6ab6
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc004936600)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1896 +0xb48 fp=0xc001d63e60 sp=0xc001d63998 pc=0x31f5048
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc004936600, {0x6704458, 0xc0048dd470}, 0x0, {0x0, 0x0, 0x0, 0x0, 0x0}, 0x0)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1818 +0x26c fp=0xc001d63f30 sp=0xc001d63e60 pc=0x31f412c
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx.func1()
	github.com/cockroachdb/cockroach/pkg/sql/internal.go:206 +0xa5 fp=0xc001d63fe0 sp=0xc001d63f30 pc=0x32cbbc5
runtime.goexit()
	GOROOT/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc001d63fe8 sp=0xc001d63fe0 pc=0x4bf321
created by github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx
	github.com/cockroachdb/cockroach/pkg/sql/internal.go:205 +0x5f1

@blathers-crl blathers-crl bot added the T-jobs label Jun 29, 2022
@stevendanna
Copy link
Collaborator

stevendanna commented Jun 29, 2022 •
edited
Loading

Similar panic from https://teamcityartifacts.crdb.io/presignedTokenAuth/6d4ebdcf-2092-4104-8d47-d9520028793d/repository/download/Cockroach_Nightlies_RoachtestNightlyGceBazel/5582417:id/follower-reads/mixed-version/single-region/run_1/artifacts.zip!/logs/2.unredacted/cockroach-stderr.log with a different stack:

goroutine 720 [running]:
panic({0x46db960, 0xc00321ca80})
	GOROOT/src/runtime/panic.go:1147 +0x3a8 fp=0xc0011f7650 sp=0xc0011f7590 pc=0x48a828
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1()
	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:58 +0x3bb fp=0xc0011f77e8 sp=0xc0011f7650 pc=0x1fb44bb
runtime.deferCallSave(0xc0011f78f0, 0xc0011f9dc8)
	GOROOT/src/runtime/panic.go:950 +0x82 fp=0xc0011f77f8 sp=0xc0011f77e8 pc=0x48a422
runtime.runOpenDeferFrame(0xc0017c3860, 0xc0017c38b0)
	GOROOT/src/runtime/panic.go:889 +0x27b fp=0xc0011f7878 sp=0xc0011f77f8 pc=0x489dbb
panic({0x46db960, 0xc00321ca80})
	GOROOT/src/runtime/panic.go:1038 +0x215 fp=0xc0011f7938 sp=0xc0011f7878 pc=0x48a695
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1()
	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:58 +0x3bb fp=0xc0011f7ad0 sp=0xc0011f7938 pc=0x1fb44bb
runtime.deferCallSave(0xc0011f7bd8, 0xc0011f9cf0)
	GOROOT/src/runtime/panic.go:950 +0x82 fp=0xc0011f7ae0 sp=0xc0011f7ad0 pc=0x48a422
runtime.runOpenDeferFrame(0xc0017c38b0, 0xc0017c3860)
	GOROOT/src/runtime/panic.go:889 +0x27b fp=0xc0011f7b60 sp=0xc0011f7ae0 pc=0x489dbb
panic({0x46db960, 0xc00321ca80})
	GOROOT/src/runtime/panic.go:1038 +0x215 fp=0xc0011f7c20 sp=0xc0011f7b60 pc=0x48a695
github.com/cockroachdb/cockroach/pkg/util/tracing.(*crdbSpan).recordFinishedChildrenLocked(0xc002779c58, {0xc005663440, 0x6c0abcf41e, 0xc0011f7d10})
	github.com/cockroachdb/cockroach/pkg/util/tracing/crdbspan.go:550 +0x325 fp=0xc0011f7ca0 sp=0xc0011f7c20 pc=0xbf3aa5
github.com/cockroachdb/cockroach/pkg/util/tracing.(*crdbSpan).childFinished(0xc002779c58, 0xc0029c2058)
	github.com/cockroachdb/cockroach/pkg/util/tracing/crdbspan.go:1069 +0x251 fp=0xc0011f7d20 sp=0xc0011f7ca0 pc=0xbf6a51
github.com/cockroachdb/cockroach/pkg/util/tracing.(*crdbSpan).finish(0xc0029c2058)
	github.com/cockroachdb/cockroach/pkg/util/tracing/crdbspan.go:279 +0x153 fp=0xc0011f7da8 sp=0xc0011f7d20 pc=0xbf2393
github.com/cockroachdb/cockroach/pkg/util/tracing.(*spanInner).Finish(0xc0029c2008)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span_inner.go:101 +0x45 fp=0xc0011f7dd8 sp=0xc0011f7da8 pc=0xbfa385
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).finishInternal(0xc0029c2000)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:263 +0xb7 fp=0xc0011f7e00 sp=0xc0011f7dd8 pc=0xbf77b7
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).Finish(0x0)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:251 +0x19 fp=0xc0011f7e18 sp=0xc0011f7e00 pc=0xbf76d9
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send·dwrap·1()
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:771 +0x26 fp=0xc0011f7e30 sp=0xc0011f7e18 pc=0x1f31c46
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0, ...}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:867 +0x987 fp=0xc0011f8288 sp=0xc0011f7e30 pc=0x1f31807
github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0, ...}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/db.go:223 +0xd8 fp=0xc0011f83d8 sp=0xc0011f8288 pc=0x1811c58
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0, ...}, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:1002 +0x14d fp=0xc0011f8518 sp=0xc0011f83d8 pc=0x18164ed
github.com/cockroachdb/cockroach/pkg/kv.(*DB).send(...)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:985
github.com/cockroachdb/cockroach/pkg/kv.(*DB).send-fm({_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/db.go:982 +0x8c fp=0xc0011f8650 sp=0xc0011f8518 pc=0x18241ac
github.com/cockroachdb/cockroach/pkg/kv.sendAndFill({0x66d8b98, 0xc000d77740}, 0xc0005f1690, 0xc001dc5080)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:847 +0xfc fp=0xc0011f8878 sp=0xc0011f8650 pc=0x1815c7c
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run(0xc000971680, {0x66d8b98, 0xc000d77740}, 0x4cc46c0)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:870 +0x74 fp=0xc0011f88c0 sp=0xc0011f8878 pc=0x1815df4
github.com/cockroachdb/cockroach/pkg/kv.(*DB).scan(0xb, {0x66d8b98, 0xc000d77740}, {0x4cc46c0, 0xc00115a270}, {0x4cc46c0, 0xc00115a2e8}, 0x0, 0x90, 0x0, ...)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:475 +0xed fp=0xc0011f8a38 sp=0xc0011f88c0 pc=0x18133ed
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Scan(...)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:486
github.com/cockroachdb/cockroach/pkg/kv/kvserver/liveness.(*NodeLiveness).GetLivenessesFromKV(0xc000f5a2c0, {0x66d8b98, 0xc000d77740})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/liveness/liveness.go:1076 +0xd9 fp=0xc0011f8ba8 sp=0xc0011f8a38 pc=0x19b5b39
github.com/cockroachdb/cockroach/pkg/upgrade/upgradecluster.NodesFromNodeLiveness({0x66d8b98, 0xc000d77740}, {0x7f00fdf0c678, 0xc000f5a2c0})
	github.com/cockroachdb/cockroach/pkg/upgrade/upgradecluster/nodes.go:44 +0x5b fp=0xc0011f8c78 sp=0xc0011f8ba8 pc=0x37d0cbb
github.com/cockroachdb/cockroach/pkg/upgrade/upgradecluster.(*Cluster).UntilClusterStable(0xc00192a5d0, {0x66d8b98, 0xc000d77740}, 0xc004b1f400)
	github.com/cockroachdb/cockroach/pkg/upgrade/upgradecluster/cluster.go:73 +0x5e fp=0xc0011f8d18 sp=0xc0011f8c78 pc=0x37cfd1e
github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.forEveryNodeUntilClusterStable({0x66d8b98, 0xc000d77740}, {0xc005565620, 0x23}, {0x66f4ff8, 0xc00192a5d0}, 0xc00321c900)
	github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:284 +0x103 fp=0xc0011f8d58 sp=0xc0011f8d18 pc=0x37ddea3
github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.bumpClusterVersion({0x66d8b98, 0xc000d77740}, {0x66f4ff8, 0xc00192a5d0}, {{0x16, 0x1, 0x0, 0x15}})
	github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:255 +0x165 fp=0xc0011f8dd0 sp=0xc0011f8d58 pc=0x37dd9e5
github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).Migrate(0xc0015b5280, {0x66d8af0, 0xc002935300}, {{0x4f8cb21, 0xc002434430}}, {{0xa, 0x0, 0x10, 0x0}}, {{0x16, ...}}, ...)
	github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:221 +0x56f fp=0xc0011f8f08 sp=0xc0011f8dd0 pc=0x37dd6cf
github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).Migrate-fm({0x66d8af0, 0xc002935300}, {{0x4f8cb21, 0xc000536400}}, {{0xd77710, 0xc0, 0x0, 0x0}}, {{0x16, 0x1, ...}}, ...)
	github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:99 +0x8b fp=0xc0011f8f68 sp=0xc0011f8f08 pc=0x3890d0b
github.com/cockroachdb/cockroach/pkg/sql.runMigrationsAndUpgradeVersion({0x66d8af0, 0xc002935300}, 0xc000a48480, {{0x4f8cb21, 0x0}}, {0x67a4aa8, 0xc000837500}, {0x67a4aa8, 0xc000836390}, 0xc000d77710)
	github.com/cockroachdb/cockroach/pkg/sql/set_cluster_setting.go:585 +0x156 fp=0xc0011f8fe8 sp=0xc0011f8f68 pc=0x3332816
github.com/cockroachdb/cockroach/pkg/sql.setVersionSetting({0x66d8af0, 0xc002935300}, 0xc000a48480, 0xc0005885a0, {0x511052b, 0xc0003b5bc0}, 0x0, {{0x4f8cb21, 0x0}}, 0xc000d88000, ...)
	github.com/cockroachdb/cockroach/pkg/sql/set_cluster_setting.go:521 +0x505 fp=0xc0011f9170 sp=0xc0011f8fe8 pc=0x3331a65
github.com/cockroachdb/cockroach/pkg/sql.writeNonDefaultSettingValue({0x66d8af0, 0xc002935300}, 0xc000a48480, {0x677e278, 0xc0005885a0}, {0x511052b, 0x7}, 0xc002b990d0, {{0x4f8cb21, 0x4}}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/set_cluster_setting.go:384 +0x1db fp=0xc0011f92e0 sp=0xc0011f9170 pc=0x333117b
github.com/cockroachdb/cockroach/pkg/sql.writeSettingInternal.func1({0x66d8af0, 0xc002935300}, 0x67571d8)
	github.com/cockroachdb/cockroach/pkg/sql/set_cluster_setting.go:320 +0x169 fp=0xc0011f93d0 sp=0xc0011f92e0 pc=0x3330bc9
github.com/cockroachdb/cockroach/pkg/kv.runTxn.func1({0x66d8af0, 0xc002935300}, 0x4c1af3)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:966 +0x27 fp=0xc0011f93f8 sp=0xc0011f93d0 pc=0x1816367
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec(0xc002d6a0a0, {0x66d8af0, 0xc002935300}, 0xc0011f9738)
	github.com/cockroachdb/cockroach/pkg/kv/txn.go:959 +0xae fp=0xc0011f96e0 sp=0xc0011f93f8 pc=0x181e9ae
github.com/cockroachdb/cockroach/pkg/kv.runTxn({0x66d8af0, 0xc002935300}, 0xc000971680, 0x17e8790)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:965 +0x5a fp=0xc0011f9758 sp=0xc0011f96e0 pc=0x181627a
github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl(0x4ecca9, {0x66d8af0, 0xc002935300}, 0x9f47918, 0x0, 0xc0017e8750)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:928 +0x89 fp=0xc0011f97a0 sp=0xc0011f9758 pc=0x1816009
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn(...)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:907
github.com/cockroachdb/cockroach/pkg/sql.writeSettingInternal({0x66d8af0, 0xc002935300}, 0x10, {0x677e278, 0xc0005885a0}, {0x511052b, 0x45df95}, {{0x4f8cb21, 0x49ce860}}, 0xc000d88000, ...)
	github.com/cockroachdb/cockroach/pkg/sql/set_cluster_setting.go:305 +0x145 fp=0xc0011f9860 sp=0xc0011f97a0 pc=0x33309c5
github.com/cockroachdb/cockroach/pkg/sql.(*setClusterSettingNode).startExec(0xc002934fc0, {{0x66d8af0, 0xc002935300}, 0xc002d39880, 0xc0015f3a08})
	github.com/cockroachdb/cockroach/pkg/sql/set_cluster_setting.go:215 +0x3df fp=0xc0011f9990 sp=0xc0011f9860 pc=0x333009f
github.com/cockroachdb/cockroach/pkg/sql.startExec.func2({0xc00130fa80, 0x4625b0}, {0x66db020, 0xc002934fc0})
	github.com/cockroachdb/cockroach/pkg/sql/plan.go:515 +0x147 fp=0xc0011f9a40 sp=0xc0011f9990 pc=0x32cb2c7
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1()
	github.com/cockroachdb/cockroach/pkg/sql/walk.go:112 +0x3e fp=0xc0011f9a78 sp=0xc0011f9a40 pc=0x33857fe
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal(0xc00130fb90, {0x66db020, 0xc002934fc0}, {0x4fcf9c0, 0x13})
	github.com/cockroachdb/cockroach/pkg/sql/walk.go:297 +0x14c9 fp=0xc0011f9b08 sp=0xc0011f9a78 pc=0x3385709
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit(0xc00130fb90, {0x66db020, 0xc002934fc0})
	github.com/cockroachdb/cockroach/pkg/sql/walk.go:79 +0xf7 fp=0xc0011f9b58 sp=0xc0011f9b08 pc=0x3384137
github.com/cockroachdb/cockroach/pkg/sql.walkPlan(...)
	github.com/cockroachdb/cockroach/pkg/sql/walk.go:43
github.com/cockroachdb/cockroach/pkg/sql.startExec({{0x66d8af0, 0xc002935300}, 0xc002d39880, 0xc0015f3a08}, {0x66db020, 0xc002934fc0})
	github.com/cockroachdb/cockroach/pkg/sql/plan.go:518 +0x125 fp=0xc0011f9bd8 sp=0xc0011f9b58 pc=0x32cb125
github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start(0xc0028c0000, {0x66d8af0, 0xc002935300})
	github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:147 +0x87 fp=0xc0011f9c18 sp=0xc0011f9bd8 pc=0x32ce207
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Init(0xc00241b400, {0x66d8af0, 0xc002935300})
	github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:159 +0xd4 fp=0xc0011f9c58 sp=0xc0011f9c18 pc=0x27cbcb4
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Init(0xc002a088c0, {0x66d8af0, 0xc002935300})
	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:90 +0x37 fp=0xc0011f9c88 sp=0xc0011f9c58 pc=0x30cc577
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedStatsCollectorImpl).Init(0xc002a088c0, {0x66d8af0, 0xc002935300})
	<autogenerated>:1 +0x33 fp=0xc0011f9cb0 sp=0xc0011f9c88 pc=0x30d7fb3
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Start.func1()
	github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:250 +0x34 fp=0xc0011f9cd8 sp=0xc0011f9cb0 pc=0x27cf014
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError(0x0)
	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91 +0x62 fp=0xc0011f9d18 sp=0xc0011f9cd8 pc=0x1fb40a2
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Start(0xc000b23860, {0x66d8af0, 0xc002935300})
	github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:249 +0x78 fp=0xc0011f9d88 sp=0xc0011f9d18 pc=0x27cef58
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Start.func1()
	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:122 +0x34 fp=0xc0011f9db0 sp=0xc0011f9d88 pc=0x30c8734
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError(0x0)
	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91 +0x62 fp=0xc0011f9df0 sp=0xc0011f9db0 pc=0x1fb40a2
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Start(0xc003d18fc0, {0x66d8af0, 0xc002935300})
	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:121 +0x6b fp=0xc0011f9e50 sp=0xc0011f9df0 pc=0x30c86ab
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run(0xc003d18fc0, {0x66d8af0, 0xc002935300})
	github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:728 +0x3a fp=0xc0011f9e90 sp=0xc0011f9e50 pc=0x21a021a
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Run(0xc003d18fc0, {0x66d8af0, 0xc002935300})
	<autogenerated>:1 +0x33 fp=0xc0011f9eb8 sp=0xc0011f9e90 pc=0x30d5d13
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run(0xc000b23680, {0x66d8af0, 0xc002935300}, 0xc003d18fc0)
	github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:475 +0x258 fp=0xc0011f9f78 sp=0xc0011f9eb8 pc=0x267dd38
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run(0xc000d0a780, {0x66d8af0, 0xc002935300}, 0x1)
	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:249 +0x205 fp=0xc0011fa018 sp=0xc0011f9f78 pc=0x30cdb65
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc0012412c0, {0x66d8b98, 0xc000d0a720}, 0xc002991ea0, 0xc002d6a000, 0xc00056a200, 0xc003b49880, 0xc0015f3cc8, 0x0)
	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:607 +0xb04 fp=0xc0011fa768 sp=0xc0011fa018 pc=0x3245284
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0x66d8b98, {0x66d8b98, 0xc000d0a720}, 0x1, 0xc002991ea0, 0xc000d0a4b0, {{0x66db020, 0xc002934fc0}, 0x0}, 0xc003b49880)
	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1455 +0x1b2 fp=0xc0011fa820 sp=0xc0011fa768 pc=0x3249072
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0xc0015f3300, {0x66d8b98, 0xc000d0a720}, 0xc0015f3a08, 0xc000d0a720, {0x6780c98, 0xc000d0a4b0}, 0x45, 0xc002681c78)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1485 +0x614 fp=0xc0011fa998 sp=0xc0011fa820 pc=0x31d3154
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc0015f3300, {0x66d8b98, 0xc000d0a600}, 0xc0015f3a08, {0x6780c98, 0xc000d0a4b0})
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1159 +0xb87 fp=0xc0011fad98 sp=0xc0011fa998 pc=0x31d0ce7
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc0015f3300, {0x66d8af0, 0xc002934f00}, {{0x67095f8, 0xc002e37ba0}, {0x5110517, 0x45}, 0x0, 0x0}, 0x0, ...)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:690 +0x2091 fp=0xc0011fb670 sp=0xc0011fad98 pc=0x31ccbb1
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc0015f3300, {0x66d8af0, 0xc002934f00}, {{0x67095f8, 0xc002e37ba0}, {0x5110517, 0x45}, 0x0, 0x0}, 0x0, ...)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:145 +0x59e fp=0xc0011fb8e0 sp=0xc0011fb670 pc=0x31c9c7e
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1({{{0x67095f8, 0xc002e37ba0}, {0x5110517, 0x45}, 0x0, 0x0}, {0xc0a6d1ceed988367, 0x1cde6fe35, 0x0}, {0xc0a6d1ceed988367, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1882 +0x2f6 fp=0xc0011fb998 sp=0xc0011fb8e0 pc=0x31bf696
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc0015f3300)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1886 +0xb48 fp=0xc0011fbe60 sp=0xc0011fb998 pc=0x31bdc28
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc0015f3300, {0x66d8b98, 0xc000d0a0f0}, 0x0, {0x0, 0x0, 0x0, 0x0, 0x0}, 0x0)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1808 +0x26c fp=0xc0011fbf30 sp=0xc0011fbe60 pc=0x31bccec
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx.func1()
	github.com/cockroachdb/cockroach/pkg/sql/internal.go:206 +0xa5 fp=0xc0011fbfe0 sp=0xc0011fbf30 pc=0x3294605
runtime.goexit()
	GOROOT/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0011fbfe8 sp=0xc0011fbfe0 pc=0x4bf321
created by github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx
	github.com/cockroachdb/cockroach/pkg/sql/internal.go:205 +0x5f1

@stevendanna
Copy link
Collaborator

stevendanna commented Jun 29, 2022 •
edited
Loading

The assertion in question was added very recently: #81079

@stevendanna
Copy link
Collaborator

Looks like it is a bug in tracing. When we don't have room for all of the spans in a record, we fallthrough to an assertion that is intended for StructuredRecordings rather than verbose recording. Will put together a fix once I have a unit test for this.

@stevendanna stevendanna mentioned this issue Jun 29, 2022
Merged
@cockroach-teamcity
Copy link
Member Author

roachtest.follower-reads/mixed-version/single-region failed with artifacts on master @ e1e99da678e29db4ef21813423f012273c99add2:

test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/run_1
	follower_reads.go:271,follower_reads.go:901,follower_reads.go:102,test_runner.go:896: failed to get follower read counts: Get "http://35.196.129.160:26258/_status/vars": dial tcp 35.196.129.160:26258: connect: connection refused

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=2 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@craig craig bot closed this as completed in 78caa5c Jul 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stevendanna stevendanna

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-jobs T-kv KV Team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

tracing: remove panic in Finish stevendanna/cockroach
3 participants
@stevendanna @tbg @cockroach-teamcity