Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: debug zip cpu profile collection fails #91203

Closed
cockroach-teamcity opened this issue Nov 3, 2022 · 14 comments
Closed

roachtest: debug zip cpu profile collection fails #91203

cockroach-teamcity opened this issue Nov 3, 2022 · 14 comments
Assignees
@michae2
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-sql-queries SQL Queries Team
Milestone
22.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Nov 3, 2022 •
edited by cockroach-jira-scripts
Loading

roachtest.unoptimized-query-oracle/disable-rules=half/seed-multi-region failed with artifacts on master @ 9c9d55d707ad9a768027e9b7a3775c7c7cde8de7:

test artifacts and logs in: /artifacts/unoptimized-query-oracle/disable-rules=half/seed-multi-region/run_1
(test_impl.go:314).Errorf: test timed out (1h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-21150
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Nov 3, 2022
@cockroach-teamcity cockroach-teamcity added this to the 22.2 milestone Nov 3, 2022
@blathers-crl blathers-crl bot added the T-sql-queries SQL Queries Team label Nov 3, 2022
@cockroach-teamcity
Copy link
Member Author

roachtest.unoptimized-query-oracle/disable-rules=half/seed-multi-region failed with artifacts on master @ 4c828ca5df8ea70b92ba58c6cb4868303cee1262:

test artifacts and logs in: /artifacts/unoptimized-query-oracle/disable-rules=half/seed-multi-region/run_1
(test_impl.go:314).Errorf: test timed out (1h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 7, 2022
Closed
@cucaroach
Copy link
Contributor

The 11/3 failure had this query in its log:

I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹goroutine 1573 [select]:›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/util/retry.(*Retry).Next(0xc006009308)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:128 +0x13e›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*planner).waitForDescriptorSchemaChanges(0xc001b1dde8, {0x67e57e8, 0xc001efc9c0}, 0x68, {0x1, {{{0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...}, ...}, ...})›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/schema_change_plan_node.go:126 +0x325›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).handleWaitingForConcurrentSchemaChanges(0xc001b1d900, {0x67e57e8, 0xc001efc9c0}, 0x49fb918?)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:3057 +0xc5›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).txnStateTransitionsApplyWrapper(0xc001b1d900, {0x67b1fc0?, 0xc0066c6f00?}, {0x4e25f40, 0xc0066f65a0}, {0x7f11b9d8dfb8?, 0xc000818f08?}, 0x0?)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2931 +0x2e5›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc001b1d900)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2111 +0x1478›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc001b1d900, {0x67e5740, 0xc005d74ec0}, 0x411252dc00000000?, 0xc0078e9dd8?, 0xc0072a1a40?)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1852 +0x20b›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc00439d7c0?, {0x67e5740?, 0xc005d74ec0?}, {0xc0011ef800?}, 0x3?, 0xc00770f488?)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:830 +0xe6›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1()›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:728 +0x40d›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:639 +0x22e›

Looks like we finished a query but there was some schema work to wait on? Although there was another pgwire conn_executor running so not sure which one caused the timeout, the other one was here:

I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/util/duration.Duration.encodePostgres({0xc007e11300?, 0x4a9a89?, 0xa?}, 0x28?)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/util/duration/pkg/util/duration/duration.go:416 +0x12e›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/util/duration.Duration.FormatWithStyle({0xa?, 0x1?, 0xc007e11340?}, 0x139ccc5?, 0x1d3e3e0?)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/util/duration/pkg/util/duration/duration.go:387 +0x3f›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql/sem/tree.FormatDuration({0x0?, 0xfffffffffffffc28?, 0xffffe59d6f8f93e0?}, 0xc0065f60b0)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/sem/tree/datum.go:3239 +0x65›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*DInterval).Format(0x5efe5efe5efe5efe?, 0x5efe5efe5efe5efe?)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/sem/tree/datum.go:3247 +0x2d›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*FmtCtx).formatNodeOrHideConstants(0xc007e11478?, {0x67a6fe0?, 0xc00a230c00?})›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/sem/tree/hide_constants.go:49 +0x1a4›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*FmtCtx).FormatNode(0xc0065f60b0, {0x67a6fe0, 0xc00a230c00})›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/sem/tree/format.go:452 +0xd5›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql/pgwire.writeTextDatumNotNull(0xc0049bd8c8, {0x6847078?, 0xc00a230c00?}, {0x1e87950?, 0xc0?, 0x0?, {0x0?, 0x578e15?}}, 0x578e15?, 0xc0055ca540)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/pgwire/types.go:236 +0x63f›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*writeBuffer).writeTextDatum(0x111309a?, {0x67e57e8, 0xc005b4aae0}, {0x6847078?, 0xc00a230c00}, {0x0, 0x2, 0x0, {0x0, 0x0}}, ...)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/pgwire/types.go:146 +0x189›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).bufferRow(0xc0049bd000, {0x67e57e8, 0xc005b4aae0}, {0xc006c32360, 0x9, 0xe?}, 0xc0049bd708)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:1353 +0x230›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*commandResult).AddRow(0xc0049bd708, {0x67e57e8, 0xc005b4aae0}, {0xc006c32360, 0x9, 0x9})›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/pgwire/command_result.go:221 +0x73›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLReceiver).Push(0xc007670a80, {0xc00a9606d0?, 0x9, 0xc006739c00?}, 0xc00756c000?)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1212 +0x389›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*copyingRowReceiver).Push(0xc0064edc00?, {0xc006739c00?, 0x9?, 0x9?}, 0x0?)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/rowflow/row_based_flow.go:462 +0x18b›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql/execinfra.Run({0x67e57e8, 0xc005b4b020}, {0x680b000, 0xc006ab82c0}, {0x67bd990?, 0xc0064edc00?})›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/execinfra/base.go:189 +0x86›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run(0xc006ab82c0, {0x67e5740?, 0xc00192ac40?})›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:733 +0x5b›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run(0xc005800960, {0x67e5740, 0xc00192ac40})›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:467 +0x254›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc00441c000, {0x67e57e8, 0xc005b4aae0}, 0xc0044287e0, 0xc009944000, 0xc007b39400, 0xc007670a80, 0xc0054ebb40, 0x0)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:688 +0xa78›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0xc005b4ab10?, {0x67e57e8, 0xc005b4aae0}, 0x0?, 0xc0044287e0, 0x521ede0?, {{0x67e7ca8?, 0xc00192c640?}, 0x0?}, 0xc007670a80)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1555 +0x1f9›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll(0xc0044287e0?, {0x67e57e8, 0xc005b4aae0}, 0xc0054eb7e8?, 0x1?, 0xc0054eb7e8, 0xc007670a80, 0x3?)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1307 +0x206›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0xc0054eb300, {0x67e57e8, 0xc005b4aae0}, 0xc0054eb7e8, 0xc005b4aae0?, {0x7f11b9d8dff0?, 0xc0049bd708}, 0x0?, 0xc009944148)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1576 +0x4c8›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc0054eb300, {0x67e57e8, 0xc001a8d050}, 0xc0054eb7e8, {0x7f11b9d8dff0, 0xc0049bd708})›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1216 +0xb99›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc0054eb300, {0x67e57e8?, 0xc001a8ced0?}, {{0x680e540, 0xc0067b2960}, {0x0, 0x0, 0x0}, {0xc0052b9000, 0x238}, ...}, ...)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:725 +0x2534›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1({0x67e57e8?, 0xc001a8ced0?})›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:129 +0xbc›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling(0x67e57e8?, {0x67e57e8?, 0xc001a8ced0?}, {0x680e540?, 0xc0067b2960?}, 0x0?, 0x958e9c0?)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2421 +0x330›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc0054eb300, {0x67e57e8, 0xc001a8ced0}, {{0x680e540, 0xc0067b2960}, {0x0, 0x0, 0x0}, {0xc0052b9000, 0x238}, ...}, ...)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:128 +0x4da›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1(0xc008315ac0, 0xc0054eb300, 0xc0083157d8, 0xc008315890, 0xc0083158d0, 0xc007e138b0, 0xc007e138a0, 0xc007e138c0)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1927 +0x305›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc0054eb300)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1932 +0x1127›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc0054eb300, {0x67e5740, 0xc006bf6a00}, 0x4115280800000000?, 0xc0089c1dd8?, 0xc006c72220?)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1852 +0x20b›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc00439d7c0?, {0x67e5740?, 0xc006bf6a00?}, {0xc0011ef800?}, 0x3?, 0xc0094f4258?)›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:830 +0xe6›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1()›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:728 +0x40d›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync›
I221103 15:04:34.789903 12 util/log/clog.go:409 ⋮ [-] 10341 +‹	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:639 +0x22e›

Will analyze this to see if there's any missing cancel checking.

@cucaroach
Copy link
Contributor

The 11/7 failure had this query goroutine still running:

I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹goroutine 7624 [runnable]:›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql/sem/eval.(*evaluator).EvalComparisonExpr(0x45f43d?, {0x67f41e8, 0xc001ab0ab0}, 0xc001a32b40)›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/sem/eval/expr.go:264 +0x1e5›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*ComparisonExpr).Eval(0xc005005570?, {0x67f41e8?, 0xc001ab0ab0?}, {0x68828e0?, 0xc00748ec00?})›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/sem/tree/eval_expr_generated.go:121 +0x34›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql/sem/eval.(*evaluator).EvalAndExpr(0x45f43d?, {0x67f41e8, 0xc001ab0ab0}, 0xc001ab0210)›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/sem/eval/expr.go:40 +0x6b›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*AndExpr).Eval(0xc007bf5b00?, {0x67f41e8?, 0xc001ab0ab0?}, {0x68828e0?, 0xc00748ec00?})›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/sem/tree/eval_expr_generated.go:71 +0x34›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql/sem/eval.(*evaluator).EvalAndExpr(0x46159f?, {0x67f41e8, 0xc001ab0ab0}, 0xc001ab0270)›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/sem/eval/expr.go:40 +0x6b›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*AndExpr).Eval(0x0?, {0x67f41e8?, 0xc001ab0ab0?}, {0x68828e0?, 0xc00748ec00?})›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/sem/tree/eval_expr_generated.go:71 +0x34›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql/sem/eval.Expr(...)›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/sem/eval/expr.go:26›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql/execinfrapb.RunFilter({0x67f41e8?, 0xc001ab0ab0?}, {0x6825888?, 0xc001ab0270?}, 0x2569687?)›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/execinfrapb/pkg/sql/execinfrapb/expr.go:229 +0x48›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql/execinfrapb.(*ExprHelper).EvalFilter(0xc006642550, {0x67f41e8?, 0xc001ab0ab0?}, {0xc0050fa000?, 0xc00789cc00?, 0x20?})›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/execinfrapb/pkg/sql/execinfrapb/expr.go:218 +0x190›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*joinerBase).render(0xc006642300, {0xc00741f800?, 0x20, 0x20?}, {0xc00789cc00?, 0x20, 0x20?})›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/rowexec/joinerbase.go:173 +0x2bf›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*hashJoiner).probeRow(0xc006642300)›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/rowexec/hashjoiner.go:349 +0x2c9›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*hashJoiner).Next(0xc006642300)›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/rowexec/hashjoiner.go:189 +0x145›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql/execinfra.Run({0x67f41e8, 0xc001ab0ab0}, {0x68196c0, 0xc006642300}, {0x67cc330?, 0xc00729a700?})›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/execinfra/base.go:186 +0x55›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run(0xc006642300, {0x67f4140?, 0xc001c0ae00?})›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:733 +0x5b›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run(0xc0094643c0, {0x67f4140, 0xc001c0ae00})›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:467 +0x254›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc0003f8500, {0x67f41e8, 0xc001e0bd40}, 0xc001907ea0, 0xc002a47080, 0xc0071b0280, 0xc006ac9500, 0xc004b0db40, 0x0)›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:688 +0xa78›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0xc001e0be30?, {0x67f41e8, 0xc001e0bd40}, 0x0?, 0xc001907ea0, 0x0?, {{0x67f6638?, 0xc0081ee000?}, 0x0?}, 0xc006ac9500)›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1553 +0x1f9›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll(0xc001907ea0?, {0x67f41e8, 0xc001e0bd40}, 0xc004b0d7e8?, 0x1?, 0xc004b0d7e8, 0xc006ac9500, 0xc001a7ab40?)›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1305 +0x206›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0xc004b0d300, {0x67f41e8, 0xc001e0bd40}, 0xc004b0d7e8, 0xc001e0bd40?, {0x7f4b34296e58?, 0xc005fa9708}, 0x0?, 0xc002a471c8)›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1572 +0x4c8›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc004b0d300, {0x67f41e8, 0xc008677ec0}, 0xc004b0d7e8, {0x7f4b34296e58, 0xc005fa9708})›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1214 +0xb99›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc004b0d300, {0x67f41e8?, 0xc008677d70?}, {{0x681ce80, 0xc001cc18b0}, {0x0, 0x0, 0x0}, {0xc0072a161f, 0x2d5}, ...}, ...)›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:723 +0x2534›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1({0x67f41e8?, 0xc008677d70?})›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:127 +0xbc›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling(0x67f41e8?, {0x67f41e8?, 0xc008677d70?}, {0x681ce80?, 0xc001cc18b0?}, 0x0?, 0x95a2ac0?)›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2417 +0x330›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc004b0d300, {0x67f41e8, 0xc008677d70}, {{0x681ce80, 0xc001cc18b0}, {0x0, 0x0, 0x0}, {0xc0072a161f, 0x2d5}, ...}, ...)›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:126 +0x4da›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1(0xc0047e5ac0, 0xc004b0d300, 0xc0047e57d8, 0xc0047e5890, 0xc0047e58d0, 0xc0050078b0, 0xc0050078a0, 0xc0050078c0)›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1927 +0x305›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc004b0d300)›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1932 +0x1127›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc004b0d300, {0x67f4140, 0xc006dd3f00}, 0x41126ac000000000?, 0xc007c21dd8?, 0xc006954b80?)›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1852 +0x20b›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc002889720?, {0x67f4140?, 0xc006dd3f00?}, {0xc00101aa00?}, 0x3?, 0xc0066faa68?)›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:830 +0xe6›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1()›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:728 +0x40d›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync›
I221107 16:19:09.816727 23 util/log/clog.go:409 ⋮ [-] 889 +‹	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:639 +0x22e›

@rytaft rytaft removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Nov 8, 2022
@rytaft
Copy link
Collaborator

rytaft commented Nov 8, 2022

Not a release blocker

@cockroach-teamcity
Copy link
Member Author

roachtest.unoptimized-query-oracle/disable-rules=half/seed-multi-region failed with artifacts on master @ ee4fdb7fff683a7916ee585ac0c1e22d5d9a4412:

test artifacts and logs in: /artifacts/unoptimized-query-oracle/disable-rules=half/seed-multi-region/run_1
(test_impl.go:314).Errorf: test timed out (1h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@DrewKimball
Copy link
Collaborator

From the most recent failure:

W221109 16:15:02.341307 211 server/node.go:1006 ⋮ [n1,summaries] 1636  health alerts detected: {Alerts:[{StoreID:1 Category:‹METRICS› Description:‹ranges.underreplicated› Value:1}]}
I221109 16:15:11.547837 1585 sql/schema_change_plan_node.go:184 ⋮ [n1,client=34.148.219.238:58348,user=root] 1637  schema change waiting for concurrent schema changes on descriptor 104, waited 40m16.920592771s so far
W221109 16:15:12.339455 211 server/node.go:1006 ⋮ [n1,summaries] 1638  health alerts detected: {Alerts:[{StoreID:1 Category:‹METRICS› Description:‹ranges.underreplicated› Value:1}]}
W221109 16:15:12.801709 1409 2@rpc/clock_offset.go:223 ⋮ [n1,rnode=2,raddr=‹10.138.0.39:26257›,class=default,rpc] 1639  latency jump (prev avg 66.11ms, current 217.70ms)
W221109 16:15:22.339382 211 server/node.go:1006 ⋮ [n1,summaries] 1640  health alerts detected: {Alerts:[{StoreID:1 Category:‹METRICS› Description:‹ranges.underreplicated› Value:1}]}
W221109 16:15:32.339131 211 server/node.go:1006 ⋮ [n1,summaries] 1641  health alerts detected: {Alerts:[{StoreID:1 Category:‹METRICS› Description:‹ranges.underreplicated› Value:1}]}
I221109 16:15:42.323040 1585 sql/schema_change_plan_node.go:184 ⋮ [n1,client=34.148.219.238:58348,user=root] 1642  schema change waiting for concurrent schema changes on descriptor 104, waited 40m47.695797212s so far
W221109 16:15:42.340099 211 server/node.go:1006 ⋮ [n1,summaries] 1643  health alerts detected: {Alerts:[{StoreID:1 Category:‹METRICS› Description:‹ranges.underreplicated› Value:1}]}
W221109 16:15:52.341598 211 server/node.go:1006 ⋮ [n1,summaries] 1644  health alerts detected: {Alerts:[{StoreID:1 Category:‹METRICS› Description:‹ranges.underreplicated› Value:1}]}
W221109 16:15:52.802470 1409 2@rpc/clock_offset.go:223 ⋮ [n1,rnode=2,raddr=‹10.138.0.39:26257›,class=default,rpc] 1645  latency jump (prev avg 66.26ms, current 212.66ms)
W221109 16:16:02.341678 211 server/node.go:1006 ⋮ [n1,summaries] 1646  health alerts detected: {Alerts:[{StoreID:1 Category:‹METRICS› Description:‹ranges.underreplicated› Value:1}]}
W221109 16:16:12.338811 211 server/node.go:1006 ⋮ [n1,summaries] 1647  health alerts detected: {Alerts:[{StoreID:1 Category:‹METRICS› Description:‹ranges.underreplicated› Value:1}]}
I221109 16:16:12.368704 1585 sql/schema_change_plan_node.go:184 ⋮ [n1,client=34.148.219.238:58348,user=root] 1648  schema change waiting for concurrent schema changes on descriptor 104, waited 41m17.741461487s so far
W221109 16:16:22.339352 211 server/node.go:1006 ⋮ [n1,summaries] 1649  health alerts detected: {Alerts:[{StoreID:1 Category:‹METRICS› Description:‹ranges.underreplicated› Value:1}]}
W221109 16:16:32.341252 211 server/node.go:1006 ⋮ [n1,summaries] 1650  health alerts detected: {Alerts:[{StoreID:1 Category:‹METRICS› Description:‹ranges.underreplicated› Value:1}]}
W221109 16:16:32.802506 1409 2@rpc/clock_offset.go:223 ⋮ [n1,rnode=2,raddr=‹10.138.0.39:26257›,class=default,rpc] 1651  latency jump (prev avg 66.18ms, current 212.69ms)
W221109 16:16:42.339303 211 server/node.go:1006 ⋮ [n1,summaries] 1652  health alerts detected: {Alerts:[{StoreID:1 Category:‹METRICS› Description:‹ranges.underreplicated› Value:1}]}
I221109 16:16:43.000200 1585 sql/schema_change_plan_node.go:184 ⋮ [n1,client=34.148.219.238:58348,user=root] 1653  schema change waiting for concurrent schema changes on descriptor 104, waited 41m48.372956729s so far
I221109 16:16:44.879025 25 util/log/clog.go:409 ⋮ [-] 1654  SIGQUIT received. stack traces:
I221109 16:16:44.879025 25 util/log/clog.go:409 ⋮ [-] 1654 +‹goroutine 25 [running]:›
I221109 16:16:44.879025 25 util/log/clog.go:409 ⋮ [-] 1654 +‹github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x1)›
I221109 16:16:44.879025 25 util/log/clog.go:409 ⋮ [-] 1654 +‹	github.com/cockroachdb/cockroach/pkg/util/log/get_stacks.go:25 +0x89›
I221109 16:16:44.879025 25 util/log/clog.go:409 ⋮ [-] 1654 +‹github.com/cockroachdb/cockroach/pkg/util/log.DumpStacks({0x6818d98, 0xc000126000}, {0x536e6ec, 0x10})›
I221109 16:16:44.879025 25 util/log/clog.go:409 ⋮ [-] 1654 +‹	github.com/cockroachdb/cockroach/pkg/util/log/clog.go:407 +0x50›
I221109 16:16:44.879025 25 util/log/clog.go:409 ⋮ [-] 1654 +‹github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func1()›
I221109 16:16:44.879025 25 util/log/clog.go:409 ⋮ [-] 1654 +‹	github.com/cockroachdb/cockroach/pkg/cli/cli.go:336 +0x59›

@cockroach-teamcity
Copy link
Member Author

roachtest.unoptimized-query-oracle/disable-rules=half/seed-multi-region failed with artifacts on master @ d6ae9653b67071cca09ed296711cecc821fc284e:

test artifacts and logs in: /artifacts/unoptimized-query-oracle/disable-rules=half/seed-multi-region/run_1
(test_impl.go:291).Fatal: parallel execution failure: failed to upload start script: : context deadline exceeded

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.unoptimized-query-oracle/disable-rules=half/seed-multi-region failed with artifacts on master @ d136c1fa50fa507c2232d7b215d98b88a83649c5:

test artifacts and logs in: /artifacts/unoptimized-query-oracle/disable-rules=half/seed-multi-region/run_1
(test_impl.go:291).Fatal: parallel execution failure: ~ ./cockroach.sh
: context deadline exceeded

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 14, 2022
Closed
@DrewKimball
Copy link
Collaborator

Seeing rpc error: code = Unknown desc = a CPU profile is already in process, try again later in the debug zips - maybe the action item for now is to fix that.

@cockroach-teamcity
Copy link
Member Author

roachtest.unoptimized-query-oracle/disable-rules=half/seed-multi-region failed with artifacts on master @ ea7c52e5a930be203318db6e338edd5dde7537bd:

test artifacts and logs in: /artifacts/unoptimized-query-oracle/disable-rules=half/seed-multi-region/run_1
(test_impl.go:314).Errorf: test timed out (1h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.unoptimized-query-oracle/disable-rules=half/seed-multi-region failed with artifacts on master @ a035d2e6e7ea57b30115ecc8ee1a5d553a9e3412:

test artifacts and logs in: /artifacts/unoptimized-query-oracle/disable-rules=half/seed-multi-region/run_1
(test_impl.go:291).Fatal: parallel execution failure: failed to upload start script: : context deadline exceeded

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 18, 2022
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.unoptimized-query-oracle/disable-rules=half/seed-multi-region failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test artifacts and logs in: /artifacts/unoptimized-query-oracle/disable-rules=half/seed-multi-region/run_1
(test_impl.go:314).Errorf: test timed out (1h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@DrewKimball DrewKimball self-assigned this Nov 22, 2022
@DrewKimball DrewKimball mentioned this issue Nov 22, 2022
Closed
@DrewKimball
Copy link
Collaborator

I think the action item for now is to fix the CPU profile gathering - that should make these much easier to debug.

@DrewKimball DrewKimball changed the title roachtest: unoptimized-query-oracle/disable-rules=half/seed-multi-region failed roachtest: debug zip cpu profile collection fails Nov 22, 2022
@DrewKimball DrewKimball removed their assignment Nov 28, 2022
@michae2 michae2 self-assigned this Dec 8, 2022
@mgartner
Copy link
Collaborator

mgartner commented Jan 19, 2023 •
edited
Loading

I'm going to close this. The randomized test failures have quieted down a bit, and this may no longer be an issue.

@mgartner mgartner moved this to Done in SQL Queries Jul 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@michae2 michae2

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-sql-queries SQL Queries Team
Projects
SQL Queries
Archived in project
Milestone
22.2
Development

No branches or pull requests
6 participants
@mgartner @rytaft @cockroach-teamcity @michae2 @DrewKimball @cucaroach