Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed #123540

Closed
cockroach-teamcity opened this issue May 3, 2024 · 9 comments
Closed

ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed #123540

cockroach-teamcity opened this issue May 3, 2024 · 9 comments
Assignees
@msbutler
Labels
branch-release-24.1.0-rc Used to mark GA and release blockers and technical advisories for 24.1.0-rc C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-1 Issues/test failures with a fix SLA of 1 month release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 3, 2024 •
edited by cockroach-jira-scripts
Loading

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on release-24.1.0-rc @ 010b7b3b3f4ad23b2f0483d8d2b450961a51470c:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantStreamingJobRetryReset (14m48s)

Stack:

goroutine 383905 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 12931
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:238 +0x58
* 
* goroutine 12986 [chan receive, 4 minutes]:
* github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x7f343f8?, 0xc00506b5f0?})
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:452 +0x94
* runtime/pprof.Do({0x7f340a8?, 0xc2ff880?}, {{0xc000c91640?, 0x7ee87a0?, 0xc00abfd808?}}, 0xc00301d7a8)
* 	GOROOT/src/runtime/pprof/runtime.go:51 +0x9d
* github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x22ae17e?)
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:450 +0x52
* created by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 12931
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:442 +0xe5
* 
* goroutine 13030 [select, 4 minutes]:
* github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x7f343f8, 0xc009b81890})
* 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:246 +0xc5
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 12932
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
* goroutine 13029 [select, 4 minutes]:
* github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x7f343f8, 0xc009b81830})
* 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:246 +0xc5
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 12932
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
* goroutine 12966 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 12932
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f
* 
* goroutine 83170 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:243 +0xcb
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 13243
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:238 +0x58
* 
* goroutine 12964 [select]:
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSenderCircuitBreakers).gcLoop(0xc0062a9908, {0x7f343f8, 0xc00b2e55c0})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_circuit_breaker.go:345 +0x27e
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 12932
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
*

Parameters:

  • attempt=1
  • run=11
  • shard=4
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-38405
@cockroach-teamcity cockroach-teamcity added branch-release-24.1.0-rc Used to mark GA and release blockers and technical advisories for 24.1.0-rc C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery labels May 3, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone May 3, 2024
@cockroach-teamcity cockroach-teamcity mentioned this issue May 3, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on release-24.1.0-rc @ 010b7b3b3f4ad23b2f0483d8d2b450961a51470c:

Fatal error:

panic: test timed out after 53m17s
running tests:
	TestTenantStreamingJobRetryReset (53m1s)

Stack:

goroutine 334477 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x265
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x45
Log preceding fatal error 

* 
* goroutine 10077 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:423 +0xbd
* created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 10016
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:420 +0x676
* 
* goroutine 10080 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:660 +0x19b
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked in goroutine 10016
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:649 +0x105
* 
* goroutine 10081 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:243 +0x19b
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 10016
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:238 +0xad
* 
* goroutine 10079 [select, 4 minutes]:
* github.com/cockroachdb/cockroach/pkg/util/tracing.(*Tracer).runPeriodicSnapshotsLoop(0xc007b44840, 0xc002536000, 0xc00d1f3080, 0xc0024141e0, 0x0)
* 	github.com/cockroachdb/cockroach/pkg/util/tracing/tracer_snapshots.go:284 +0x1b6
* github.com/cockroachdb/cockroach/pkg/util/tracing.(*Tracer).PeriodicSnapshotsLoop(0xc007b44840, 0xc002536000, 0xc0024141e0)
* 	github.com/cockroachdb/cockroach/pkg/util/tracing/tracer_snapshots.go:250 +0x105
* github.com/cockroachdb/cockroach/pkg/server.newSQLServer.func1({0xc00a62ec08?, 0x0?})
* 	github.com/cockroachdb/cockroach/pkg/server/server_sql.go:554 +0xee
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10016
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d
* 
* goroutine 10076 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x1cd
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 10016
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x8e
* 
* goroutine 10083 [chan receive, 4 minutes]:
* github.com/cockroachdb/pebble.(*cleanupManager).mainLoop(0xc00b3fa8c0)
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/cleaner.go:161 +0x1e7
* github.com/cockroachdb/pebble.openCleanupManager.func1.1({0xdc28d18?, 0xc005a29920?})
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/cleaner.go:96 +0x2f
* runtime/pprof.Do({0xdc289c8, 0x132d1740}, {{0xc0001f4c60?, 0x8dc4070?, 0x7fa6ce895400?}}, 0xc0044defc0)
* 	GOROOT/src/runtime/pprof/runtime.go:51 +0x118
* github.com/cockroachdb/pebble.openCleanupManager.func1()
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/cleaner.go:95 +0x92
* created by github.com/cockroachdb/pebble.openCleanupManager in goroutine 10016
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/cleaner.go:94 +0x4fd
* 
*

Parameters:

  • attempt=1
  • race=true
  • run=1
  • shard=4
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on release-24.1.0-rc @ 34bbed697e77466f52012a80f0f6ae496c0d2889:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantStreamingJobRetryReset (14m53s)

Stack:

goroutine 377468 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

* 
* goroutine 9017 [select, 4 minutes]:
* github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x7f43558, 0xc008cbb620})
* 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:246 +0xc5
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 8956
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
* goroutine 11217 [select, 4 minutes]:
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans.func1({0x7f43520, 0xc0060591d0})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:255 +0x13a
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans.Group.GoCtx.func3()
* 	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x1f
* golang.org/x/sync/errgroup.(*Group).Go.func1()
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x56
* created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 11211
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x96
* 
* goroutine 8989 [chan receive, 4 minutes]:
* github.com/cockroachdb/pebble.(*cleanupManager).mainLoop(0xc00dfc2960)
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/cleaner.go:161 +0xe5
* github.com/cockroachdb/pebble.openCleanupManager.func1.1({0x7f43558?, 0xc00913a9c0?})
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/cleaner.go:96 +0x17
* runtime/pprof.Do({0x7f43208?, 0xc310940?}, {{0xc000e11140?, 0xc002436fd0?, 0x4a77bd?}}, 0xc002436fc0)
* 	GOROOT/src/runtime/pprof/runtime.go:51 +0x9d
* github.com/cockroachdb/pebble.openCleanupManager.func1()
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/cleaner.go:95 +0x56
* created by github.com/cockroachdb/pebble.openCleanupManager in goroutine 8956
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/cleaner.go:94 +0x265
* 
* goroutine 8990 [chan receive, 4 minutes]:
* github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x7f43558?, 0xc00913aa20?})
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:452 +0x94
* runtime/pprof.Do({0x7f43208?, 0xc310940?}, {{0xc000e11200?, 0x0?, 0x0?}}, 0xc0024377a8)
* 	GOROOT/src/runtime/pprof/runtime.go:51 +0x9d
* github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:450 +0x52
* created by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 8956
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:442 +0xe5
* 
* goroutine 15409 [select, 3 minutes]:
* github.com/cockroachdb/cockroach/pkg/kv/kvserver/idalloc.(*Allocator).start.func1({0x7f43558, 0xc001970f00})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/idalloc/id_alloc.go:151 +0x73b
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 15408
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
*

Parameters:

  • attempt=1
  • run=4
  • shard=4
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on release-24.1.0-rc @ 34bbed697e77466f52012a80f0f6ae496c0d2889:

Fatal error:

panic: test timed out after 53m17s
running tests:
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x92
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10798
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x125
	TestTenantStreamingJobRetryReset (53m2s)

Stack:

goroutine 336766 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x265
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x45
Log preceding fatal error 

	GOROOT/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc00664e4c8)
	GOROOT/src/sync/waitgroup.go:116 +0xa5
golang.org/x/sync/errgroup.(*Group).Wait(0xc00664e4c0)
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:56 +0x38
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc00664e4c0?, {0xdc3d460?, 0xc00bb03b80?}})
	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x65
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0xdc3d498, 0xc00a9be900}, {0xc00c79fb58, 0x2, 0x9585e80?})
	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:192 +0x199
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc00321d680, {0xdc3d498, 0xc00a9be900}, {0xdc92320, 0xc009841900})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:343 +0xfb1
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).start.func2({0xdc3d498, 0xc00a9be870})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:242 +0x576
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9865
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d

goroutine 9866 [select, 53 minutes]:
github.com/cockroachdb/cockroach/pkg/server.startSystemLogsGC.func1({0xdc3d498, 0xc0098d7da0})
	github.com/cockroachdb/cockroach/pkg/server/server_systemlog_gc.go:263 +0x314
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9413
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d

goroutine 11527 [select, 53 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0032de940)
	google.golang.org/grpc/external/org_golang_google_grpc/balancer_conn_wrappers.go:115 +0x150
created by google.golang.org/grpc.newCCBalancerWrapper in goroutine 11524
	google.golang.org/grpc/external/org_golang_google_grpc/balancer_conn_wrappers.go:76 +0x50b

goroutine 10504 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc00321d680, {0xdc3d460, 0xc00bb03b80}, {0xdc92320, 0xc009841900}, 0xc00c71f500)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:383 +0x11f
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0xdc3d460, 0xc00bb03b80})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:340 +0x6d
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x47
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x92
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9872
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x125

goroutine 10977 [select, 53 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans.func1({0xdc3d460, 0xc007a02aa0})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:255 +0x174
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans.Group.GoCtx.func3()
	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x47
golang.org/x/sync/errgroup.(*Group).Go.func1()

Parameters:

  • attempt=1
  • race=true
  • run=1
  • shard=4
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on release-24.1.0-rc @ 34bbed697e77466f52012a80f0f6ae496c0d2889:

Fatal error:

panic: test timed out after 53m17s
running tests:
	TestTenantStreamingJobRetryReset (53m1s)

Stack:

goroutine 243861 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x265
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x45
Log preceding fatal error 

* github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0xdc3d498, 0xc002888090})
* 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:246 +0x145
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9386
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d
* 
* goroutine 9488 [select, 4 minutes]:
* github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0xdc3d498, 0xc0028880f0})
* 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:246 +0x145
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9386
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d
* 
* goroutine 9539 [chan receive, 4 minutes]:
* github.com/cockroachdb/cockroach/pkg/server.startHTTPService.func1({0xdc3d498, 0xc0028897d0})
* 	github.com/cockroachdb/cockroach/pkg/server/server_http.go:295 +0x71
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9386
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d
* 
* goroutine 9489 [select, 4 minutes]:
* github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1({0xdc3d498, 0xc002889680})
* 	github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:168 +0x165
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9386
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d
* 
* goroutine 9538 [select, 4 minutes]:
* github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/hlc/pkg/util/hlc/hlc.go:275 +0x20c
* created by github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps in goroutine 9386
* 	github.com/cockroachdb/cockroach/pkg/util/hlc/pkg/util/hlc/hlc.go:267 +0x237
* 
* goroutine 9435 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x1cd
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9385
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x8e
* 
* goroutine 9434 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:423 +0xbd
* created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 9385
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:420 +0x676
* 
*

Parameters:

  • attempt=1
  • race=true
  • run=1
  • shard=4
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue May 6, 2024
Closed
@exalate-issue-sync exalate-issue-sync bot added the P-1 Issues/test failures with a fix SLA of 1 month label May 6, 2024
@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on release-24.1.0-rc @ 34bbed697e77466f52012a80f0f6ae496c0d2889:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantStreamingJobRetryReset (14m51s)

Stack:

goroutine 368977 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

* github.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0xc004ab8c08, 0xc00e87f340, {0x7f89350, 0xc00ea064e0})
* 	github.com/cockroachdb/cockroach/pkg/server/node.go:1846 +0x265
* github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func2.1({0xc00bd31a40?, 0x7f43558?}, {0x7f7d2c0?, 0xc00c4b51a0?})
* 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1032 +0x6c
* github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x65de880, 0xc004ab8c08}, {0x7f7d230, 0xc00ea05080}, 0xbb0ef10, 0xc0023b60e0)
* 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:165 +0x35b
* github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1({0x65de880?, 0xc004ab8c08?}, {0x7f7d230?, 0xc00ea05080?})
* 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:847 +0x59
* github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x65de880, 0xc004ab8c08}, {0x7f7d230, 0xc00ea05080}, 0xc00ea05050?, 0xc0014fd8c0)
* 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:178 +0x6c
* github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1({0x65de880?, 0xc004ab8c08?}, {0x7f7d230?, 0xc00ea05080?})
* 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:850 +0x7a
* github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0xc002d3c000?, {{0x78f?}, {0x7f8fc50?, 0xc007c42a80?}}}, {0x65de880, 0xc004ab8c08}, {0x7f7d230, 0xc00ea05080}, 0xbb0ef10, 0xc0014fd8c0)
* 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:157 +0x384
* github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1({0x65de880?, 0xc004ab8c08?}, {0x7f7d230?, 0xc00ea05080?})
* 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:850 +0x7a
* github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x66?, 0x40?})
* 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:145 +0x27
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc002b6b5f0, {0x7f43558, 0xc00ea05050}, {0xc00da53e78?, 0xbe49cb?}, 0xc00da53e78)
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:337 +0xcb
* github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x65de880, 0xc004ab8c08}, {0x7f7d230, 0xc00ea05080}, 0xbb0ef10, 0xc0014fd8c0)
* 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:144 +0xdf
* github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1({0x65de880?, 0xc004ab8c08?}, {0x7f7d230?, 0xc00ea05080?})
* 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:850 +0x7a
* github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run({0xc00561c660, 0x4, 0x4}, {0x65de880, 0xc004ab8c08}, {0x7f7d230, 0xc00ea05080}, 0xbb0ef10, 0xc0023b60e0)
* 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:852 +0x11e
* github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func2()
* 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1037 +0x11d
* created by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 14613
* 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1025 +0x2d0
* 
* goroutine 14666 [select, 4 minutes]:
* database/sql.(*DB).connectionOpener(0xc00f2ea680, {0x7f43520, 0xc00f268fa0})
* 	GOROOT/src/database/sql/sql.go:1246 +0x87
* created by database/sql.OpenDB in goroutine 11237
* 	GOROOT/src/database/sql/sql.go:824 +0x14c
* 
* goroutine 13975 [select]:
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc005b08600, {0x7f43520, 0xc0085f61e0}, {0x7f98840, 0xc00490e0c0}, 0xc0093dd6e0)
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:383 +0xe7
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x7f43520?, 0xc0085f61e0?})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:340 +0x36
* github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x1f
* golang.org/x/sync/errgroup.(*Group).Go.func1()
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x56
* created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 13905
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x96
* 
*

Parameters:

  • attempt=1
  • run=21
  • shard=4
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on release-24.1.0-rc @ 34bbed697e77466f52012a80f0f6ae496c0d2889:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantStreamingJobRetryReset (14m50s)

Stack:

goroutine 416461 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 15053
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
* goroutine 12410 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:243 +0xcb
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 12350
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:238 +0x58
* 
* goroutine 86093 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:243 +0xcb
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 12587
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:238 +0x58
* 
* goroutine 12443 [select, 4 minutes]:
* github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1({0x7f43558, 0xc00ac85020})
* 	github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:168 +0x109
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 12351
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
* goroutine 12469 [select, 4 minutes]:
* github.com/cockroachdb/cockroach/pkg/sql.(*runnerCoordinator).init.func1({0xc0020ec408?, 0x1?})
* 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:161 +0xe9
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 12438
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
* goroutine 12468 [select, 4 minutes]:
* google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00dc20840)
* 	google.golang.org/grpc/external/org_golang_google_grpc/balancer_conn_wrappers.go:115 +0x6d
* created by google.golang.org/grpc.newCCBalancerWrapper in goroutine 12351
* 	google.golang.org/grpc/external/org_golang_google_grpc/balancer_conn_wrappers.go:76 +0x22b
* 
* goroutine 12412 [chan receive, 4 minutes]:
* github.com/cockroachdb/pebble.(*cleanupManager).mainLoop(0xc00b04a640)
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/cleaner.go:161 +0xe5
* github.com/cockroachdb/pebble.openCleanupManager.func1.1({0x7f43558?, 0xc008c93110?})
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/cleaner.go:96 +0x17
* runtime/pprof.Do({0x7f43208?, 0xc310940?}, {{0xc000dfc8e0?, 0x0?, 0x0?}}, 0xc0023fcfc0)
* 	GOROOT/src/runtime/pprof/runtime.go:51 +0x9d
* github.com/cockroachdb/pebble.openCleanupManager.func1()
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/cleaner.go:95 +0x56
* created by github.com/cockroachdb/pebble.openCleanupManager in goroutine 12350
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/cleaner.go:94 +0x265
* 
*

Parameters:

  • attempt=1
  • run=20
  • shard=4
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on release-24.1.0-rc @ 34bbed697e77466f52012a80f0f6ae496c0d2889:

Fatal error:

panic: test timed out after 53m17s
running tests:
	TestTenantStreamingJobRetryReset (53m1s)

Stack:

goroutine 336462 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x265
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x45
Log preceding fatal error 

* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:953 +0x47
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc00cf5ddb0, 0xc002793558)
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:135 +0x2be
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc01020ec08, {0xdc3d460, 0xc00cf5da90}, 0xc007079e00, {{0xc0095005c8, 0x1, 0x8}, {0xc0095005df, 0x1, 0x1}}, ...)
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:952 +0x10ba
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc01020ec08, {0xdc3d460, 0xc00cf5da90}, 0xc007079e00, {{0xc0095005c8, 0x1, 0x8}, {0xc0095005df, 0x1, 0x1}}, ...)
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:654 +0x725
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans.func1.1({0xdc3d460, 0xc00cf5da90})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:283 +0x26c
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans.func1.Group.GoCtx.2()
* 	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x47
* golang.org/x/sync/errgroup.(*Group).Go.func1()
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x92
* created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 12904
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x125
* 
* goroutine 9948 [select, 4 minutes]:
* github.com/cockroachdb/cockroach/pkg/sql.(*runnerCoordinator).init.func1({0xc00320ec08?, 0xc008cff9b0?})
* 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:161 +0x126
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9967
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d
* 
* goroutine 9876 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:660 +0x19b
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked in goroutine 9874
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:649 +0x105
* 
* goroutine 9937 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:243 +0x19b
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 9875
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:238 +0xad
* 
* goroutine 23044 [chan send, 3 minutes]:
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.sendMetadata(...)
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:525
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans.func1({0xdc3d460, 0xc0031c9770})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:288 +0xcf4
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans.Group.GoCtx.func3()
* 	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x47
* golang.org/x/sync/errgroup.(*Group).Go.func1()
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x92
* created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 23042
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x125
* 
*

Parameters:

  • attempt=1
  • race=true
  • run=1
  • shard=4
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@github-actions GitHub Actions
Copy link

github-actions bot commented May 7, 2024

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on release-24.1.0-rc @ 418905b98eb6c8cf4c035ae6f70464c61a50529a:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantStreamingJobRetryReset (14m48s)

Stack:

goroutine 395009 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 13075
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f
* 
* goroutine 13100 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:423 +0x85
* created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 13075
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:420 +0x3be
* 
* goroutine 13103 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 13075
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f
* 
* goroutine 13095 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 13075
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f
* 
* goroutine 24228 [semacquire, 3 minutes]:
* sync.runtime_Semacquire(0xc00d9522b8?)
* 	GOROOT/src/runtime/sema.go:62 +0x25
* sync.(*WaitGroup).Wait(0x4a8d60?)
* 	GOROOT/src/sync/waitgroup.go:116 +0x48
* golang.org/x/sync/errgroup.(*Group).Wait(0xc0066a7e00)
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:56 +0x25
* github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0066a7e00?, {0x7f43520?, 0xc007a30820?}})
* 	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x47
* github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x7f43558?, 0xc00a6ba930?}, {0xc00e073cf0, 0x2, 0x5d1e740?})
* 	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:192 +0xff
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc005b3aa80, {0x7f43558, 0xc00a6ba930}, {0x7f988a0, 0xc007a306e0})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:343 +0x886
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).start.func2({0x7f43558, 0xc00a6ba8d0})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:242 +0x37b
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 24213
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
* 
* goroutine 81729 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:243 +0xcb
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 13350
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:238 +0x58
* 
*

Parameters:

  • attempt=1
  • run=1
  • shard=4
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@dt
Copy link
Member

dt commented May 8, 2024

fixed by #123676

@dt dt closed this as completed May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@msbutler msbutler

Labels
branch-release-24.1.0-rc Used to mark GA and release blockers and technical advisories for 24.1.0-rc C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-1 Issues/test failures with a fix SLA of 1 month release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Projects
No open projects
Disaster Recovery Backlog
Status: Done
Milestone
24.1
Development

No branches or pull requests
3 participants
@dt @cockroach-teamcity @msbutler