Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/streamingccl/streamingest: TestTenantStreamingMultipleNodes failed #123397

Closed
github-actions bot opened this issue May 1, 2024 · 2 comments · Fixed by #123519
Closed

ccl/streamingccl/streamingest: TestTenantStreamingMultipleNodes failed #123397

github-actions bot opened this issue May 1, 2024 · 2 comments · Fixed by #123519
Assignees
@msbutler
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Milestone
24.1

Comments

@github-actions
Copy link

github-actions bot commented May 1, 2024 •
edited by cockroach-jira-scripts
Loading

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed on master @ 6eeddf14e5420000793f2c5334ca1ac26f0ca12d:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantStreamingMultipleNodes (14m57s)
	TestTenantStreamingMultipleNodes/fromSystem=true (14m31s)

Stack:

goroutine 105307 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

* sync.(*Cond).Wait(0x30?)
* 	GOROOT/src/sync/cond.go:70 +0x85
* github.com/cockroachdb/pebble/record.(*flusherCond).Wait(...)
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:369
* github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0xc006bde000, {0x5c8c520?, 0xc353840?})
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:643 +0x6fd
* runtime/pprof.Do({0x7f3b1a8?, 0xc353840?}, {{0xc000e20bc0?, 0xc001c6cc80?, 0x7ef19a0?}}, 0xc001acb7c0)
* 	GOROOT/src/runtime/pprof/runtime.go:51 +0x9d
* github.com/cockroachdb/pebble/record.NewLogWriter.func2()
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:562 +0x56
* created by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 42215
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:561 +0x4b8
* 
* goroutine 42245 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 42215
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f
* 
* goroutine 42240 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:423 +0x85
* created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 42215
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:420 +0x3be
* 
* goroutine 42234 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:423 +0x85
* created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 42215
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:420 +0x3be
* 
* goroutine 42235 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 42215
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f
* 
* goroutine 42239 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 42215
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f
* 
* goroutine 42236 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:423 +0x85
* created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 42215
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:420 +0x3be
* 
*

Parameters:

  • attempt=1
  • run=1
  • shard=15
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-38343
@github-actions github-actions bot added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery labels May 1, 2024
@github-actions github-actions bot added this to the 24.1 milestone May 1, 2024
@msbutler msbutler mentioned this issue May 2, 2024
Merged
@msbutler msbutler self-assigned this May 2, 2024
msbutler added a commit to msbutler/cockroach that referenced this issue May 2, 2024
@msbutler
rangefeed: fix sendMetadata deadlock
d5dab48 
If the rangefeed client's context is cancelled before a metadata event is sent
on the event channel, the rangefeed could hang forever. This bug was introduced
in cockroachdb#123001.

Fixes cockroachdb#123355
Fixes cockroachdb#123397

Release note: none
@cockroach-teamcity
Copy link
Member

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed on master @ 065119ec0859bae39caa683c7960524ccfc00c85:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantStreamingMultipleNodes (14m57s)
	TestTenantStreamingMultipleNodes/fromSystem=true (14m28s)

Stack:

goroutine 102194 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 43202
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f
* 
* goroutine 43298 [chan receive, 2 minutes]:
* github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x7f55df8?, 0xc00937c030?})
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:417 +0x94
* runtime/pprof.Do({0x7f55aa8?, 0xc378280?}, {{0xc000cee440?, 0x112b940?, 0x7faca60?}}, 0xc004fedfa8)
* 	GOROOT/src/runtime/pprof/runtime.go:51 +0x9d
* github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc004fedfb8?)
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:415 +0x52
* created by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 43202
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:407 +0xe5
* 
* goroutine 43291 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:423 +0x85
* created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 43202
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:420 +0x3be
* 
* goroutine 43300 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:253 +0xcb
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 43202
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:248 +0x58
* 
* goroutine 43240 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:253 +0xcb
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 43202
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:248 +0x58
* 
* goroutine 43284 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 43202
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f
* 
* goroutine 43292 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 43202
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f
* 
* goroutine 43283 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:423 +0x85
* created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 43202
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:420 +0x3be
* 
*

Parameters:

  • attempt=1
  • run=24
  • shard=15
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed on master @ 6e23d32dd0d36181f5c886c9a0eda9c756005bd6:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantStreamingMultipleNodes (14m57s)
	TestTenantStreamingMultipleNodes/fromSystem=false (14m57s)

Stack:

goroutine 75311 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

* syscall.Write(...)
* 	GOROOT/src/syscall/syscall_unix.go:209
* internal/poll.ignoringEINTRIO(...)
* 	GOROOT/src/internal/poll/fd_unix.go:736
* internal/poll.(*FD).Write(0xc0010202a0, {0xc007084000, 0x196ad, 0x1a000})
* 	GOROOT/src/internal/poll/fd_unix.go:380 +0x368
* os.(*File).write(...)
* 	GOROOT/src/os/file_posix.go:46
* os.(*File).Write(0xc000134820, {0xc007084000?, 0x196ad, 0xc0065c1dc8?})
* 	GOROOT/src/os/file.go:189 +0x51
* fmt.Fprintf({0x7f04c00, 0xc000134820}, {0x66ab826, 0xd}, {0xc0065c1dc8, 0x2, 0x2})
* 	GOROOT/src/fmt/print.go:225 +0x97
* github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x7f57918, 0xc01ad438c0}, 0x2, 0x1, 0x0, 0x88?, {0x66d763f, 0x14}, {0xc0065c1e78, 0x2, ...})
* 	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:94 +0x3f8
* github.com/cockroachdb/cockroach/pkg/util/log.shoutfDepth(...)
* 	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:116
* github.com/cockroachdb/cockroach/pkg/util/log.Shoutf(...)
* 	github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:1047
* github.com/cockroachdb/cockroach/pkg/util/log.DumpStacks({0x7f57918, 0xc01ad438c0}, {0x66a786a, 0xc})
* 	github.com/cockroachdb/cockroach/pkg/util/log/clog.go:429 +0x105
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Quiesce(0xc003358120, {0x7f57918, 0xc01ad438c0})
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:607 +0x19d
* github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).stopServers.func1(0xc00ad66230?, 0xc003358120)
* 	github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:156 +0xda
* created by github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).stopServers in goroutine 38
* 	github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:152 +0x14f
* 
* goroutine 63917 [sync.Cond.Wait, 2 minutes]:
* sync.runtime_notifyListWait(0xc012f620a8, 0x1bf)
* 	GOROOT/src/runtime/sema.go:569 +0x159
* sync.(*Cond).Wait(0xc18596d784b6da07?)
* 	GOROOT/src/sync/cond.go:70 +0x85
* github.com/cockroachdb/pebble/record.(*flusherCond).Wait(...)
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:369
* github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0xc012f62000, {0x5ca1400?, 0xc37a440?})
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:643 +0x6fd
* runtime/pprof.Do({0x7f575c8?, 0xc37a440?}, {{0xc000e3ab80?, 0xc00a3fd560?, 0xc01235ef70?}}, 0xc01235efc0)
* 	GOROOT/src/runtime/pprof/runtime.go:51 +0x9d
* github.com/cockroachdb/pebble/record.NewLogWriter.func2()
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:562 +0x56
* created by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 276
* 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:561 +0x4b8
* 
* goroutine 63905 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:253 +0xcb
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 2788
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:248 +0x58
* 
*

Parameters:

  • attempt=1
  • run=19
  • shard=15
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

craig bot pushed a commit that referenced this issue May 6, 2024
@msbutler
Merge #123519
a7b52e0 
123519: rangefeed: fix sendMetadata deadlock r=msbutler a=msbutler

If the rangefeed client's context is cancelled before a metadata event is sent
on the event channel, the rangefeed could hang forever. This bug was introduced
in #123001.

Fixes #123355
Fixes #123397

Release note: none

Co-authored-by: Michael Butler <[email protected]>
@craig craig bot closed this as completed in 22c698a May 6, 2024
blathers-crl bot pushed a commit that referenced this issue May 6, 2024
@msbutler
rangefeed: fix sendMetadata deadlock
74b8e5a 
If the rangefeed client's context is cancelled before a metadata event is sent
on the event channel, the rangefeed could hang forever. This bug was introduced
in #123001.

Fixes #123355
Fixes #123397

Release note: none
@blathers-crl blathers-crl bot mentioned this issue May 6, 2024
Merged
blathers-crl bot pushed a commit that referenced this issue May 6, 2024
@msbutler
rangefeed: fix sendMetadata deadlock
0c8a1cf 
If the rangefeed client's context is cancelled before a metadata event is sent
on the event channel, the rangefeed could hang forever. This bug was introduced
in #123001.

Fixes #123355
Fixes #123397

Release note: none
@blathers-crl blathers-crl bot mentioned this issue May 6, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@msbutler msbutler

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Projects
No open projects
Disaster Recovery Backlog
Status: Done
Milestone
24.1
Development

Successfully merging a pull request may close this issue.

rangefeed: fix sendMetadata deadlock msbutler/cockroach
2 participants
@cockroach-teamcity @msbutler