ccl/streamingccl/streamingest: TestTenantStreamingUnavailableStreamAddress failed #105956

cockroach-teamcity · 2023-07-01T07:27:34Z

ccl/streamingccl/streamingest.TestTenantStreamingUnavailableStreamAddress failed with artifacts on release-23.1 @ e12e85479312972b551677203849d29aeb38ad5f:

=== RUN   TestTenantStreamingUnavailableStreamAddress
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingUnavailableStreamAddress2756112255
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:367: condition failed to evaluate within 45s: stream ingestion has not recorded any progress yet, waiting to advance pos 1688195267.796178305,0
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingUnavailableStreamAddress2756112255
--- FAIL: TestTenantStreamingUnavailableStreamAddress (500.33s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery _{This test on roachdash | Improve this report!

Jira issue: CRDB-29284}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2023-07-02T07:19:42Z

ccl/streamingccl/streamingest.TestTenantStreamingUnavailableStreamAddress failed with artifacts on release-23.1 @ e12e85479312972b551677203849d29aeb38ad5f:

=== RUN   TestTenantStreamingUnavailableStreamAddress
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingUnavailableStreamAddress2276726807
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:367: condition failed to evaluate within 45s: stream ingestion has not recorded any progress yet, waiting to advance pos 1688281123.781038089,0
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingUnavailableStreamAddress2276726807
--- FAIL: TestTenantStreamingUnavailableStreamAddress (532.68s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-07-03T07:40:07Z

ccl/streamingccl/streamingest.TestTenantStreamingUnavailableStreamAddress failed with artifacts on release-23.1 @ e12e85479312972b551677203849d29aeb38ad5f:

=== RUN   TestTenantStreamingUnavailableStreamAddress
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingUnavailableStreamAddress809722336
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:367: condition failed to evaluate within 45s: stream ingestion has not recorded any progress yet, waiting to advance pos 1688368802.484397016,0
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingUnavailableStreamAddress809722336
--- FAIL: TestTenantStreamingUnavailableStreamAddress (578.63s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-07-04T06:26:10Z

ccl/streamingccl/streamingest.TestTenantStreamingUnavailableStreamAddress failed with artifacts on release-23.1 @ a26ee7efd223c92cda531152837302f03a14f461:

=== RUN   TestTenantStreamingUnavailableStreamAddress
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingUnavailableStreamAddress4241893779
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:367: condition failed to evaluate within 45s: stream ingestion has not recorded any progress yet, waiting to advance pos 1688450696.941843740,0
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingUnavailableStreamAddress4241893779
--- FAIL: TestTenantStreamingUnavailableStreamAddress (593.06s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

tbg · 2023-07-04T12:10:39Z

#106104 also got this on master here. Seems unrelated. Also, this seems to have started failing very regularly on 23.1, did something get backported?

test_server_shim.go:467: migration-job-find-already-completed: failed to connect to n1 at 127.0.0.1:43615: initial connection heartbeat failed: grpc: connection error: desc = "error reading server preface: http2: frame too large" [code 14/Unavailable]

cockroach-teamcity · 2023-07-05T07:33:21Z

ccl/streamingccl/streamingest.TestTenantStreamingUnavailableStreamAddress failed with artifacts on release-23.1 @ f1108313b2b51ffab176b6b230bcc15b0259f170:

=== RUN   TestTenantStreamingUnavailableStreamAddress
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingUnavailableStreamAddress3527897146
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:367: condition failed to evaluate within 45s: waiting for stream ingestion job progress 1688541124.298665829,0 to advance beyond 1688541133.778928428,0
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingUnavailableStreamAddress3527897146
--- FAIL: TestTenantStreamingUnavailableStreamAddress (507.43s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

tbg · 2023-07-05T08:37:07Z

and again here: https://teamcity.cockroachdb.com/viewLog.html?buildId=10786073&buildTypeId=Cockroach_BazelEssentialCi

replication_stream_e2e_test.go:542: error executing 'SELECT * FROM d.t1': pq: failed to connect to n1 at 127.0.0.1:35415: initial connection heartbeat failed: grpc: client cluster ID "6b445207-5c57-4bc2-b1b5-8b62627806f3" doesn't match server cluster ID "40f5a19a-3ff0-4930-ba72-bdb97e8bf3df" [code 2/Unknown]

Informs cockroachdb#105956 Release note: None

106149: c2c: skip TestTenantStreamingUnavailableStreamAddress r=msbutler a=msbutler Informs #105956 Release note: None Co-authored-by: Michael Butler <[email protected]>

Informs #105956 Release note: None

106310: streamingest: unskip a multi-node c2c test, and skip under deadlock r=lidorcarmel a=lidorcarmel These tests are slow and flaky under deadlock. Epic: none Fixes: #105956 Fixes: #99900 Release note: None 106765: rfcs: rename the filename to adhere to the pattern r=yuzefovich a=yuzefovich Recently merged RFC on query SST metrics was missing the prepended date in the filename. Epic: None. Release note: None Co-authored-by: Lidor Carmel <[email protected]> Co-authored-by: Yahor Yuzefovich <[email protected]>

These tests are slow and flaky under deadlock. Epic: none Fixes: cockroachdb#105956 Fixes: cockroachdb#99900 Release note: None

cucaroach · 2023-07-18T12:11:07Z

I just hit this variation in 23.1 CI:

Failed
=== RUN   TestTenantStreamingUnavailableStreamAddress
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingUnavailableStreamAddress2833981180
    test_log_scope.go:79: use -show-logs to present logs inline
    replication_stream_e2e_test.go:514: error executing 'SELECT * FROM d.t1': pq: no inbound stream connection
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingUnavailableStreamAddress2833981180
--- FAIL: TestTenantStreamingUnavailableStreamAddress (50.53s)

https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/10949986?hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandBuildChangesSection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true

cockroach-teamcity added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-disaster-recovery labels Jul 1, 2023

cockroach-teamcity added this to the 23.1 milestone Jul 1, 2023

msbutler added a commit to msbutler/cockroach that referenced this issue Jul 5, 2023

c2c: skip TestTenantStreamingUnavailableStreamAddress

23f70c5

Informs cockroachdb#105956 Release note: None

msbutler mentioned this issue Jul 5, 2023

c2c: skip TestTenantStreamingUnavailableStreamAddress #106149

Merged

craig bot pushed a commit that referenced this issue Jul 5, 2023

Merge #106149

a343d0a

106149: c2c: skip TestTenantStreamingUnavailableStreamAddress r=msbutler a=msbutler Informs #105956 Release note: None Co-authored-by: Michael Butler <[email protected]>

rafiss added the skipped-test label Jul 5, 2023

blathers-crl bot pushed a commit that referenced this issue Jul 5, 2023

c2c: skip TestTenantStreamingUnavailableStreamAddress

f4e3a9f

Informs #105956 Release note: None

blathers-crl bot mentioned this issue Jul 5, 2023

release-23.1: c2c: skip TestTenantStreamingUnavailableStreamAddress #106189

Merged

lidorcarmel self-assigned this Jul 5, 2023

lidorcarmel mentioned this issue Jul 6, 2023

streamingest: unskip a multi-node c2c test, and skip under deadlock #106310

Merged

exalate-issue-sync bot added the sync-me label Jul 11, 2023

craig bot closed this as completed in 8f009ec Jul 13, 2023

This was referenced Jul 13, 2023

release-23.1: streamingest: unskip a multi-node c2c test, and skip under deadlock #106785

Closed

release-23.1: streamingest: unskip a multi-node c2c test, and skip under deadlock #106862

Merged

cucaroach mentioned this issue Jul 18, 2023

release-23.1: copy: fix vectorized copy bug with CSV header #106967

Merged

github-project-automation bot added this to Disaster Recovery Backlog Aug 28, 2024

github-project-automation bot moved this to Done in Disaster Recovery Backlog Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ccl/streamingccl/streamingest: TestTenantStreamingUnavailableStreamAddress failed #105956

ccl/streamingccl/streamingest: TestTenantStreamingUnavailableStreamAddress failed #105956

cockroach-teamcity commented Jul 1, 2023 •

edited by cockroach-jira-scripts

Loading

cockroach-teamcity commented Jul 2, 2023

cockroach-teamcity commented Jul 3, 2023

cockroach-teamcity commented Jul 4, 2023

tbg commented Jul 4, 2023 •

edited

Loading

cockroach-teamcity commented Jul 5, 2023

tbg commented Jul 5, 2023 •

edited

Loading

cucaroach commented Jul 18, 2023

ccl/streamingccl/streamingest: TestTenantStreamingUnavailableStreamAddress failed #105956

ccl/streamingccl/streamingest: TestTenantStreamingUnavailableStreamAddress failed #105956

Comments

cockroach-teamcity commented Jul 1, 2023 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Jul 2, 2023

cockroach-teamcity commented Jul 3, 2023

cockroach-teamcity commented Jul 4, 2023

tbg commented Jul 4, 2023 • edited Loading

cockroach-teamcity commented Jul 5, 2023

tbg commented Jul 5, 2023 • edited Loading

cucaroach commented Jul 18, 2023

cockroach-teamcity commented Jul 1, 2023 •

edited by cockroach-jira-scripts

Loading

tbg commented Jul 4, 2023 •

edited

Loading

tbg commented Jul 5, 2023 •

edited

Loading