Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

backupccl: TestBackupRestoreResume panics when run within a tenant #88527

Closed
msbutler opened this issue Sep 22, 2022 · 2 comments
Closed

backupccl: TestBackupRestoreResume panics when run within a tenant #88527

msbutler opened this issue Sep 22, 2022 · 2 comments
Assignees
@msbutler
Labels
A-disaster-recovery C-test-failure Broken test (automatically or manually discovered). T-disaster-recovery

Comments

@msbutler
Copy link
Collaborator

msbutler commented Sep 22, 2022 •
edited by cockroach-jira-scripts
Loading

To reproduce, stress the test on top of #88271, with DisableTenantTestTenant set to false.

54 panic: interval: inverted range [recovered]                                     
   55         panic: interval: inverted range                                         
   56                                                                                 
   57 goroutine 23159 [running]:                                                      
   58 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc0076cee10, {0x9c6c510, 0xc007596930})
   59         github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:248 +0xa5     
   60 panic({0x85176a0, 0xc000116030})                                                
   61         GOROOT/src/runtime/panic.go:884 +0x212                                  
   62 github.com/cockroachdb/cockroach/pkg/util/interval.(*rangeTree).Sub(0xc00e7b24b0, {{0xc004b25dd8, 0x4, 0x8}, {0xc004b25e50, 0x5, 0x8}})
   63         github.com/cockroachdb/cockroach/pkg/util/interval/range_group.go:639 +0x4d0
   64 github.com/cockroachdb/cockroach/pkg/roachpb.(*SpanGroup).Sub(0xc00efed1d0, {0xc0061c26c0?, 0x1, 0x1?})
   65         github.com/cockroachdb/cockroach/pkg/roachpb/pkg/roachpb/span_group.go:54 +0x377
   66 github.com/cockroachdb/cockroach/pkg/ccl/backupccl.filterSpans({0x0?, 0x0?, 0x0?}, {0xc0061c26c0, 0x1, 0x1})
   67         github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_job.go:90 +0x65
   68 github.com/cockroachdb/cockroach/pkg/ccl/backupccl.backup({0x9c6c510, 0xc00f8e8e10}, {0x9cb9838?, 0xc00b2600d8}, {0xc00a94fd70, 0x2e}, 0x0?, 0xc009436000, {0x9cb5e70, 0xc0061c24e0}, ...)
   69         github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_job.go:155 +0x11e
   70 github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*backupResumer).Resume(0xc007596900, {0x9c6c510, 0xc00f8e8e10}, {0x8678900?, 0xc00b2600d8})
   71         github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_job.go:643 +0x207c
   72 github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2(0x9c6c510?, 0x9c6c510?, {0x9c45a08?, 0xc007596900?}, {0x9c6c510?, 0xc00f8e8e10?}, {0x8678900?, 0xc00b2600d8?}, 0xc00efef8c0)
   73         github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1239 +0xcc        
   74 github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc001f29340, {0x9c6c510, 0xc003903ef0}, {0x8678900, 0xc00b2600d8}, {0x9c45a08, 0xc007596900}, 0xc00ae41d40, {0xc004200368, 0x7}, ...)
   75         github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1240 +0x6ab       
   76 github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc001f29340, {0x9c6c468, 0xc00c0a6d80}, {0x9c45a08, 0xc007596900}, 0xc00ae41d40, {0xc004200368, 0x7}, {0xc004552048, 0x16})
   77         github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:410 +0x5c6           
   78 github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1({0xc005c7d500?, 0xc008e95fd0?})
   79         github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:330 +0xfa            
   80 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
   81         github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:489 +0x146    
   82 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
   83         github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480 +0x43b    
   84

Jira issue: CRDB-19865
@msbutler msbutler added C-test-failure Broken test (automatically or manually discovered). T-disaster-recovery labels Sep 22, 2022
@blathers-crl
Copy link

blathers-crl bot commented Sep 22, 2022

cc @cockroachdb/disaster-recovery

@msbutler msbutler mentioned this issue Sep 22, 2022
Merged
@msbutler msbutler self-assigned this Sep 27, 2022
msbutler added a commit to msbutler/cockroach that referenced this issue Sep 27, 2022
@msbutler
backupccl: allow backupRestoreTestSetup[Empty]WithParams to test with…
c321fef 
…in tenant

Previously, all backup unit tests that called backupRestoreTestSetupWithParams
and backupRestoreTestSetupEmptyWithParams were disabled to run within tenants,
signficantly decreasing test coverage.

This patch allows tests that call these helper methods to be probabilistically
run within tenants (including most data driven tests). This patch also manually
disables some of these tests to be run within tenants. Future work should
continue to enable bulk unit tests to run within tenants by default.

Fixes cockroachdb#88381, cockroachdb#88527, cockroachdb#88453, cockroachdb#88380
Release note: None
msbutler added a commit to msbutler/cockroach that referenced this issue Oct 14, 2022
@msbutler
backupccl: allow backupRestoreTestSetup[Empty]WithParams to test with…
81fc39c 
…in tenant

Previously, all backup unit tests that called backupRestoreTestSetupWithParams
and backupRestoreTestSetupEmptyWithParams were disabled to run within tenants,
signficantly decreasing test coverage.

This patch allows tests that call these helper methods to be probabilistically
run within tenants (including most data driven tests). This patch also manually
disables some of these tests to be run within tenants. Future work should
continue to enable bulk unit tests to run within tenants by default.

Fixes cockroachdb#88381, cockroachdb#88527, cockroachdb#88453, cockroachdb#88380
Release note: None
msbutler added a commit to msbutler/cockroach that referenced this issue Oct 19, 2022
@msbutler
backupccl: allow backupRestoreTestSetup[Empty]WithParams to test with…
fd7f48d 
…in tenant

Previously, all backup unit tests that called backupRestoreTestSetupWithParams
and backupRestoreTestSetupEmptyWithParams were disabled to run within tenants,
signficantly decreasing test coverage.

This patch allows tests that call these helper methods to be probabilistically
run within tenants (including most data driven tests). This patch also manually
disables some of these tests to be run within tenants. Future work should
continue to enable bulk unit tests to run within tenants by default.

Fixes cockroachdb#88381, cockroachdb#88527, cockroachdb#88453, cockroachdb#88380
Release note: None
craig bot pushed a commit that referenced this issue Oct 19, 2022
@msbutler
Merge #88271
e711c31 
88271: backupccl: allow backupRestoreTestSetup[Empty]WithParams to test within a tenant r=adityamaru a=msbutler

Previously, all backup unit tests that called backupRestoreTestSetupWithParams
and backupRestoreTestSetupEmptyWithParams were disabled to run within tenants,
signficantly decreasing test coverage.

This patch allows tests that call these helper methods to be probabilistically
run within tenants (including most data driven tests). This patch also manually
disables some of these tests to be run within tenants. Future work should
continue to enable bulk unit tests to run within tenants by default.

Fixes #88381, #88527, #88453, #88380
Release note: None

Co-authored-by: Michael Butler <[email protected]>
@msbutler
Copy link
Collaborator Author

closed by #88271

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@msbutler msbutler

Labels
A-disaster-recovery C-test-failure Broken test (automatically or manually discovered). T-disaster-recovery
Projects
No open projects
Disaster Recovery Backlog
Archived in project
Milestone
No milestone
Development

No branches or pull requests
1 participant
@msbutler