Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

importccl: TestCSVImportCanBeResumed failed under testrace #74932

Closed
adityamaru opened this issue Jan 18, 2022 · 1 comment · Fixed by #74926
Closed

importccl: TestCSVImportCanBeResumed failed under testrace #74932

adityamaru opened this issue Jan 18, 2022 · 1 comment · Fixed by #74926
Assignees
@irfansharif
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.

Comments

@adityamaru
Copy link
Contributor

Example build: https://teamcity.cockroachdb.com/viewLog.html?buildId=4140135&buildTypeId=Cockroach_UnitTests

DATA RACE:

Write at 0x00c00128b340 by goroutine 232:
  github.com/google/btree.(*items).insertAt()
      /go/src/github.com/cockroachdb/cockroach/vendor/github.com/google/btree/btree.go:148 +0x28a
  github.com/google/btree.(*node).insert()
      /go/src/github.com/cockroachdb/cockroach/vendor/github.com/google/btree/btree.go:316 +0x1d8
  github.com/google/btree.(*BTree).ReplaceOrInsert()
      /go/src/github.com/cockroachdb/cockroach/vendor/github.com/google/btree/btree.go:702 +0x5fc
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*storeReplicaBTree).ReplaceOrInsertReplica()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_replica_btree.go:182 +0x44
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).addReplicaInternalLocked()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_create_replica.go:271 +0x20e
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SplitRange()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_split.go:312 +0x4ea
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.splitPostApply()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_split.go:149 +0xd5
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleSplitResult()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_application_result.go:245 +0x917
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).handleNonTrivialReplicatedEvalResult()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_application_state_machine.go:1245 +0x847
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_application_state_machine.go:1127 +0x81c
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.StateMachine.ApplySideEffects-fm()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:55 +0x85
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:206 +0x1f0
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:291 +0x277
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:247 +0xd2
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:870 +0x1d3d
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:511 +0x14f
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:507 +0x17b
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:305 +0x2e1
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker-fm()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:250 +0x4d
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:488 +0x228

Previous read at 0x00c00128b340 by goroutine 77:
  github.com/google/btree.(*node).iterate()
      /go/src/github.com/cockroachdb/cockroach/vendor/github.com/google/btree/btree.go:510 +0x1e9
  github.com/google/btree.(*BTree).AscendRange()
      /go/src/github.com/cockroachdb/cockroach/vendor/github.com/google/btree/btree.go:750 +0x1a8
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*storeReplicaBTree).ascendRange()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_replica_btree.go:225 +0x2e
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*storeReplicaBTree).VisitKeyRange()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_replica_btree.go:142 +0x264
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).onSpanConfigUpdate()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:2339 +0x411
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Start.func4()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:1950 +0xe4
  github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvsubscriber.(*handler).invoke()
      /go/src/github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvsubscriber/kvsubscriber.go:441 +0x12d
  github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvsubscriber.(*KVSubscriber).run()
      /go/src/github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvsubscriber/kvsubscriber.go:377 +0x170e
  github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvsubscriber.(*KVSubscriber).Start.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvsubscriber/kvsubscriber.go:185 +0x2a4
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:488 +0x228
@adityamaru adityamaru added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Jan 18, 2022
@adityamaru
Copy link
Contributor Author

adityamaru commented Jan 18, 2022 •
edited
Loading

cc: @irfansharif potential fallout from #73876? Assigning this to you speculatively, because of the kvsubscriber stack, feel free to throw it back to me if it's unrelated.

irfansharif added a commit to irfansharif/cockroach that referenced this issue Jan 18, 2022
@irfansharif
kvserver: squash data race
3aaf50b 
Fixes cockroachdb#74932; we were reading from the replicas map without having
acquired the prerequisite lock.

Release note: None
@irfansharif irfansharif linked a pull request Jan 18, 2022 that will close this issue
*: de-flake TestPaginatedBackupTenant and remove data race #74926
Merged
@craig craig bot closed this as completed in 3218353 Jan 18, 2022
@irfansharif irfansharif mentioned this issue Jan 28, 2022
Closed
23 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@irfansharif irfansharif

Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

*: de-flake TestPaginatedBackupTenant and remove data race irfansharif/cockroach
2 participants
@irfansharif @adityamaru