Skip to content

Commit

Permalink
backupccl: distribute RESTORE work using DistSQL
Browse files Browse the repository at this point in the history
This commit replaces restore's old method of coordinating work on a
single node to instead use DistSQL. It creates a 2 stage DistSQL flow.
The first stage assigns chunks to SplitAndScatter processors which
forward spans that they scattered to the second stage made up of the
RestoreData processors which ingest the data. The SplitAndScatter
processors attempt to send the work to the RestoreData colocated with
the leaseholder of the span after the scattering.

Release note: None.
  • Loading branch information
pbardea committed Jul 28, 2020
1 parent 3a1b423 commit c634534
Show file tree
Hide file tree
Showing 7 changed files with 474 additions and 385 deletions.
277 changes: 160 additions & 117 deletions pkg/ccl/backupccl/backup.pb.go

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions pkg/ccl/backupccl/backup.proto
Original file line number Diff line number Diff line change
Expand Up @@ -175,4 +175,5 @@ message ScheduledBackupExecutionArgs {
message RestoreProgress {
RowCount summary = 1 [(gogoproto.nullable) = false];
int64 progressIdx = 2;
roachpb.Span dataSpan = 3 [(gogoproto.nullable) = false];
}
1 change: 1 addition & 0 deletions pkg/ccl/backupccl/restore_data_processor.go
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,7 @@ func runRestoreData(
progDetails := RestoreProgress{}
progDetails.Summary = countRows(importRes.(*roachpb.ImportResponse).Imported, spec.PKIDs)
progDetails.ProgressIdx = entry.ProgressIdx
progDetails.DataSpan = entry.Span
details, err := gogotypes.MarshalAny(&progDetails)
if err != nil {
return err
Expand Down
Loading

0 comments on commit c634534

Please sign in to comment.