Merge #51959 #52269 #52273 #52343

51959: backupccl: distribute RESTORE work using DistSQL r=dt,yuzefovich a=pbardea

This commit replaces restore's old method of coordinating work on a
single node to instead use DistSQL. It creates a 2 stage DistSQL flow.
The first stage assigns chunks to SplitAndScatter processors which
forward spans that they scattered to the second stage made up of the
RestoreData processors which ingest the data. The SplitAndScatter
processors attempt to send the work to the RestoreData colocated with
the leaseholder of the span after the scattering.

Release note: None.

52269: roachtest: correctly start crdb in acceptance/rapid_restart r=andreimatei a=andreimatei

--start-single-node was needed. Without it, I think the test's killing
of a node raced with that node dieing by itself, and sometimes the race
resulted in `cockroach stop` first seeing the process but then
`/bin/bash: line 8: kill: (16016) - No such proces`

Fixes #52060

Release note: None

52273: roachtest: remove confusing "--- PASS" log lines r=andreimatei a=andreimatei

Two workloads were printing PASS/FAIL lines from inside the predicates
passed to Searcher.Search. This is really confusing when reading the
test's output, because they don't correspond to the test's disposition.

Release note: None

52343: roachtest: reflake (skip) disk-stalled test for release-19.1 r=irfansharif a=irfansharif

Fixes #52181. This test doesn't make use of all the `roachtest start`
smartness to figure out what sub-command to use and instead constructs
the raw start command by hand.

Given our support policy, let's just bump the minimum version this test
expects to run. This feels like another instance where #51897 would be 
nice to have.

Release note: None

Co-authored-by: Paul Bardea <[email protected]>
Co-authored-by: Andrei Matei <[email protected]>
Co-authored-by: irfan sharif <[email protected]>

pkg/ccl/backupccl/backup.proto

@@ -180,4 +180,5 @@ message ScheduledBackupExecutionArgs {
 message RestoreProgress {
   RowCount summary = 1 [(gogoproto.nullable) = false];
   int64 progressIdx = 2;
+  roachpb.Span dataSpan = 3 [(gogoproto.nullable) = false];
 }

pkg/ccl/backupccl/restore_data_processor.go

@@ -159,6 +159,7 @@ func runRestoreData(
 		progDetails := RestoreProgress{}
 		progDetails.Summary = countRows(importRes.(*roachpb.ImportResponse).Imported, spec.PKIDs)
 		progDetails.ProgressIdx = entry.ProgressIdx
+		progDetails.DataSpan = entry.Span
 		details, err := gogotypes.MarshalAny(&progDetails)
 		if err != nil {
 			return err