roachtest: test AOST restore in backup-restore/* roachtests

[msbutler]

This patch allows the backup-restore driver to run and validate AOST restores from revision history backups. If the driver created a revision history backup, there's a 50% chance it will restore from an AOST between the full backup end time and the last incremental start time. A future patch will allow for AOST restores from non-revision history backups.

Epic: none

Release note: none

[cockroach-teamcity]

This change is 

[msbutler]

I took this PR for a few test drives and it seems to work.

[renatolabs]

Made an optional comment about failure reproducibility but otherwise looks good!

Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @adityamaru and @msbutler)

---

pkg/cmd/roachtest/tests/mixed_version_backup.go line 1043 at r1 (raw file):

	// Choose a random aost between min and max
	restoreAOST := hlc.Timestamp{
		WallTime: rng.Int63n(max.WallTime-min.WallTime) + min.WallTime,

For consideration: instead of a completely arbitrary timestamp in this range (which could be large), consider dividing this duration in N parts (say, N=5) and picking the time at the beginning of one of these slices. I believe this should be pretty sufficient in terms of correctness coverage while preserving some sense of similarity of runs when the same rng is used.

With the current implementation, each run will pick a potentially wildly different restoreAOST (since the Int63n argument will be different) even if run with the same seed. Say a bug only happens when a timestamp close to the full backup time is picked: re-running the test with the same seed will not help focus on that scenario.

---

pkg/cmd/roachtest/tests/mixed_version_backup.go line 2254 at r1 (raw file):

	restoreStmt := fmt.Sprintf(
		"%s FROM LATEST IN '%s'%s %s",
		restoreCmd, bc.uri(), aostClause, optionsStr,

Nit: you don't need the aostClause variable, you can just call aostFor(bc.restoreAOST) here.

[renatolabs]

While you have your hands in this test, maybe you can sneak in this request I had made previously 😄 Or maybe I'm the only one who misses that, hehe.

#112373 (comment)

[msbutler]

done

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @adityamaru and @renatolabs)

---

pkg/cmd/roachtest/tests/mixed_version_backup.go line 1043 at r1 (raw file):

Previously, renatolabs (Renato Costa) wrote…

For consideration: instead of a completely arbitrary timestamp in this range (which could be large), consider dividing this duration in N parts (say, N=5) and picking the time at the beginning of one of these slices. I believe this should be pretty sufficient in terms of correctness coverage while preserving some sense of similarity of runs when the same rng is used.

With the current implementation, each run will pick a potentially wildly different restoreAOST (since the Int63n argument will be different) even if run with the same seed. Say a bug only happens when a timestamp close to the full backup time is picked: re-running the test with the same seed will not help focus on that scenario.

ah very good call. fixed.

---

pkg/cmd/roachtest/tests/mixed_version_backup.go line 2254 at r1 (raw file):

Previously, renatolabs (Renato Costa) wrote…

Nit: you don't need the aostClause variable, you can just call aostFor(bc.restoreAOST) here.

done

[renatolabs]

Thank you!

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @adityamaru)

[msbutler]

TFTR!

bors r=renatolabs

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)

[renatolabs]

blathers backport 23.1

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

error creating merge commit from acae46b to blathers/backport-release-23.1-113534: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 23.1 failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[renatolabs]

Ah, it's missing #112356. @msbutler thoughts on backporting both to 23.1?