release-22.2.0: release-22.2: release-22.1: backupccl: reintroduce previously offline tables with manifest.DescriptorChanges

[blathers-crl]

Backport 2/2 commits from #89102 on behalf of @msbutler.

/cc @cockroachdb/release

---

Backport 1/2 commits from #88488.

/cc @cockroachdb/release

---

Backport 1/1 commits from #87312.

/cc @cockroachdb/release

---

getReintroducedSpans finds all tables included in the current and previous
backup that may have undergone a non-mvcc operation. The current backup will
then back up these tables' spans from ts = 0, as the previous backup may have
missed certain non-mvcc operations (e.g. ClearRange, non-mvcc AddSSTable).

To find these tables, getReintroducedSpans must find all tables covered in the
previous backup that were in the offline state at previous backup start time.

This function assumed that the previous backup's
manifest.Descriptors field would contain all tables covered in the previous
backup; however, while investigating #88042, we discovered that this assumption
is not correct in pre 22.2 branches. During revision history backups, a table
with an in-progress import (e.g. offline at backup time) can get backed up and
included in manifest.DescriptorChanges but not in manifest.Descriptors. This
implies getReintroducedSpans missed reintroducing spans from this table,
implying that backup chains have missed backing up some data.

It's worth noting this bug could only affect a backup chain in 22.2 iff the
backup that observed the table go offline was taken on a 22.1
cluster. Because 22.2 backups explicitly back up all offline tables, all tables
in manifest.DescriptorChanges should also be in manifest.Descriptors. In other
words, before this patch, the following sequence could lead to a corrupt backup
chain:

• t0: begin IMPORT on foo, on a pre 22.2 cluster
• t1: conduct a revision history backup of foo
• t2: complete or cancel the import of foo, via a non-mvcc operation
• t3: upgrade the cluster to 22.2
• t4: conduct a revision history incremental backup of foo
  • foo's span will not get re-introduced

This patch fixes getReintroducedSpans to ensure it reintroduces tables included
in manifest.DescriptorChanges whose last revision brought the table offline. In
addition, this patch adds a check to restore planning which causes the restore
to fail immedidiately if any restore target is missing an introduced span. The
patch tests this check on a corrupt backup taken on v21.2.

Release note(bug fix): fix apart of Tech Advisory
https://cockroachlabs.atlassian.net/browse/TSE-198

Release justification: bug fix

---

Release justification:

[blathers-crl]

Thanks for opening a backport.

Please check the backport criteria before merging:

• Patches should only be created for serious issues or test-only changes.
• Patches should not break backwards-compatibility.
• Patches should change as little code as possible.
• Patches should not change on-disk formats or node communication protocols.
• Patches should not add new functionality.
• Patches must not add, edit, or otherwise modify cluster versions; or add version gates.

If some of the basic criteria cannot be satisfied, ensure that the exceptional criteria are satisfied within.

• There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
• The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
• New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters.
• The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.

Add a brief release justification to the body of your PR to justify this backport.

Some other things to consider:

• What did we do to ensure that a user that doesn’t know & care about this backport, has no idea that it happened?
• Will this work in a cluster of mixed patch versions? Did we test that?
• If a user upgrades a patch version, uses this feature, and then downgrades, what happens?

[cockroach-teamcity]

This change is