backupccl: reintroduce previously offline tables with manifest.DescriptorChanges #89981
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
msbutler
force-pushed
the
butler-master-reintro
branch
from
a61551e to
f23ce37
Compare
msbutler
changed the title
msbutler
requested review from
rhu713,
stevendanna and
dt
and removed request for
a team and
rhu713
|
this is a trivial forward port of the 22.2 patch to master. just needs a stamp! |
stevendanna
approved these changes
Oct 19, 2022
|
TFTR! bors r=stevendanna |
|
Build failed: |
|
bor r- |
|
bors r- |
…ptorChanges getReintroducedSpans finds all tables included in the current and previous backup that may have undergone a non-mvcc operation. The current backup will then back up these tables' spans from ts = 0, as the previous backup may have missed certain non-mvcc operations (e.g. ClearRange, non-mvcc AddSSTable). To find these tables, getReintroducedSpans must find all tables covered in the previous backup that were in the offline state at previous backup start time. This function assumed that the previous backup's manifest.Descriptors field would contain all tables covered in the previous backup; however, while investigating cockroachdb#88042, we discovered that this assumption is not correct in pre 22.2 branches. During revision history backups, a table with an in-progress import (e.g. offline at backup time) can get backed up and included in manifest.DescriptorChanges but not in manifest.Descriptors. This implies getReintroducedSpans missed reintroducing spans from this table, implying that backup chains have missed backing up some data. It's worth noting this bug could only affect a backup chain in 22.2 iff the backup that observed the table go offline was taken on a 22.1 cluster. Because 22.2 backups explicitly back up all offline tables, all tables in manifest.DescriptorChanges should also be in manifest.Descriptors. In other words, before this patch, the following sequence could lead to a corrupt backup chain: - t0: begin IMPORT on foo, on a pre 22.2 cluster - t1: conduct a revision history backup of foo - t2: complete or cancel the import of foo, via a non-mvcc operation - t3: upgrade the cluster to 22.2 - t4: conduct a revision history incremental backup of foo - foo's span will not get re-introduced This patch fixes getReintroducedSpans to ensure it reintroduces tables included in manifest.DescriptorChanges whose last revision brought the table offline. In addition, this patch adds a check to restore planning which causes the restore to fail immedidiately if any restore target is missing an introduced span. The patch tests this check on a corrupt backup taken on v21.2. Release note(bug fix): fix apart of Tech Advisory https://cockroachlabs.atlassian.net/browse/TSE-198 Release justification: bug fix
msbutler
force-pushed
the
butler-master-reintro
branch
from
f23ce37 to
d80f505
Compare
|
bors r=adityamaru |
|
Build succeeded: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
getReintroducedSpans finds all tables included in the current and previous
backup that may have undergone a non-mvcc operation. The current backup will
then back up these tables' spans from ts = 0, as the previous backup may have
missed certain non-mvcc operations (e.g. ClearRange, non-mvcc AddSSTable).
To find these tables, getReintroducedSpans must find all tables covered in the
previous backup that were in the offline state at previous backup start time.
This function assumed that the previous backup's
manifest.Descriptors field would contain all tables covered in the previous
backup; however, while investigating #88042, we discovered that this assumption
is not correct in pre 22.2 branches. During revision history backups, a table
with an in-progress import (e.g. offline at backup time) can get backed up and
included in manifest.DescriptorChanges but not in manifest.Descriptors. This
implies getReintroducedSpans missed reintroducing spans from this table,
implying that backup chains have missed backing up some data.
It's worth noting this bug could only affect a backup chain post 22.1 iff the
backup that observed the table go offline was taken on a 22.1
cluster. Because post 22.1 backups explicitly back up all offline tables, all tables
in manifest.DescriptorChanges should also be in manifest.Descriptors. In other
words, before this patch, the following sequence could lead to a corrupt backup
chain:
This patch fixes getReintroducedSpans to ensure it reintroduces tables included
in manifest.DescriptorChanges whose last revision brought the table offline. In
addition, this patch adds a check to restore planning which causes the restore
to fail immedidiately if any restore target is missing an introduced span. The
patch tests this check on a corrupt backup taken on v21.2.
Fixes #88042
Release note(bug fix): fix apart of Tech Advisory
https://cockroachlabs.atlassian.net/browse/TSE-198