backupccl: generate fewer spans for protected timestamps

[ajwerner]

Describe the problem

The protected timestamp subsystem was designed, perhaps mistakenly, under an assumption that the total number and size of protected spans would be bounded and relatively small (megabytes at most). This assumption was partially coming from an assumption that the entity which would be protected is an entire table. Table spans tend to be quite small in terms of byte size as they are just a byte and an varint integer. In order to deal with interleaved tables appropriately, backup jobs generate spans for every table index. This can lead to a major explosion in the total number of spans. When too many spans are created, the record can exceed the default size of the limits of the protected timestamp subsystem. While the setting on the maximum number of spans now seems too small, at some point we'll begin to worry about the actual size of the records. While not a general and infinitely scalable solution, creating smaller sets of spans and then compressing those spans is likely to make for a very large improvement in the total number of spans generated.

As a first pass this means:

• Protecting the full table span for any non-interleaved table being backed up.
• Merging contiguous table spans into a single span.

The difficulty will be dealing with the edge cases of overlapping spans but I believe we have the technology for this. The above optimizations should lead to full cluster backup going from all index spans to very few table spans.

Epic CRDB-7078

[blathers-crl]

Hi @ajwerner, please add a C-ategory label to your issue. Check out the label system docs.

While you're here, please consider adding an A- label to help keep our repository tidy.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[ajwerner]

cc @adityamaru

[adityamaru]

Is my understanding correct that the only case we should see overlapping spans is when a tables index is interleaved in another table?
The code seems to suggest that when selecting the spans to backup, if any of the indexes of the current table are interleaved then we construct a span with the greatest ancestors {tableID, indexID}, which would overlap with one of the spans outputted when processing that ancestor table itself.

[ajwerner]

Is my understanding correct that the only case we should see overlapping spans is when a tables index is interleaved in another table?

That sounds right to me.

[adityamaru]

Just leaving a note for myself to think about a further optimization where we merge contiguous full table spans as well, on top of the work done in the above PR. Will need to think about the invariants to apply when deciding if two table spans should be merged or not.

[adityamaru]

We addressed part of this in #55794. We did attempt a change where we coalesced spans across table boundaries as well but the added complexity to backups span selection did not instill confidence in checking in that change.

We would like to pursue #70582 instead since cluster backups are what most customers use, and where this is probably most impactful. Database backups would also benefit from coalition across table boundaries but this is a lower priority in my opinion. I'm closing this issue in favor of the one linked above.