Memory bound number of open SSTs in restore data processor #102722
Labels
A-disaster-recovery
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-disaster-recovery
Currently, there's no limit on the number of external SST readers that can be open in the restore data processor. Because the unit of processing in the restore data processor is an import span and all of the files that overlap that span, it is possible that for restores of backups with many incremental layers, the overhead of the SST readers on all of the simultaneously open files is too large to fit in available memory. During ad-hoc runs of restore on fixtures with many incremental layers, it's been observed occasionally that the restore would fail because of OOMs of this nature.
We should address this by associating the opening of an external SST reader with a memory cost, and then bound the total memory that could be used by adding a memory monitor to the restore data processor.
Jira issue: CRDB-27615
The text was updated successfully, but these errors were encountered: