Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory bound number of open SSTs in restore data processor #102722

Closed
rhu713 opened this issue May 2, 2023 · 1 comment · Fixed by #93324
Closed

Memory bound number of open SSTs in restore data processor #102722

rhu713 opened this issue May 2, 2023 · 1 comment · Fixed by #93324
Assignees
Labels
A-disaster-recovery C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-disaster-recovery

Comments

@rhu713
Copy link
Contributor

rhu713 commented May 2, 2023

Currently, there's no limit on the number of external SST readers that can be open in the restore data processor. Because the unit of processing in the restore data processor is an import span and all of the files that overlap that span, it is possible that for restores of backups with many incremental layers, the overhead of the SST readers on all of the simultaneously open files is too large to fit in available memory. During ad-hoc runs of restore on fixtures with many incremental layers, it's been observed occasionally that the restore would fail because of OOMs of this nature.

We should address this by associating the opening of an external SST reader with a memory cost, and then bound the total memory that could be used by adding a memory monitor to the restore data processor.

Jira issue: CRDB-27615

@rhu713 rhu713 added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-disaster-recovery T-disaster-recovery labels May 2, 2023
@rhu713 rhu713 self-assigned this May 2, 2023
@blathers-crl
Copy link

blathers-crl bot commented May 2, 2023

cc @cockroachdb/disaster-recovery

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-disaster-recovery C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-disaster-recovery
Projects
No open projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant