Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kvserver: add roachtests for raft memory pressure #111259

Open
pav-kv opened this issue Sep 26, 2023 · 1 comment
Open

kvserver: add roachtests for raft memory pressure #111259

pav-kv opened this issue Sep 26, 2023 · 1 comment
Assignees
Labels
A-kv-replication Relating to Raft, consensus, and coordination. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv KV Team

Comments

@pav-kv
Copy link
Collaborator

pav-kv commented Sep 26, 2023

A CRDB node can host (tens of) thousands of Raft instances, which operate semi-independently. Each instance has a limited footprint, e.g. it can pull up to a certain amount of data in memory, and have a certain amount of in-flight messages. There is no node-wide resource limit for Raft, and, as a result, under certain circumstances (#73376, #102840, #105338) a node gets overloaded and OOMs. There is a work track to add memory limits and prevent these overflows.

In the meantime, there are a number of tests (such as #110764) susceptible to this issue, and failing occasionally. We should have a roachtest that reproduces such high-memory-usage scenarios more reliably, and use it to measure and guide improvements.

Jira issue: CRDB-31840

Epic CRDB-39898

@pav-kv pav-kv added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-kv-replication Relating to Raft, consensus, and coordination. T-kv-replication labels Sep 26, 2023
@pav-kv pav-kv self-assigned this Sep 26, 2023
@blathers-crl
Copy link

blathers-crl bot commented Sep 26, 2023

cc @cockroachdb/replication

@exalate-issue-sync exalate-issue-sync bot added T-kv KV Team and removed T-kv-replication labels Jun 28, 2024
@github-project-automation github-project-automation bot moved this to Incoming in KV Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-kv-replication Relating to Raft, consensus, and coordination. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv KV Team
Projects
No open projects
Status: Incoming
Development

No branches or pull requests

1 participant