Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kvserver: track bytes size of raft receive queue #82144

Closed
lunevalex opened this issue May 31, 2022 · 0 comments · Fixed by #80648
Closed

kvserver: track bytes size of raft receive queue #82144

lunevalex opened this issue May 31, 2022 · 0 comments · Fixed by #80648
Assignees
@tbg
Labels
A-kv-observability A-kv-replication Relating to Raft, consensus, and coordination. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) sync-me sync-me-5

Comments

@lunevalex
Copy link
Collaborator

lunevalex commented May 31, 2022 •
edited by cockroach-jira-scripts
Loading

In #80155 we observed OOMs below raft. It has been diffcult diagnosing these issues due to a lack of observability. We should add metrics about the state of the raft queue to help understand how utilized the queues are.

cr.store.raft.rcvd.queued_bytes: gauge (sum of size of all entries waiting to be handed to raft)
cr.store.raft.rcvd.stepped_bytes: counter (sum of size of all entries handed to RawNode.Step)
cr.store.raft.rcvd.dropped_bytes: counter (sum of size of all entries that were dropped because recv queue filled up)

Jira issue: CRDB-16464
@lunevalex lunevalex added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-kv-replication Relating to Raft, consensus, and coordination. A-kv-observability T-kv-replication labels May 31, 2022
@lunevalex lunevalex linked a pull request May 31, 2022 that will close this issue
kvserver: track bytes size of raft receive queue #80648
Merged
@mari-crl mari-crl added sync-me and removed sync-me labels Jun 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tbg tbg

Labels
A-kv-observability A-kv-replication Relating to Raft, consensus, and coordination. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) sync-me sync-me-5
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kvserver: track bytes size of raft receive queue tbg/cockroach
4 participants
@jlinder @tbg @lunevalex @mari-crl