Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kvserver: load imbalance not observable from logs and debug.zip #107694

Closed
tbg opened this issue Jul 27, 2023 · 1 comment
Closed

kvserver: load imbalance not observable from logs and debug.zip #107694

tbg opened this issue Jul 27, 2023 · 1 comment
Labels
A-kv-distribution Relating to rebalancing and leasing. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv KV Team

Comments

@tbg
Copy link
Member

tbg commented Jul 27, 2023

I was investigating #106140 (comment) and it is obvious there is a load imbalance which the store rebalancer is unable to fix. From the roachtest artifacts, I was unable to determine which replicas were driving the load during the test; the store rebalancer logging stops just shy of being useful enough for this purpose.

The debug.zip shows some replicas with high CPU per second, however they are quiesced and they don't seem to have enough write activity to justify that they drove the load over the tens of minutes the test was running.

Printing, e.g., the topk drivers of cpu/sec whenever lease rebalancing can no longer make progress, or something like that, would've been helpful here.

Of course historical hot ranges would be even better. As far as I know, there is no way to get that from a debug.zip.

Jira issue: CRDB-30162

@tbg tbg added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-kv-distribution Relating to rebalancing and leasing. T-kv KV Team labels Jul 27, 2023
@kvoli
Copy link
Collaborator

kvoli commented Sep 7, 2023

The ask down the bottom here is the most useful, I'm going to close this issue in favor of #98108.

@kvoli kvoli closed this as completed Sep 7, 2023
@github-project-automation github-project-automation bot moved this to Closed in KV Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-kv-distribution Relating to rebalancing and leasing. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv KV Team
Projects
No open projects
Archived in project
Development

No branches or pull requests

2 participants