Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ui: Hot Ranges page doesn't show top ranges by QPS (instead by CPU usage) #99605

Closed
koorosh opened this issue Mar 26, 2023 · 4 comments · Fixed by #99716
Closed

ui: Hot Ranges page doesn't show top ranges by QPS (instead by CPU usage) #99605

koorosh opened this issue Mar 26, 2023 · 4 comments · Fixed by #99716
Assignees
@kvoli
Labels
A-cluster-observability Related to cluster observability branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. GA-blocker O-community Originated from the community X-blathers-triaged blathers was able to find an owner

Comments

@koorosh
Copy link
Collaborator

koorosh commented Mar 26, 2023 •
edited by cockroach-jira-scripts
Loading

Describe the problem

Initially, load balancing between replicas relies on QPS and replicaRanking collected stats for top
hottest replicas by QPS. The same stats are used by Hot Ranges page in Db Console to show ranges
and it is defined as:

The Hot ranges list displays the ranges with the highest queries per second (QPS) from each node store.

in docs

Following change 8b202b9 switched load balancing to rely on CPU usage instead and it in turn affected the list of
ranges that are used in Db Console, now it shows hottest ranges by CPU instead.

Environment:

  • CockroachDB version: starting from 23.1

Related to #98619

Jira issue: CRDB-25973
@koorosh koorosh added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-cluster-observability A-cluster-observability Related to cluster observability labels Mar 26, 2023
@blathers-crl
Copy link

blathers-crl bot commented Mar 26, 2023

Hello, I am Blathers. I am here to help you get the issue triaged.

Hoot - a bug! Though bugs are the bane of my existence, rest assured the wretched thing will get the best of care here.

I have CC'd a few people who may be able to assist you:

If we have not gotten back to your issue within a few business days, you can try the following:

  • Join our community slack channel and ask on #cockroachdb.
  • Try find someone from here if you know they worked closely on the area and CC them.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl bot added O-community Originated from the community X-blathers-triaged blathers was able to find an owner labels Mar 26, 2023
@koorosh koorosh mentioned this issue Mar 26, 2023
Closed
kvoli added a commit to kvoli/cockroach that referenced this issue Mar 27, 2023
@kvoli
kvserver: use qps for hot ranges sorting
bde4b6c 
We introduced CPU balancing by default in cockroachdb#97424. This had the side
effect of changing the hot ranges api to return the hottest replicas by
CPU, rather than QPS.

This patch updates the replica rankings struct to support tracking both
by CPU and QPS simultaneously. The hot ranges API collects the top k by
QPS and the store rebalancer collects depending on the setting of
`kv.allocator.load_based_rebalancing.objective`, which is by default
`cpu`.

Resolves: cockroachdb#99605

Release note (bug fix): The hot ranges UI page would show hot ranges by
CPU and not QPS, depending on the value of
`kv.allocator.load_based_rebalancing.objective` (default `cpu`). Now the
UI page will always collect based on QPS.
@kvoli kvoli mentioned this issue Mar 27, 2023
Merged
kvoli added a commit to kvoli/cockroach that referenced this issue Mar 28, 2023
@kvoli
kvserver: use qps for hot ranges sorting
d00eb94 
We introduced CPU balancing by default in cockroachdb#97424. This had the side
effect of changing the hot ranges api to return the hottest replicas by
CPU, rather than QPS.

This patch updates the replica rankings struct to support tracking both
by CPU and QPS simultaneously. The hot ranges API collects the top k by
QPS and the store rebalancer collects depending on the setting of
`kv.allocator.load_based_rebalancing.objective`, which is by default
`cpu`.

Resolves: cockroachdb#99605

Release note (bug fix): The hot ranges UI page would show hot ranges by
CPU and not QPS, depending on the value of
`kv.allocator.load_based_rebalancing.objective` (default `cpu`). Now the
UI page will always collect based on QPS.
craig bot pushed a commit that referenced this issue Mar 30, 2023
@renatolabs @kvoli
Merge #99669 #99716
7074da6 
99669: roachtest: bump jepsen version r=smg260 a=renatolabs

This commit bumps the jepsen version now to include a fix that allows
jepsen to set custom range sizes. See:

cockroachdb/jepsen#34

Epic: none

Release note: None

99716: kvserver: use qps for hot ranges sorting r=koorosh a=kvoli

We introduced CPU balancing by default in #97424. This had the side effect of changing the hot ranges api to return the hottest replicas by CPU, rather than QPS.

This patch updates the replica rankings struct to support tracking both by CPU and QPS simultaneously. The hot ranges API collects the top k by QPS and the store rebalancer collects depending on the setting of `kv.allocator.load_based_rebalancing.objective`, which is by default `cpu`.

Resolves: #99605


Co-authored-by: Renato Costa <[email protected]>
Co-authored-by: Austen McClernon <[email protected]>
@craig craig bot closed this as completed in 41e3497 Mar 30, 2023
@blathers-crl blathers-crl bot mentioned this issue Mar 30, 2023
Merged
blathers-crl bot pushed a commit that referenced this issue Mar 30, 2023
@kvoli
kvserver: use qps for hot ranges sorting
24d06ef 
We introduced CPU balancing by default in #97424. This had the side
effect of changing the hot ranges api to return the hottest replicas by
CPU, rather than QPS.

This patch updates the replica rankings struct to support tracking both
by CPU and QPS simultaneously. The hot ranges API collects the top k by
QPS and the store rebalancer collects depending on the setting of
`kv.allocator.load_based_rebalancing.objective`, which is by default
`cpu`.

Resolves: #99605

Release note (bug fix): The hot ranges UI page would show hot ranges by
CPU and not QPS, depending on the value of
`kv.allocator.load_based_rebalancing.objective` (default `cpu`). Now the
UI page will always collect based on QPS.
@kvoli kvoli reopened this Apr 3, 2023
@blathers-crl
Copy link

blathers-crl bot commented Apr 3, 2023

Hi @kvoli, please add branch-* labels to identify which branch(es) this release-blocker affects.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@kvoli kvoli self-assigned this Apr 3, 2023
@kvoli kvoli added the branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 label Apr 3, 2023
@kvoli
Copy link
Collaborator

kvoli commented Apr 3, 2023

Will be closed on #100211

@kvoli
Copy link
Collaborator

kvoli commented Apr 3, 2023

closed on #100211

@kvoli kvoli closed this as completed Apr 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kvoli kvoli

Labels
A-cluster-observability Related to cluster observability branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. GA-blocker O-community Originated from the community X-blathers-triaged blathers was able to find an owner
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kvserver: use qps for hot ranges sorting kvoli/cockroach
2 participants
@koorosh @kvoli