kv: split rate limits and metrics for read and write requests #53510
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This makes the code cleaner and avoids the `ba.IsWrite()` call for read-only requests.
|
This change is |
nvanbenschoten
force-pushed
the
nvanbenschoten/readWriteRequests
branch
from
434d6de to
3bcec6c
Compare
This commit splits the existing request rate limit into two categories, read requests and write requests. Experimentation has shown that the fixed cost of a request is dramatically different between these two categories, primarily because write requests need to go through Raft while read requests do not. By splitting the limits and metrics along this dimension, we expect to be able to more accurately model the cost of KV traffic and more effectively tune rate limits. In making the split, the commit replaces the existing metric: ``` kv.tenant_rate_limit.requests_admitted ``` with the following two new metrics: ``` kv.tenant_rate_limit.read_requests_admitted kv.tenant_rate_limit.write_requests_admitted ``` The commit also replaced the existing two settings: ``` kv.tenant_rate_limiter.requests.rate_limit kv.tenant_rate_limiter.request.burst_limit ``` with the following four new settings: ``` kv.tenant_rate_limiter.read_requests.rate_limit kv.tenant_rate_limiter.read_requests.burst_limit kv.tenant_rate_limiter.write_requests.rate_limit kv.tenant_rate_limiter.write_requests.burst_limit ``` Release justification: Low-risk, high benefit change.
nvanbenschoten
force-pushed
the
nvanbenschoten/readWriteRequests
branch
from
3bcec6c to
4c9110f
Compare
ajwerner
approved these changes
Aug 26, 2020
There was a problem hiding this comment.
Reviewed 1 of 1 files at r1, 15 of 15 files at r2.
Reviewable status:complete! 1 of 0 LGTMs obtained
|
TFTR! bors r+ |
|
Build failed: |
|
PR description was missing release justification. bors r+ |
|
Build failed (retrying...): |
|
Build failed (retrying...): |
|
This PR was included in a batch that was canceled, it will be automatically retried |
|
Build succeeded: |
ajwerner
added a commit
to ajwerner/cockroach
that referenced
this pull request
Aug 31, 2020
This test was broken in cockroachdb#53510 which augmented the limiter machinery to deal with read and write requests independently. The problem is that the timer machinery works to synchronize the first blocked request but not subsequent requests. That leaves the test-writer with no fundamental means to ensure that subsequent blocked commands synchronize with a metrics command. This lack of synchronization on metrics was a pain-point when writing the original tests and led to excessive use of timers as synchronization. Rather than trying to add excess synchronization, this commit just adds a succeeds soon around the metrics. This works well. Fixes cockroachdb#53590 Release justification: non-production code changes Release note: None
craig bot
pushed a commit
that referenced
this pull request
Aug 31, 2020
53661: kvserver/tenantrate: fix flakey test r=nvanbenschoten a=ajwerner This test was broken in #53510 which augmented the limiter machinery to deal with read and write requests independently. The problem is that the timer machinery works to synchronize the first blocked request but not subsequent requests. That leaves the test-writer with no fundamental means to ensure that subsequent blocked commands synchronize with a metrics command. This lack of synchronization on metrics was a pain-point when writing the original tests and led to excessive use of timers as synchronization. Rather than trying to add excess synchronization, this commit just adds a succeeds soon around the metrics. This works well. Fixes #53590 Release justification: non-production code changes Release note: None Co-authored-by: Andrew Werner <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #53483.
This commit splits the existing request rate limit into two categories, read requests and write requests. Experimentation has shown that the fixed cost of a request is dramatically different between these two categories, primarily because write requests need to go through Raft while read requests do not. By splitting the limits and metrics along this dimension, we expect to be able to more accurately model the cost of KV traffic and more effectively tune rate limits.
In making the split, the commit replaces the existing metric:
with the following two new metrics:
The commit also replaced the existing two settings:
with the following four new settings:
Release justification: Low-risk, high benefit change.