rdkafka_performance: Don't sleep while waiting for delivery reports #1918
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
edenhill
suggested changes
Aug 5, 2018
examples/rdkafka_performance.c
Outdated
| if (rate_sleep) { | ||
| rd_ts_t next = rd_clock() + (rd_ts_t) rate_sleep; | ||
| do { | ||
| rd_kafka_poll(rk, RD_MIN(100, rate_sleep)); |
There was a problem hiding this comment.
rd_kafka_poll()'s timeout is in milliseconds while rate_sleep is in microseconds.
There was a problem hiding this comment.
I suggest calculating the remaining time (of rate_sleep) in each loop and using that for timeout, which removes the need for RD_MIN()
There was a problem hiding this comment.
Done -- but now I ended up with an RD_MAX to avoid polling with a negative timeout :)
ankon
force-pushed
the
pr/perf-low-message-rate-latency
branch
from
4ee1063 to
8fbea5a
Compare
edenhill
suggested changes
Aug 9, 2018
examples/rdkafka_performance.c
Outdated
| if (rate_sleep) { | ||
| rd_ts_t next = rd_clock() + (rd_ts_t) rate_sleep; | ||
| do { | ||
| int poll_timeout = RD_MAX(0, (next - rd_clock()) / 1000); |
There was a problem hiding this comment.
this variable is only used once, move the right side directly into poll() argument
There was a problem hiding this comment.
need int cast as so:
(int)((next - rd_clock()) / 1000)
There was a problem hiding this comment.
Done, and re-indented to avoid a long line.
Sleeping here means we won't be looking for delivery reports, which in latency mode is problematic: The latency is calculated in the delivery report callback based on the time when that is called, which would be offset by the slept time.
ankon
force-pushed
the
pr/perf-low-message-rate-latency
branch
from
8fbea5a to
b9f3de2
Compare
edenhill
approved these changes
Aug 9, 2018
|
Thank you for this! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Sleeping here means we won't be looking for delivery reports, which in latency mode
is problematic: The latency is calculated in the delivery report callback based on the time
when that is called, which would be offset by the slept time.
In my case I'm using the tool with low message rates (1-10 messages/s), and the latencies reported
by the producer ended up between 1s (1 message/s) and 100ms (10 messages/s). Looking into this
showed that the sleeping for the
rate_sleepmeans that the tool doesn't process deliveryreports until it has slept -- but the latency is based on the actual time of running the delivery report callback,
not the time the response was accepted by rdkafka.
Fix this problem by replacing the sleep with a busy loop that polls (with a hand-wavy timeout of at most 100ms).
I opted for not adding a
latency_modebranch, but just do the polling in either case when the rate limiter is enabled.