producer performance Question

[write2jaydeep]

Read the FAQ first: https://github.com/edenhill/librdkafka/wiki/FAQ

Description

Hi I have been using rdkafka_performance.c and trying to catch bottleneck if any.

I did two cases:

1. original rdkafka_performance.c which is sending 31 bytes at that time getting throughput
   around 65k messages/second
   2)I have modified it to send 5111bytes string instead of 31 bytes at that time getting throughput around 7500 messages/second

both Kafka broker and librdkafka are running on separate GCP VM

can you please elaborate on why it is decreasing drastically??

here I have attached screenshots
https://prnt.sc/nz7eqf
https://prnt.sc/nz7fbf

Thanks
JD

IMPORTANT: Always try to reproduce the issue on the latest released version (see https://github.com/edenhill/librdkafka/releases), if it can't be reproduced on the latest version the issue has been fixed.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): 1.0.0
• Apache Kafka version: 1.1.1
• librdkafka client configuration: default
• Operating system: Ubantu Linux

[edenhill]

https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#high-throughput

[write2jaydeep]

@edenhill Thank you! I read the same link you have suggested.
here I even tried with different configuration/combination of linger.ms and batch.num.messages to see the difference.
but the question I do have is that why does it make a difference performance with 31 bytes and 5111bytes (more than 31 bytes) with the same set of configuration?

let me enable msg debug and send you the outputs for both cases.

Thanks
JD

[write2jaydeep]

I have a similar line of problem issue no

[write2jaydeep]

@edenhill
okay, I got it. we only can achieve max 3.8MB/s speed through librdkafka.(on openVOS platform)
now next scenario happens as follow:
I set the following

message.max.bytes=10000000
queue.buffering.max.ms=1000
batch.num.messages=20000

my application thread is too fast to produce msgs(3000msgs/second) into librdkafka queue, at the result LOCAL Queue max out. I tried to increase queue.buffering.max.messages=300000;queue.buffering.max.kbytes=1572864 but application thread too fast, at the result application running out of memory resources (max out 1GB memory) because of response rates, are too slow (in my platform)

can you please tell me how to handle Queue pressure to throttled it down?

Thanks
JD

[edenhill]

This sounds like a network congestion or thread contention issue, probably the latter.
OpenVOS is not a supported platform, so you will need to do your own profiling of the code to see where it is spending time.

[write2jaydeep]

@edenhill yup, I feel the same. Thank you!

[write2jaydeep]

@edenhill I know openVOS is not supported platform. it would be very grateful if you can throw some lights on it.
if I run multiple rdkafka_performance programs(two terminals) on the same machine, on both instances I am getting 3.8MB/S transfer rate. if the network would the problem it should be less (1/2 on both)

I appreciate your help!

Thanks
JD

[edenhill]

Run a CPU profiler, whatever exists for OpenVOS 🤷‍♂ Such as -pg / gmon

[write2jaydeep]

@edenhill I guess, -pg or gmon is not available on VOS. later I will definitely look for other options if possible for profiling.
Thank you for your valuable support! I figured out work-around. I will run multiple application which will push messages in a round-robin fashion to the same Kafka cluster to achieve higher throughput. this works well. [I am able to get almost 2X performance]

Thanks
JD