WIFI-13597: fix: modified kafka manager to use poll in producer

[i-chvets]

Description

During latest experiments with large number of APs, we narrowed down that memory is consumed by Kafka internal queue on GW (producer). And with large number of messages producer cannot keep up with emptying this queue.
One noticeable suspect was identified in flush() call here
Looks like, flushing on every message slows down producer to 100 messages per second.

The solution was to use poll() to allow for faster message transmission in peak times.

Related Jira: https://telecominfraproject.atlassian.net/browse/WIFI-13597

Summary of changes:

• Modified code in KafkaManager to use poll instead of flush for every messages sent. flush is used only on empty internal notification queue in idle times.

[stephb9959]

Have you tested this with very few devices: < 5 for example. We tried this method in the past, however we found messages were lingering in Kafka and not being received by other consumers on the bus. The fix looks good, just want to make sure we are not trading on problem for another.

[i-chvets]

Have you tested this with very few devices: < 5 for example. We tried this method in the past, however we found messages were lingering in Kafka and not being received by other consumers on the bus. The fix looks good, just want to make sure we are not trading on problem for another.

I did experiments and the queue goes down to zero pretty often, that where flush() is executed. It looks like messages are being sent in batches of 10, which is fast enough to empty the queue.
Even with 10K APs flush() was triggered multiple times a second under load (will confirm with 25K APs test as well). The rate of messages is still 100/second or so, but I guess the use of poll() on every message and executing flush() on bunches of ~10 messages improves latency.

Memory did not go above 700MB for the whole hour of test.

I ran 1000 and 5000 APs tests for short time, all looked good. What is the indication of Kafka messages issue?
I will get 5 APs simulation running for 1 hour. What should I look for to confirm it is working properly?

[stephb9959]

Have you tested this with very few devices: < 5 for example. We tried this method in the past, however we found messages were lingering in Kafka and not being received by other consumers on the bus. The fix looks good, just want to make sure we are not trading on problem for another.

I did experiments and the queue goes down to zero pretty often, that where flush() is executed. It looks like messages are being sent in batches of 10, which is fast enough to empty the queue. Even with 10K APs flush() was triggered multiple times a second under load (will confirm with 25K APs test as well). The rate of messages is still 100/second or so, but I guess the use of poll() on every message and executing flush() on bunches of ~10 messages improves latency.

Memory did not go above 700MB for the whole hour of test.

I ran 1000 and 5000 APs tests for short time, all looked good. What is the indication of Kafka messages issue? I will get 5 APs simulation running for 1 hour. What should I look for to confirm it is working properly?

What we have seen was that Kafka was a long delay between publish and consume: 30 seconds to 1 minute. If you are not seeing that, then this is a good fix.

[i-chvets]

Results of 25,000 AP simulation 1 hour test. Memory never grows beyond 1.5GB

[stephb9959]

LGTM