[QUERY]EventProcessorClientBuilder.processEventBatch - Should max wait time make a big difference in performance

[mzesri]

Query/Question
I am writing an app that streams events from event hubs ranging from 1 event per second or lower to 10k events per second. If I use .processEvent, I run into a 2500 event limit. I think it is probably because the processEvent call back can only be called 2500 times per second. I don't know if there is a way to change that. I am happy to see the .processEventBatch method which significantly improved the performance in high volume streaming. Because I also have to handle low volume events, I thought I'd set the max wait time to 1 second. This does allow me to stream the slow events in a timely fashion. Then I saw a huge performance difference between using any max wait time and not using max wait time at all. Using a max wait time could be 10 times slower or up to 50 times slower. I am puzzled by this because none of my batches should last longer than a second. Regardless of how I set the max wait time, 100 milliseconds, 1 second or 10 seconds, I experience the same performance hit.

There are lots of logs like this:

[2021-06-05T14:37:39,017][INFO ][c.a.m.e.i.AmqpReceiveLinkProcessor] [reactor-executor-4est_thread_info] linkName[3_a01b10_1622928756785] entityPath[globalradaresritest/ConsumerGroups/esrifeatureservice/Partitions/3] credits[1] Link is empty. Adding more credits.

[2021-06-05T14:37:39,017][INFO ][c.a.c.a.i.ReactorReceiver] [reactor-executor-4est_thread_info] connectionId[MF_965874_1622928756784] linkName[3_a01b10_1622928756785] adding credits[0]

[2021-06-05T14:37:39,041][INFO ][c.a.m.e.i.AmqpReceiveLinkProcessor] [reactor-executor-4est_thread_info] linkName[3_a01b10_1622928756785] entityPath[globalradaresritest/ConsumerGroups/esrifeatureservice/Partitions/3] credits[1] Link is empty. Adding more credits.

[2021-06-05T14:37:39,041][INFO ][c.a.c.a.i.ReactorReceiver] [reactor-executor-4est_thread_info] connectionId[MF_965874_1622928756784] linkName[3_a01b10_1622928756785] adding credits[0]

[2021-06-05T14:37:39,061][INFO ][c.a.m.e.i.AmqpReceiveLinkProcessor] [reactor-executor-4est_thread_info] linkName[3_a01b10_1622928756785] entityPath[globalradaresritest/ConsumerGroups/esrifeatureservice/Partitions/3] credits[1] Link is empty. Adding more credits.

[2021-06-05T14:37:39,062][INFO ][c.a.c.a.i.ReactorReceiver] [reactor-executor-4est_thread_info] connectionId[MF_965874_1622928756784] linkName[3_a01b10_1622928756785] adding credits[0]

[2021-06-05T14:37:39,082][INFO ][c.a.m.e.i.AmqpReceiveLinkProcessor] [reactor-executor-4est_thread_info] linkName[3_a01b10_1622928756785] entityPath[globalradaresritest/ConsumerGroups/esrifeatureservice/Partitions/3] credits[1] Link is empty. Adding more credits.

[2021-06-05T14:37:39,082][INFO ][c.a.c.a.i.ReactorReceiver] [reactor-executor-4est_thread_info] connectionId[MF_965874_1622928756784] linkName[3_a01b10_1622928756785] adding credits[0]

Don't know if this is normal or if there is something wrong on my side. BTW, this event hub has 4 partitions. The performance difference exists whether I use an in-memory checkpoint store or a blob store.

Why is this not a Bug or a feature Request?

1. not sure if this is expected.
2. not sure if there is something wrong on my side or if there is a workaround.

Setup (please complete the following information if applicable):

• OS: [e.g. iOS]. iOS
• IDE: [e.g. IntelliJ]. IntelliJ
• Library/Libraries: [e.g. com.azure:azure-core:1.16.0 (groupId:artifactId:version)] azure-messaging-eventhubs 5.7.1

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Query Added
• Setup information Added

[alzimmermsft]

@conniey could you take a look at this issue

[conniey]

There is a performance degradation bug (#20841) when we were stabilising the SDK. The max wait time should not make a difference in performance. All that happens behind the scenes is that it accumulates events during that time period and then emits them when the time period has elapsed.

[ghost]

Hi @mzesri. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.