azure-spring-cloud-starter-eventhubs stops picking up events and won't start again till service restart

[J4cku]

Hi,
We observed a weird problem that is near impossible for us to debug.
After some time of working fine our EventHub listener stops picking up new events (no idea why).
Logs look fine as always, e.g.

2021-03-15 10:10:52.315  INFO [qiaspace-device-registration,,,] 1 --- [     elastic-15] c.a.m.e.PartitionBasedLoadBalancer       : Starting load balancer for 06bbbb0e-f94f-476d-b330-d3fc229922dc
2021-03-15 10:10:52.315  INFO [qiaspace-device-registration,,,] 1 --- [     elastic-15] c.a.m.e.PartitionBasedLoadBalancer       : CheckpointStore returned 2 ownership records
2021-03-15 10:10:52.315  INFO [qiaspace-device-registration,,,] 1 --- [     elastic-15] c.a.m.e.PartitionBasedLoadBalancer       : Event Hubs service returned 2 partitions
2021-03-15 10:10:52.316  INFO [qiaspace-device-registration,,,] 1 --- [     elastic-15] c.a.m.e.PartitionBasedLoadBalancer       : Number of active ownership records 2
2021-03-15 10:10:52.316  INFO [qiaspace-device-registration,,,] 1 --- [     elastic-15] c.a.m.e.PartitionBasedLoadBalancer       : Number of active event processors 1
2021-03-15 10:10:52.316  INFO [qiaspace-device-registration,,,] 1 --- [     elastic-15] c.a.m.e.PartitionBasedLoadBalancer       : Expected min partitions per event processor = 2, expected number of event processors with additional partition = 0
2021-03-15 10:10:52.316  INFO [qiaspace-device-registration,,,] 1 --- [     elastic-15] c.a.m.e.PartitionBasedLoadBalancer       : Load is balanced with this event processor owning 2 partitions
2021-03-15 10:10:52.318  INFO [qiaspace-device-registration,,,] 1 --- [     elastic-15] c.a.m.e.PartitionBasedLoadBalancer       : Load balancing completed successfully

Load balancing is done regularly, but for some reason until I restart the service it is not picking up new events.
I have literally no idea what can we do to help investigate the problem. Same thing happen with integration as well as binder.
It seams to have worked fine earlier when we used @AzureListener (but we moved to webflux, and that way does not work without spring-web)

[J4cku]

I kinda lied. @AzureMessageListener still works with webflux; we are trying that now

[joshfree]

@conniey could you please take a look?

[conniey]

This looks like it could be related to #18070. The spring starter uses azure-messaging-eventhubs under the hood.

[saragluna]

@J4cku which version of the starter are you using? Have you tried our latest release?

https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/spring/azure-spring-cloud-starter-eventhubs/CHANGELOG.md#release-history

[J4cku]

@saragluna I haven't as the version is not even present yet in maven central:

[saragluna]

@J4cku
Ah, you could try with version 2.2.0. We're in the process of releasing 2.3.0 today. But it's blocked by our eng pipeline.

[J4cku]

@saragluna I already did with 2.2.0 and it fails there

[wujack778]

@J4cku Is it convenient to post your source code address?

[saragluna]

@J4cku any update on this issue?

[J4cku]

@saragluna we ditched the binder implementation; we use now @AzureMessageListener implementation instead since that one does not get stuck

[saragluna]

@J4cku glad the listener approach works, but do you mind providing the pom file when using the binder? It will help us investigate what's wrong with the binder.

[conniey]

This is probably related to #18070. We released an update to the underlying library this morning.

[yiliuTo]

As mentioned by @conniey , this issue has been fixed in com.azure:azure-messaging-eventhubs;5.7.0. So you could use azure-spring-cloud-stream-binder-eventhubs 2.4.0 and forward. If the issue still exists, please open this issue again.