[BUG] Messages from Sessions are not FIFO when you abandon

[KnyGoo]

Describe the bug
When we receive multiple messages from one session, but at one point we abandon one of these messages. The messages are not consumed FIFO anymore

Exception or Stack Trace
2022-07-19 18:08:52.303 INFO 19388 --- [oundedElastic-2] b.s.c.a.servicebus.ProcessorContainer : Received message 9d0be482-dca9-4b93-b07b-b81e65c1f6f8 from all-to-documentfactoryservice.job
2022-07-19 18:08:52.323 INFO 19388 --- [oundedElastic-2] b.s.c.a.processor.SessionQueueDemo : Received message from session queue: 1
2022-07-19 18:08:53.390 INFO 19388 --- [oundedElastic-2] b.s.c.a.servicebus.ProcessorContainer : Received message e74b2cb6-3076-4fab-af00-15df5f1628b2 from all-to-documentfactoryservice.job
2022-07-19 18:08:53.390 INFO 19388 --- [oundedElastic-2] b.s.c.a.processor.SessionQueueDemo : Received message from session queue: 2
2022-07-19 18:08:54.460 INFO 19388 --- [oundedElastic-2] b.s.c.a.servicebus.ProcessorContainer : Received message 2dc4f7fe-1ab7-4b08-8d1f-b02e35ea7518 from all-to-documentfactoryservice.job
2022-07-19 18:08:54.461 INFO 19388 --- [oundedElastic-2] b.s.c.a.processor.SessionQueueDemo : Received message from session queue: 3
2022-07-19 18:08:55.496 INFO 19388 --- [oundedElastic-2] b.s.c.a.servicebus.ProcessorContainer : Received message 3007f257-76f8-43d7-b996-4e30a3cd4c09 from all-to-documentfactoryservice.job
2022-07-19 18:08:55.496 INFO 19388 --- [oundedElastic-2] b.s.c.a.processor.SessionQueueDemo : Received message from session queue: 4
2022-07-19 18:08:56.530 INFO 19388 --- [oundedElastic-2] b.s.c.a.servicebus.ProcessorContainer : Received message 9d0be482-dca9-4b93-b07b-b81e65c1f6f8 from all-to-documentfactoryservice.job
2022-07-19 18:08:56.530 INFO 19388 --- [oundedElastic-2] b.s.c.a.processor.SessionQueueDemo : Received message from session queue: 1
2022-07-19 18:08:57.563 INFO 19388 --- [oundedElastic-2] b.s.c.a.servicebus.ProcessorContainer : Received message e74b2cb6-3076-4fab-af00-15df5f1628b2 from all-to-documentfactoryservice.job
2022-07-19 18:08:57.563 INFO 19388 --- [oundedElastic-2] b.s.c.a.processor.SessionQueueDemo : Received message from session queue: 2
2022-07-19 18:08:58.600 INFO 19388 --- [oundedElastic-2] b.s.c.a.servicebus.ProcessorContainer : Received message 2dc4f7fe-1ab7-4b08-8d1f-b02e35ea7518 from all-to-documentfactoryservice.job
2022-07-19 18:08:58.600 INFO 19388 --- [oundedElastic-2] b.s.c.a.processor.SessionQueueDemo : Received message from session queue: 3
2022-07-19 18:08:59.733 INFO 19388 --- [oundedElastic-2] b.s.c.a.servicebus.ProcessorContainer : Received message 3007f257-76f8-43d7-b996-4e30a3cd4c09 from all-to-documentfactoryservice.job
2022-07-19 18:08:59.733 INFO 19388 --- [oundedElastic-2] b.s.c.a.processor.SessionQueueDemo : Received message from session queue: 4

To Reproduce
Send multiple messages with same session id. Abandon the message when receiving.

Expected behavior
Messages should still be FIFO

Setup (please complete the following information):

• OS: Windows
• IDE: IntelliJ
• Library/Libraries: com.azure:azure-messaging-servicebus:7.9.0
• Java version: 17
• Frameworks: Spring Boot

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

[joshfree]

@yiliuTo could you follow up with @KnyGoo on this github issue?

[yiliuTo]

Hi @joshfree , seems @KnyGoo is using com.azure:azure-messaging-servicebus:7.9.0 direclty while within the Spring Boot framework, thus I suppose @Azure/azsdk-sb-java can help to solve this issue.

[liukun-msft]

Hi @KnyGoo,

This is a know issue: #24064. The root cause has been addressed and I am working on a fix for the ordering issue when session number bigger than 1.

Can I know your receiver configurations, like prefetchCount, maxConcurrentSessions? (It will be better if you can provide the repro code snippet)

/cc @joshfree @yiliuTo

[KnyGoo]

Hi @liukun-msft ,

Thank you for the quick response. We are using Azure Service Bus in a professional environment and this is a major issue for us.

Some example code to reproduce:

public static void main(String[] args) throws Exception {
        ClientSecretCredential clientSecretCredential =
                new ClientSecretCredentialBuilder()
                        .tenantId("...")
                        .clientId("...")
                        .clientSecret("...")
                        .executorService(Executors.newCachedThreadPool())
                        .build();

        ServiceBusProcessorClient processorClient = new ServiceBusClientBuilder()
                .credential("...", clientSecretCredential)
                .transportType(AmqpTransportType.AMQP_WEB_SOCKETS)
                .sessionProcessor()
                .receiveMode(ServiceBusReceiveMode.PEEK_LOCK)
                .disableAutoComplete()
                .queueName("...")
                .maxConcurrentSessions(1)
                .maxConcurrentCalls(1)
                .maxAutoLockRenewDuration(Duration.ofSeconds(10))
                .processMessage(context -> {
                    LOGGER.info("Received message from session queue: {}", context.getMessage().getMessageId());
                    context.abandon();
                })
                .processError(context -> {})
                .buildProcessorClient();

        System.out.println("Starting the processor");
        processorClient.start();

        TimeUnit.SECONDS.sleep(200);
        System.out.println("Stopping and closing the processor");
        processorClient.close();
}

Output (we try 3 times):
11:17:22.553 [boundedElastic-2] INFO be.securex.commons.azureservicebusdemo.processor.SessionQueueDemo - Received message from session queue: 2975f29c-bb53-475e-a395-0cac998ebf09
11:17:22.618 [boundedElastic-2] INFO be.securex.commons.azureservicebusdemo.processor.SessionQueueDemo - Received message from session queue: 26587b25-920a-4ccb-ba83-74b68ac404a6
11:17:22.678 [boundedElastic-2] INFO be.securex.commons.azureservicebusdemo.processor.SessionQueueDemo - Received message from session queue: 2975f29c-bb53-475e-a395-0cac998ebf09
11:17:22.750 [boundedElastic-2] INFO be.securex.commons.azureservicebusdemo.processor.SessionQueueDemo - Received message from session queue: 26587b25-920a-4ccb-ba83-74b68ac404a6
11:17:22.819 [boundedElastic-2] INFO be.securex.commons.azureservicebusdemo.processor.SessionQueueDemo - Received message from session queue: 2975f29c-bb53-475e-a395-0cac998ebf09
11:17:22.875 [boundedElastic-2] INFO be.securex.commons.azureservicebusdemo.processor.SessionQueueDemo - Received message from session queue: 26587b25-920a-4ccb-ba83-74b68ac404a6
11:17:22.932 [boundedElastic-2] INFO be.securex.commons.azureservicebusdemo.processor.SessionQueueDemo - Received message from session queue: 581a9454-acc5-45f7-a332-63a0ad3391ef
11:17:22.993 [boundedElastic-2] INFO be.securex.commons.azureservicebusdemo.processor.SessionQueueDemo - Received message from session queue: c702b6a0-b855-4199-9213-39a7837643a2
11:17:23.050 [boundedElastic-2] INFO be.securex.commons.azureservicebusdemo.processor.SessionQueueDemo - Received message from session queue: 581a9454-acc5-45f7-a332-63a0ad3391ef
11:17:23.108 [boundedElastic-2] INFO be.securex.commons.azureservicebusdemo.processor.SessionQueueDemo - Received message from session queue: c702b6a0-b855-4199-9213-39a7837643a2

[liukun-msft]

Hi @KnyGoo, Thanks for providing the repro code. I'll check it and keep you updated.

[liukun-msft]

Hi @KnyGoo
We have fixed for this issue when maxConcurrentSession=1 and will be released on the mid of August. Please update your service bus to the latest version at that time. If you have any question, please let me know.

[KnyGoo]

Hi @liukun-msft ,

Will this issue also be fixed for maxConcurrentSessions > 1? Setting this to 1 in production environment will limit troughput....

At this moment we cofigure our service bus as follow:

• maxConcurrentCalls = 1 => otherwise message order is not quaranteed because messages from same session get picked up
• maxConcurrentSessions > 1 => otherwise troughput is very low

[liukun-msft]

Hi @KnyGoo
The fix for maxConcurrentSession>1 is still in progress and is tracked by issue #27336.

Can I know some background of how you create sessions and whether they are stable? As a walkaround, you can create multiple threads and use ServiceBusSessionReceiverAsyncClient to receive from each specific session.

Here is a sample: https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/servicebus/azure-messaging-servicebus/src/samples/java/com/azure/messaging/servicebus/ReceiveNamedSessionAsyncSample.java

[KnyGoo]

Hi @liukun-msft ,

You suggest creating ServiceBusSessionReceiverAsyncClient in an endless loop? Isn't this the reason why we are using processor instead of receiver? What if we loose connection, we should handle this ourselves?

Our use case:
We have different API's that each send a message with sessionId = unique identifier of some record in DB. We would like to handle these messages in the exact order (that's why we need sessions). We scale our application on multiple nodes, but would like to handle the same session on only one node to avoid conflicts. For some actions the order is also very important.

[liukun-msft]

Hi @KnyGoo

Processor is definitely recommended for your case.

However, because the fix for session process ordering issue is still in progress and may take some time for testing, if you don't want to wait, I think one possible solution is to use ServiceBusSessionReceiverAsyncClient to receive from each session in multiple threads. And right, you need to handle client error by yourself. (Receiver can still retry when connection loose, but if a non-retriable error is thrown out or retry exhausted, you need to handle that error and maybe call acceptSession() again to get a new receiver client to receive.)

Once issue #27336 is solved, you can switch to the processor. And I'll keep you updated for our progress.

[anuchandy]

This is a part of this work #33688 and work is still in progress.

[anuchandy]

Resolving this. Refer https://learn.microsoft.com/en-us/azure/developer/java/sdk/troubleshooting-messaging-service-bus-overview#upgrade-to-715x for details about version 7.15.x and how to opting-in new session rework.