[BUG] Service bus | Can't disable the prefetch count

[yuhaii]

Describe the bug
We found that the message will goto deadletter queue. The message lock duration is set at 20 seconds, max delivery count is set as 1.

After further investigation, we found that the prefetch is enabled. So when thread sleep 40 seconds ( delay(40L * 1000L) ), it continue peek message. But complete operation is after 40 seconds. So server side found those prefetched message lock lost after 20 seconds and moved them to DLQ.

private suspend fun queueReceiveWorker(receiverClient: IMessageReceiver) {
while (true) {
try {
delay(15L * 1000L)

        receiverClient.setPrefetchCount(0)
        logger.info("######## PrefetchCount: " + receiverClient.getPrefetchCount())
        val message: IMessage? = receiverClient.receiveAsync().await()

        if (message == null) { 
            logger.info("No message")
            delay(40L * 1000L) 
        } else { 
            val remainLockDuration = Duration.between(Instant.now(), message.lockedUntilUtc) 
            if (remainLockDuration < Duration.ofSeconds(5)) { 
                logger.info("Message ${message.messageId} has ${remainLockDuration}, less than 5 seconds!")
                if (!remainLockDuration.isNegative) { 
                    receiverClient.renewMessageLockAsync(message.lockToken).await()
                }
            }

            //delay(5L * 1000L)
            //receiverClient.renewMessageLockAsync(message.lockToken).await()
            //delay(5L * 1000L)
            //receiverClient.renewMessageLockAsync(message.lockToken).await()
            //delay(5L * 1000L)
            receiverClient.completeAsync(message.lockToken).await()
        }
    } catch (e: Exception) { 
        logger.info("Error in processing: ${e.message}")
        e.printStackTrace()
    }
}

}

According to SDK document the prefetch should be disabled by default. And when we tried to disable prefetch count, it didn't work. It still prefetch the message with prefetchcout == 0

Exception or Stack Trace
Message lock lost exception

To Reproduce
The test programs are built by gradle. If you want to build and try them, make sure you have JDK 8 (or later, e.g. 11) installed and the command “java” is in $PATH. Nothing else is required. Then just invoke gradlew.bat(In Windows) or gradlew(which is a shell script, in Linux) to build. The command line is: gradlew build. After you type gradlew build, gradle wrapper will automatically download gradle binaries and dependencies, and then build the project. In my build script, I set the output location of built jar archive to build/libs. You can find the built artifacts there. Use java –jar NameSpaceConnectionString QueueName to launch the built program. Make sure to set the queue’s max delivery count to 1, and lock duration to 20 seconds, then run the receiver first, wait 30 seconds, and run the sender at last. You can find many dead letter messages, caused by lock expire -> redeliver -> dead letter.

Below are my build, test steps:

1. install Java SDK 8 : jdk-8u271-windows-x64.exe
2. Run below command to build the source code:
   PS C:\Users\yuhai\Downloads\test-program\azure-service-bus-prefetch-demo> .\gradlew.bat build
3. Run below command to run the app:
   PS C:\Users\yuhai\Downloads\test-program\azure-service-bus-prefetch-demo\build\libs> java -jar .\azure-service-bus-prefetch-demo-0.0.1-SNAPSHOT.jar "Endpoint=sb://yh-sb-standard.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=rmvBkK/te8kXwEXw4AqrzEaKP9/eNYVLvJ1rGe/ErnY=" "testqueue1"
4. Wait a while till to run into :
   if (message == null) {
   logger.info("No message")
   delay(40L * 1000L)
5. Sending message to the hub with service bus explorer tool

Code Snippet

azure-service-bus-prefetch-demo.zip

Expected behavior

1. By default the prefetch should be disabled, NOT opening.
2. The receiverClient.setPrefetchCount(0) should able to disable prefetch

Screenshots

Setup (please complete the following information):

• OS: [e.g. iOS] Windows
• IDE : [e.g. IntelliJ]: Gradle
• Version of the Library used: microsoft-azure-servicebus 3.4.0

[yuhaii]

Hello @srnagar , could you please help checking this issue? thanks in advance.

[srnagar]

@yvgopal Could you please take a look at this issue?

[rui-ss]

Hi @srnagar , @yvgopal Please kindly help us check this issue and advise, thank you!

[aerisnju]

I created a simpler program azure-service-bus-prefetch-demo-simplified.zip to reproduce it. The simpler program does not use multi-threading or coroutines. The program just use its main thread to send and receive message from the same queue.

To run the program, first create a queue with the setting:

DeadLetterQueue: enabled
LockDuration: 20 seconds
MaxDeliveryCount: 1

Then use gradlew.bat build to build the program, and use

java -jar build/libs/azure-service-bus-prefetch-demo-simplified-0.0.1-SNAPSHOT.jar <namespace connection string> <queue name>

to run the built jar.

Expected result

No dead message is produced and the sent message gets received and completed successfully.

Actual result

A dead message is produced, and the receiver gets the expired message.

[yuhaii]

Hello @yvgopal, good day. Could you please update us with the current progress? Thank you.

[yvgopal]

It is not prefetch. But it is a known issue with the Java SDK.
What is happening here:
Receiver requests a message when there are no messages in the queue.
Receive request times out after the given timeout.
But the request for messages already sent to the service is not cancelled . (cause).
A message is sent to the queue now.
The message is received by the receiver. But there are no pending receive requests, so the message remains in the receiver cache. If no receive requests are made, the message remains in the receiver cache. Eventually the lock of the message is lost in the service and the message is abandoned. As max delivery count is 1, it gets deadlettered.

It is not prefetch, but the already made but timed out receive requests that cause this. A known issue. We will make a proper fix sometime in the future. Until then, please work around this issue. You can either

1. Increase max delivery count.
2. Use long poll, that is long receive timeout, on receive requests with queues or subscriptions that see little to no traffic.

I have marked it as a feature request. But it will probably take time for a fix to be delivered.

[yuhaii]

Thank you, @yvgopal. Please help update this thread when the issue been fixed in the future. Merry Christmas and Happy new year!

[yvgopal]

I am closing this issue as this SDK is deprecated now and a new SDK is released.

[ramya-rao-a]

Please note that the latest version of the new SDK azure-messaging-servicebus has the fix for this issue. Discussion around this problem can be found in #26632