[Service Bus] Batching receiver in receiveAndDelete mode loses messages when working with large number of messages

[ramya0820]

• Package Name: @azure/service-bus
• Package Version: 1.1.0

Describe the bug
The batching receiver in Service Bus does not work as expected in receiveAndDelete mode i.e., when fetching large number of messages, a significantly different number of messages are being taken off.
For example, when doing receiveMessages(300), often times about only 100 messages are actually received and about 120 messages are deleted from the Service Bus entity.

To Reproduce

    const connectionString = "<insert-connection-string>";
    const sbClient = ServiceBusClient.createFromConnectionString(connectionString);

    const subscriptionClient = sbClient.createSubscriptionClient(topicName, subscriptionName);
    const receiver = await subscriptionClient.createReceiver(ReceiveMode.receiveAndDelete);

    const numOfMessages = 300;
    try {
      const messages = await receiver.receiveMessages(numOfMessages);
      await receiver.close();

      should.equal(
        messages.length,
        numOfMessages,
        `Expected ${numOfMessages} but received ${messages.length}`
      );

      // Manually check message count on the Service Bus entity

      await subscriptionClient.close();
    } finally {
      await sbClient.close();
    }

Expected behavior
As is the case with peekLock mode, the exact number of messages requested are to be retrieved preferably. And the number of messages retrieved must equal the number of messages deleted from the Service Bus entity.

[soates]

When will this be merged and released? We are seeing this exact issue. I don't think it matters on the batch size. We see this issue with a batch size of 20.

[ramya-rao-a]

Thanks for reporting @soates

We are working on the fix at the moment and should have a patch update ready soon.
We will post an update here when ready.

Meanwhile, can you share which version of the service bus library you are currently using?

[soates]

Hey, we are using version 1.1.0. We really want to use this package in production but are running into many issues which we did not expect with the stability of this package..

[ramya-rao-a]

@soates We should be able to get the patch update out by tomorrow.

Can you create a new issue for all the other issues you are finding with the package? We would like to help in any way we can and would appreciate any feedback you have to make the package better.

[ramya-rao-a]

@soates This bug has been fixed in the latest version (1.1.1) of the @azure/service-bus package.

Thanks for your patience.

[ghost]

Thanks for working with Microsoft on GitHub! Tell us how you feel about your experience using the reactions on this comment.

[ramya-rao-a]

Some details on the root cause for this bug for future reference:

• The promise returned by the receiveMessages method is resolved if more than 1 second has elapsed since the last message that was collected.
• This was important for PeekLock mode as waiting too much resulted in earlier messages getting their locks expired.
• ReceiveAndDelete mode is slower ([Service Bus] ReceiveAndDelete mode gets lesser messages than PeekLock mode when using receiveMessages #6349 is logged to investigate why), therefore this timer always gets triggered and in flight messages get ignored resulting in data loss

In PR #6265, we remove this timer when receiveMessages is used in ReceiveAndDelete mode which fixes the data loss issue.

[ramya0820]

While we practically have seen the issue so far in just the receiveAndDelete mode and could be a service side behavior/issue - the possibility of new message timer exceeding for peekLock mode does exist theoretically? (Changes in #6601 got me thinking about how if we don't handle the credit drain event, we might end up in receiving messages over wire but not in code for newMessageWaitTime exceeding case)

For peekLock mode, this may result in delivery count of messages getting incremented if the messages get received over link and not by our code - which may be problematic across large number of requests?
If so, we might want to handle #5757 changes for peekLock mode as well?

The concern about lock getting lost on messages seems to be an expected service behavior and may be something to address as well. (Most likely by the user. From SDK side, could so by disallowing receivers in peekLock mode to hold onto messages for too long, or use autoLockRenewal (Sorry if there was an another issue already addressing this cause I recall discussing this earlier)
cc: @ramya-rao-a @chradek