[BUG] [Servicebus|Azure.Messaging.ServiceBus] CreateMessageBatchAsync does not calculate the size correctly

[reza-esfandyari]

Here is the sample code:
I am not sure if the method is not calculating the size correctly or there is some mismatch between the way that CreateMessageBatchAsync and SendMessagesAsync validate the size

            for (var i = 0; i < integrationEvents.Length; i++)
            {
                if (!messageBatch.TryAddMessage((GenerateMessage(integrationEvents[i]))))
                {
                    try
                    {
                        sender.SendMessagesAsync(messageBatch).GetAwaiter().GetResult();
                    }
                    catch (Exception e)
                    {
                        _logger.LogError(e,
                            "ServiceBus - Failed to send the message");
                    }
                    
                    geBatch.Dispose();
                    messageBatch = sender.CreateMessageBatchAsync().GetAwaiter().GetResult();

                    if (!messageBatch.TryAddMessage((GenerateMessage(integrationEvents[i]))))
                    {
                        Console.WriteLine($"Failed to fit message number in a batch {i}");
                        break;
                    }
                }

            }

            try
            {
                sender.SendMessagesAsync(messageBatch).GetAwaiter().GetResult();
                messageBatch.Dispose();
            }
            catch (Exception e)
            {
                _logger.LogError(e,
                    "ServiceBus - Failed to send the message");
            }

Azure.Messaging.ServiceBus.ServiceBusException: The message (id:56329192, size:293920 bytes) is larger than is currently allowed (262144 bytes). (MessageSizeExceeded)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.SendBatchInternalAsync(Func`1 messageFactory, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.SendBatchInternalAsync(Func`1 messageFactory, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c__DisplayClass18_0.<<SendBatchAsync>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation(Func`2 operation, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation(Func`2 operation, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.SendBatchAsync(ServiceBusMessageBatch messageBatch, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.SendMessagesAsync(ServiceBusMessageBatch messageBatch, CancellationToken cancellationToken)
   at Adia.Core.IntegrationService.ServiceBus.EventBusServiceBus.Publish(IEnumerable`1 events, QueueOrTopicName queueOrTopicName) in D:\repos\Adia-Service-Bulk-Timesheet-Validation\Source\Adia.Core\IntegrationService\ServiceBus\EventBusServiceBus.cs:line 136

there is not much in message custom property but I assume when the batch calculate the size it should consider it too

[JoshLove-msft]

@reza-esfandyari, thanks for submitting the issue. Any chance you can include the logic behind your GenerateMessage method? I haven't been able to repro the issue when setting a message body and application properties.

[reza-esfandyari]

Hi @JoshLove-msft, Thanks for getting back to me quickly, here is the content

 private static ServiceBusMessage GenerateMessage(IntegrationEvent @event)
        {
            var message = new ServiceBusMessage
            {
                MessageId = Guid.NewGuid().ToString(),
                Body = new BinaryData(JsonConvert.SerializeObject(@event)),
                ApplicationProperties =
                {
                    {"EventName", @event.GetType().Name.Replace(IntegrationEventSuffix, "")},
                    {"ServiceName", "TODO"}
                }
            };
            return message;
        }

I include my sample payload structure and hope it help

 public class IntegrationEvent
    {
        public IntegrationEvent()
        {
            Id = Guid.NewGuid();
            CreationDate = DateTime.UtcNow;
            Version = 1;
        }

        [JsonConstructor]
        public IntegrationEvent(Guid id, DateTime createDate)
        {
            Id = id;
            CreationDate = createDate;
        }

        [JsonProperty] public Guid Id { get; private set; }

        [JsonProperty] public DateTime CreationDate { get; private set; }

        [JsonProperty] public int Version { get; private set; }
    }

 public class CandidateCreatedIntegrationEvent : IntegrationEvent
    {
        public int CandidateId { get; set; }
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public DateTime BirthDate { get; set; }
        public byte[] RowVersion { get; set; }
    }


 var sampleIntegrationMessages = new Collection<IntegrationEvent>();
            for (var i = 0; i < 1000; i++)
            {
                var sampleIntegrationMsg1 = new CandidateCreatedIntegrationEvent
                {
                    FirstName = "Reza" + i,
                    CandidateId = i,
                    BirthDate = DateTime.Now,
                    LastName = "Esfandyari" + i
                };

                sampleIntegrationMessages.Add(sampleIntegrationMsg1);
            }
       
          _eventBus.Publish(sampleIntegrationMsg);

[JoshLove-msft]

It looks like the issue is that we are not reserving space for properties that go on the batch envelope, namely the messageId, sessionId, and partitionKey from the first message in the batch. Thanks @jsquire for pointing this out and thank you @reza-esfandyari for submitting the issue! We will get this patched up.

[reza-esfandyari]

Thanks, guys.
any estimate on when I can have the fix @JoshLove-msft? Are you going to update this issue ticket?

[JoshLove-msft]

Thanks, guys.
any estimate on when I can have the fix @JoshLove-msft? Are you going to update this issue ticket?

Yep, it should be available in our next release which is slated for the second week of Feb.