Can not subscribe to a topic with more than 1 partition when using Confluent.Kafka v2.2.0 in .NET Framework

[MasoudKarimi92]

Description

It seems there is a bug when using Confluent.Kafka v2.2.0 in .NET Framework projects. When I want to subscribe to a topic with more than 1 partition it always crashes at Subscribe() method. There is no problem in using this package with .NET Core and .NET 6 or 7 projects. Also when I downgraded the package to v2.1.1 the issue resolved. Here is the error in v2.2.0 in .NET Framework projects:

rd_tmpabuf_alloc0: rd_kafka_topic_info_new_with_rack:1860: requested size 16 + 5 > 20

How to reproduce

1 - run kafka on docker
2- Create a topic with more than one partition, for example 3 partitions:
kafka-topics --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 3 --topic test
3- produce some message in this topic
4- write a simple console app in .NET Framework which tries to subscribe to this topic and consume messages
the app will crash at Subscribe()

here is my code for consumer:

`var config = new ConsumerConfig
{
BootstrapServers = "localhost:9092",
GroupId = "kafka-dotnet-getting-started",
AutoOffsetReset = AutoOffsetReset.Earliest
};

        const string topic = "test";

        CancellationTokenSource cts = new CancellationTokenSource();
        Console.CancelKeyPress += (_, e) => {
            e.Cancel = true; // prevent the process from terminating.
            cts.Cancel();
        };

        using (var consumer = new ConsumerBuilder<string, string>(
            config).Build())
        {
            consumer.Subscribe(topic);
            try
            {
                while (true)
                {
                    var cr = consumer.Consume(cts.Token);
                    Console.WriteLine($"Consumed event from topic {topic} with key {cr.Message.Key,-10} and value {cr.Message.Value}");
                }
            }
            catch (OperationCanceledException)
            {
                // Ctrl-C was pressed.
            }
            catch (Exception ex)
            {

            }
            finally
            {
                consumer.Close();
            }

`

Checklist

Please provide the following information:

• A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• Confluent.Kafka nuget version.
• Apache Kafka version.
• Client configuration.
• Operating system.
• Provide logs (with "debug" : "..." as necessary in configuration).
• Provide broker log excerpts.
• Critical issue.

[glazkovalex]

I confirm the presence of a problem in version 2.2.0 at .Net 7.0!

When trying to subscribe to the topic "sample", the error 'Access violation' occurs: rd_tmpabuf_alloc0: rd_kafka_topic_info_new_with_rack:1860: requested size 16 + 7 > 20.
When trying to subscribe to the topic "sample-temp", the error 'Access violation' occurs: rd_tmpabuf_alloc0: rd_kafka_topic_info_new_with_rack:1860: requested size 16 + 13 > 28.
But with the name of the topic "sample-8", the subscription works for some reason!

My test:

    // Given
    const string topic = "sample";
    var bootstrapServer = _kafkaContainer.GetBootstrapAddress();
    var producerConfig = new ProducerConfig
    {
        BootstrapServers = bootstrapServer,
    };
    var consumerConfig = new ConsumerConfig
    {
        BootstrapServers = bootstrapServer,
        GroupId = "sample-consumer",
        AutoOffsetReset = AutoOffsetReset.Earliest,
    };
    var message = new Message<string, string> { Value = Guid.NewGuid().ToString("D") };

    // When
    ConsumeResult<string, string> consumeResult;
    using (var producer = new ProducerBuilder<string, string>(producerConfig).Build())
    {
        var deliveryResult = await producer.ProduceAsync(topic, message).ConfigureAwait(false);
    }

    using (var consumer = new ConsumerBuilder<string, string>(consumerConfig).Build())
    {
        consumer.Subscribe(topic); // This is where 'Access violation' occurs!!!
        consumeResult = consumer.Consume(TimeSpan.FromSeconds(15));
    }

    // Then
    Assert.Equal(message.Value, consumeResult.Message.Value);

[gregdlv]

same here;
Unable to subscribe to any topic, consume function will make it crash with the error 'Access violation' : rd_tmpabuf_alloc0 requested size 16 + 16 > 28.

[bsimovic]

I have the exact same issue on .NET Framework. Tried it on versions 4.8, 4.7.2 and 4.6.2 and it crashes with an access violation. However, unlike posters above, I can't reproduce it in .NET 6.0 or 7.0.

It is very strange and it seems it has something to do with the name of the topic you are subscribing to. 3 and 11 character topic names work fine while for instance a 6 character topic name will induce the crash.

I'm hoping this will get looked into.

EDIT: Also, for me at least, it is not affected by the number of partitions.
EDIT 2: Confirming the topic name is the issue here. The error printed out will state: requested size 16 + {topic_name_length+1} > 20

[jonny-binns]

I also had the same issue with .NET 4.8, the fix for me was to roll back confluent.kafka and librdkafka.redist to 2.2.1. Then I cleaned and re-built my project and it worked.

This isn't an optimal fix but if you need to get your app going this should work.

[anchitj]

I tried reproducing with Dotnet SDK 6.0 and 7.0 on Ubuntu and Mac, but couldn't reproduce it. Will try checking with Dotnet Framework 4.8 on Windows machine.

[emasab]

Hello, is this happening on a 32bit architecture? Seems similar to this issue that we've fixed recently with this PR

[MasoudKarimi92]

I tried reproducing with Dotnet SDK 6.0 and 7.0 on Ubuntu and Mac, but couldn't reproduce it. Will try checking with Dotnet Framework 4.8 on Windows machine.

I also had no problem using this package with .NET 6 or 7 on Windows machine. The problem only exists in .NET Framework.

[MasoudKarimi92]

Hello, is this happening on a 32bit architecture? Seems similar to this issue that we've fixed recently with this PR

No, I faced this issue on a 64bit architecture.

[anchitj]

Thanks, I was able to reproduce this on Windows with dotnet framework 4.8. I think above PR by @emasab should fix the issue. I'll test it once we have the librdkafka 2.3.0 release done and will confirm here.

[alfhv]

same issue on my side, all consumers migrated to v2.2.0 crashes when executing subscribe() in line :
https://github.com/confluentinc/confluent-kafka-dotnet/blob/v2.2.0/src/Confluent.Kafka/Impl/LibRdKafka.cs#L1044
(dotnet core 7, windows 64)

[anchitj]

We have released 2.3.0. Can you test with that and confirm if the issue is there?

[alfhv]

Fixed on my side

[fabmoll]

Fixed here too :)
Thx !!

[anchitj]

Thanks for confirming.