Last stored offset not synchronized to broker

[fabionaspolini]

Hello, I have a dotnet consumer that receives a load of data at certain scheduled times. From the logs I can see that it has processed up to the last message, but the last stored offset is not synchronized with the Kafka broker, and there is always an erroneous lag in the statistics.

The consumer has "auto commit" enabled with a 5 second interval; "auto store offset" disabled; and "partition eof" enabled. The "StoreOffset()" method is executed after processing a batch of 5,000 messages or when "partition eof" is reached.

When the end of the partition is reached, the application continues running (there is no shutdown/crash) and executes a new "Consume()". Now with no pending message in the broker, the thread is stuck waiting for new publications.

1. I would like to understand exactly how the job that performs commits in the background works when auto commit is enabled. Does the fact that the thread is stuck waiting for a new message to be received prevent autocommit from happening?

2. I also tried calling "Commit()" forcedly when reaching the end of the partition, but it didn't sync the offset. Is there any different behavior when trying to call it manually when "auto commit" is enabled?

3. Any idea what might be causing the offset not to sync?

4. Any chance this is an issue for the librdkafka repo?

I'm using the latest version 2.8.0