Duplicate message from RDKafka

[nagendrababupv]

Hi Edenhill,

I have auto-commit true with a commit interval of 8000 ms and I able to consume the same message multiple times in 4ms only i suspected that multiple consumers are called but I could see only one rd_kafka_consume0 is being called

Thread 30 (Thread 0x7f7652bfd700 (LWP 14565)):
#0 0x00000039b4c0ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type to continue, or q to quit---
#1 0x00000000007ff2a9 in cnd_timedwait ()
#2 0x00000000007aca65 in rd_kafka_q_pop_serve.localalias.12 ()
#3 0x0000000000774fc5 in rd_kafka_consume0 () at rdtime.h:257
#4 0x000000000076806a in RdKafka::KafkaConsumerImpl::consume(int) (this=Unhandled dwarf expression opcode 0xf3
) at KafkaConsumerImpl.cpp:114
#5 0x0000000000761546 in CConsumer::mefn_ReadTopic() () at Consumer.cpp:75

Read the FAQ first: https://github.com/edenhill/librdkafka/wiki/FAQ

Description

How to reproduce

<your steps how to reproduce goes here, or remove section if not relevant>

IMPORTANT: Always try to reproduce the issue on the latest released version (see https://github.com/edenhill/librdkafka/releases), if it can't be reproduced on the latest version the issue has been fixed.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): <REPLACE with e.g., v0.10.5 or a git sha. NOT "latest" or "current">
• Apache Kafka version: <REPLACE with e.g., 0.10.2.3>
• librdkafka client configuration: <REPLACE with e.g., message.timeout.ms=123, auto.reset.offset=earliest, ..>
• Operating system: <REPLACE with e.g., Centos 5 (x64)>
• Provide logs (with debug=.. as necessary) from librdkafka
• Provide broker log excerpts
• Critical issue

[nagendrababupv]

message.timeout.ms=1000
librdkafka version = 1.6.0
Kafka version=2.13-2.7.0
OS= "uname -a
Linux CHNKFK01 4.18.0-240.10.1.el8_3.x86_64 #1 SMP Wed Dec 16 03:30:52 EST 2020 x86_64 x86_64 x86_64 GNU/Linu"
cat /etc/redhat-release
Red Hat Enterprise Linux release 8.3 (Ootpa)

[nagendrababupv]

[stsuser@dev-01 logs]$ grep 627391 ADM_VIf4_21_03_2021_*
ADM_VIf4_21_03_2021_20_03_20_01.log:[Consumer.cpp|00088|20:20:22:971|DG]Read message at offset :627391
ADM_VIf4_21_03_2021_20_03_20_01.log:[Consumer.cpp|00088|20:22:45:976|DG]Read message at offset :627391

This time I can see Time Gap is more. not 4ms but I can see few requests with 4ms as well.
The producer is doing once only as per this.
consumer group-id also same not different consumer id not found.
Single thread only running
Try the log

I Tested every time commit the message so no duplicate message. But it is a very slow of consuming part.
How to handle this case

[nagendrababupv]

I am getting this error maybe this is the reason for the Duplicates
Commit failed :Broker: Specified group generation id is not valid for offset: 403038 and topic :T_RE2CP

[edenhill]

Failing to commit will yield duplicate messages since an older committed offset will be used as starting point.

[nagendrababupv]

Yes commit failed. when I commit without argument it is working.