Error when consuming: error decoding packet: CRC didn't match expected 0x0

[rislah]

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.
Sarama Version: e7238b1
Kafka Version: 2.0.0
Go Version: 1.10.1

Configuration

Kafka is Wurstmeisters docker image file, nothing besides env variables changed.

Sarama SyncProducer for producing and consumer no configuration. Just NewConsumer, looping over Partitions and trying to get msg from ConsumePartition

Logs

kafka: error while consuming user.created/0: kafka: error decoding packet: CRC didn't match expected 0x0 got 0xe38a6876

Problem Description

Cant consume. Getting that error

[eapache]

🤔 I haven't had a chance to try against Kafka 2.0 yet, they've probably changed something in the protocol that's causing this, but I can't find any documentation on what API changes are included in this version...

[yusufsyaifudin]

I got same issue when using this version https://github.com/Shopify/sarama/releases/tag/v1.17.0 35324cf48e33d8260e1c7c18854465a904ade249 and the latest tags of wurstmeister/kafka docker. I try to use 2.11-0.11.0.2 and 2.11-1.1.1 tags and now it works. But, when I try to use 2.11-2.0.0 tag (which the latest tag may refer to), I got the same error again.

[39Er]

I got the same issue. Version: 2.11-2.0.0.

[stremovskyy]

I got the same issue. Version: 2.12-2.0.0.
Any News?

[wladh]

I'll take a look tomorrow.

[wladh]

It only fails if you set the version to pre V0_11_0_0 (which is also the default).
It fails because this feature: https://cwiki.apache.org/confluence/display/KAFKA/KIP-283%3A+Efficient+Memory+Usage+for+Down-Conversion (and this subsequent improvement: apache/kafka#5173)
When the converted message don't fit exactly in the pre-allocated buffer it might generate an overflow message which has offset=0xFFFFFFFF and the length larger than what's left of the batch size, with the idea that the consumer will ignore this supposedly partial message. Beyond length all the bytes in the message are 0.
However, since we push the length and then the CRC decoder on the stack, we first check the CRC of the message which of course fails.
So, it looks like we'll need to special case this overflow message and check at the beginning if block length is greater than what's remaining in the decoders.

[sundy-li]

I meet the same problem, kafka 2.0 version

kafka: error decoding packet: CRC didn't match expected 0x0 got 0xe38a6876

[zhoubowen-sky]

kafka: error decoding packet: CRC didn't match expected

[rislah]

use 1.1.1 tag of wurstmeister docker image like wusuf said till fix

[sundy-li]

@eapache I think there is no big API change on kafka 2.0, because we use old c++ client librdkafka, it still works fine on kafka 2.0. So I think maybe there is something implemention missed by sarama client.

Do you have any plan to fix this issue? We really want to use Go code to read datas from the kafka 2.0. Thank you.

[gecube]

I believe that this issue had to be re-opened.

[wladh]

Why? Are you still seeing the issue on master?

[kirandvrs]

Why? Are you still seeing the issue on master?

I dont think the issue exists with the latest master code...Its working...Consumer is able to receive the message produced...log below from consumer output...

2018-09-12 13:40:30.5469877 -0700 MST m=+0.075200201 Received messages Something Cool

[harshitdx29]

Hi,

I am getting the following error:

kafka: error decoding packet: CRC didn't match. Can someone help? Let me know if any other information is required.

[kirandvrs]

Plz get the updated latest code ..issue will be solved

On Mon, Sep 17, 2018 at 4:45 AM Harshit Gupta ***@***.***> wrote: Hi, I am getting the following error: kafka: error decoding packet: CRC didn't match. Can someone help? Let me know if any other information is required. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1144 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Ag1qV8FNcED1i7DqBoJIsh52C-gNjARlks5ub4twgaJpZM4VrgMs> .

-- kiran...........

[gnatecheng]

I have the same problem, kafka_2.11-2.2.1-cp1.jar version or kafka_2.11-2.0.1-cp1.jar

2019/08/02 18:03:41 kafka: error while consuming test1/0: kafka: error decoding packet: CRC didn't match expected 0x0 got 0xe38a6876