base64 encoded message error: Error while decoding packet: Unexpected messageFormat

[anirban-sap]

Versions

Sarama Version: master
Kafka Version: 0.10.0
Go Version: 1.6.2

Problem Description

I have the following payload struct:

type Payload struct {
    ImageID string `json:"imageID"`
    Content []byte `json:"content"`
}

where Content contains a base64 encoding of an image. The payload can be published successfully via a producer and I can validate using the Kafka console consumer.

However, when I try to consume the payload following the consumer in Sarama Godoc:

partitionConsumer, err := s.Consumer.ConsumePartition(topic, 0, OffsetNewest)
        if err != nil {
            panic(err)
        }

msg := <-partitionConsumer.Messages():
log.Printf("Consumed message offset: %d\n", msg.Offset)

I get the following error on the console: kafka: Error while decoding packet: Unexpected messageFormat

I am not using any compression in the producer. Is there a recommendation what I might be doing wrong?

[eapache]

"Unexpected messageFormat" is not an error that sarama returns, are you sure it's not coming from some other code?

[anirban-sap]

The consumer code is straight from https://godoc.org/github.com/Shopify/sarama#Consumer - seems to work fine if I pass a base64 encoded string such as "Hello World" but for images, I get the error message. The only other reference of the error is in this other Sarama issue #635

[eapache]

Oh. You're not actually using sarama master then, that error was removed a month ago.

[anirban-sap]

I cloned the repo last week and could see the same behavior in kafka-console-consumer under the tools folder of sarama. So, if I run

echo "SGVsbG8sIOS4lueVjA==" | ./kafka-console-producer -topic=topic1

the string shows up in sarama kafka-console-consumer and the kafka-console-consume in kafka/bin folder. However, if I pass in a (much) longer base64 encoded string of a jpeg file, for example, the sarama kafka-console-consumer does not echo (crashes?) the string whereas the kafka-console-consumer from kafka/bin works. Not sure if I am missing something obvious.

[anirban-sap]

Can confirm that my simple pub/sub test is working with Kafka v0.8.2.2 but not with v0.10.0

[eapache]

That error message was literally removed in a PR that was merged on June 20th: https://github.com/Shopify/sarama/pull/681/files#diff-b95c50d14ed7b4bb1cfecdc4e498d4dcL97. That string no longer exists in the code base at all.

That PR added support for consuming from Kafka v0.10 so if you're using code from before that, I wouldn't expect it to work.