Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with filebeat kafka output with lz4 compression #6414

Closed
meitei11 opened this issue Feb 19, 2018 · 2 comments
Closed

Issue with filebeat kafka output with lz4 compression #6414

meitei11 opened this issue Feb 19, 2018 · 2 comments
Labels

Comments

@meitei11
Copy link

I am trying to setup filebeat (6.2) with kafka output using lz4 compression, but always getting error as
ERR Invalid kafka configuration: kafka: invalid configuration (lz4 compression requires Version >= V0_10_0_0)

My output configuration is:

output.kafka:
hosts: ["kafka1:10092","kafka2:10092","kafka3:10092"]
topic: '%{[fields.log_topic]}'
partition.round_robin:
reachable_only: false
required_acks: 1
compression: lz4
max_message_bytes: 1000000
version: 0.11.0.0

I have tried giving the version config both with and without quotes. Both resulted in same error.

Filebeat version: 6.2.1
Kafka Version: 0.11.0.0

I have tried with putting version as 0.10
In this case putting version without quote as
version: 0.10
yielded an error
Exiting: error initializing publisher: unknown/unsupported kafka version '0.1' accessing 'output.kafka' (source:'filebeat.yml')

If I use with quotes as
version: "0.10"
gives the original error as
ERR Invalid kafka configuration: kafka: invalid configuration (lz4 compression requires Version >= V0_10_0_0)

I had a look at kafka.go file at gihub and noticed the change in sarama version mapping from "0.10.2.1" onwards.

After looking up at sarama codebase tried using version as "0.10.2.0" still same error.

Looks like it is a bug of version mapping or bug at sarama library in the function IsAtLeast

@Kareny
Copy link

Kareny commented Apr 13, 2018

Is there any updates on this bug?

@urso
Copy link

urso commented May 9, 2018

Recent update (#7025) should fix the issue.

@urso urso closed this as completed May 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants