Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support current kafka compression types #361

Open
technomage opened this issue Aug 31, 2019 · 3 comments
Open

Support current kafka compression types #361

technomage opened this issue Aug 31, 2019 · 3 comments

Comments

@technomage
Copy link

Currently kafka supports more compression types by default than kafka_ex which can cause a client to fail when attempting to read such a topic. In particular confluent platform uses some of the newer compression types. At a minimum it would be nice to be able to configure kafka_ex to not die on such topics.

@dantswain
Copy link
Collaborator

dantswain commented Apr 26, 2020

@technomage which compression types are you using? The implementation of compression support so far has mostly been due to the needs of someone who has time and know-how to do the implementation :/

@technomage
Copy link
Author

Confluent uses one not currently supported which crashes a client trying to read from those topics.

@joshuawscott
Copy link
Member

@technomage lz4 isn't yet supported, but a PR would be welcome:

https://cwiki.apache.org/confluence/display/KAFKA/Compression
lz4 is type 3 - it would be added to https://github.com/kafkaex/kafka_ex/blob/76c752b82c48d04281b8fcf4ad4f14c4fbed335b/lib/kafka_ex/compression.ex
to basically wrap the lz4 compress/decompress operations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants