Support lz4 compression

[eapache]

https://github.com/cloudflare/golz4 should be simple enough

https://issues.apache.org/jira/browse/KAFKA-1456
https://issues.apache.org/jira/browse/KAFKA-1493

[eapache]

And yet as with snappy, the framing format kafka uses is not supported by the "normal" go lz4 package.

http://fastcompression.blogspot.ca/2013/04/lz4-streaming-format-final.html

[edenhill]

Again? That's annoying...

[eapache]

This is still not documented on the official protocol page, we don't need it, and nobody else seems to be asking for it. I'm gonna icebox this until somebody actually wants it.

[cdstelly]

Kafka v 0.10 supports LZ4 with correct offsets:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-57+-+Interoperable+LZ4+Framing
https://issues.apache.org/jira/browse/KAFKA-3160

Can this be reopened?

[eapache]

I'll re-open, but I don't have any time to implement this in the next little while so it won't mean much.

[rtreffer]

See #786

[rtreffer]

@eapache I guess this can be closed. I tried the LZ4 on our 0.10.0.1 cluster. It didn't outperform GZIP for our usecase but it works.

[eapache]

Yup, thanks. If anybody really wants the weird framing support then that can be another ticket but I doubt anyone cares.