Zlib compression

[zackwine]

#26

Adds support for zlib compression.

See the example config below:

[SERVICE]
     Flush 20

[INPUT]
    Name        forward
    Listen      0.0.0.0
    Port        24224

[OUTPUT]
    Name            kinesis
    Match           *
    region          us-west-2
    stream          my-kinesis-stream-name
    compression     zlib
    append_newline  true

This will ensure individual records are zlib compressed as they are sent to Kinesis. This feature has been found to be compatible with the fluentd zlib compression (tested with the same decompression lambda we use in production).

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[sonofachamp]

LGTM. Thanks for contributing!

[PettitWesley]

@zackwine Thank you!

[zackwine]

Bump. Anything I can do to expedite getting a release with this feature?

FYI: Further load testing shows this will reduce record throughput by 20-40% when enabled (depending on concurrency configuration). This is expected given the additional CPU required to compress each record.

[PettitWesley]

@zackwine Things are a bit tight right now... I will try to do a release this week for this + KPL if I can. Will let you know if anything is needed from you, but I think you're good.