add zstd compression type #112
Conversation
|
Fixes #2 |
There was a problem hiding this comment.
Some very minor nitpicks, but tested locally, and it does what was advertised.
@lhuet Would you mind adding changelogs, and bumping the version? As this introduces a new option, we would typically prefer a bump in the minor version.
@karenzone Would you mind checking the documentation?
Co-authored-by: Rob Bavey <[email protected]>
Co-authored-by: Karen Metts <[email protected]>
Co-authored-by: Rob Bavey <[email protected]>
| load_kafka_data(config) | ||
| end | ||
|
|
||
| # NOTE: depends on zstd-ruby gem which is using a C-extension |
There was a problem hiding this comment.
The spec validating the new behaviour is commented out, presumably because we need to do something else to get the C-extension compiled/installed.
What else needs to be done? If a user were to install this plugin, would zstd compression work out of the box, or would they need to side-channel load that zstd-ruby gem somehow?
There was a problem hiding this comment.
The zstd-ruby is required only for test as the Kafka ruby client is used for the tests.
The plugin itself uses the Java Kafka client that support the zstd compression out of the box.
Initially, I tested it successfully with only modifying the kafka.rb file (adding 'zstd' as 'allowed' compression type):
For the LZ4, it is identical. The extlz4 gem use a C-extension that the current test suite does not support:
As for zstd, the Java Kafka client supports the lz4 compression out of the box.
There was a problem hiding this comment.
I tested this locally, using the kafka input to pull data from the zstd compressed topic that was written to by the updated kafka output, given that the zstd integration with the ruby-kafka gem that is used for integration tests only uses a gem without jruby support.
|
@karenzone I've approved the code portion, are the docs good to go? |
|
@lhuet This is good to go. Thank you for your contribution! I will update this ticket when the publishing process is complete |
|
This plugin has now been published as |
Merge in THOT/logstash-integration-kafka from dev-edefaria-siem to siem * commit 'd66d027404b5afd6e68e282f669a33e45d3cb392': update multiple dependancy and allow to build with jdk18 Update kafka version for integration tests (logstash-plugins#123) [Doc] Improve doc for keystore types (logstash-plugins#121) Describe potenatial problems in `topic_pattern` setting (logstash-plugins#122) Fix CHANGELOG.md link to PR (logstash-plugins#119) bump kafka client to 2.8.1 (logstash-plugins#115) Feat: added connections_max_idle_ms setting for output (logstash-plugins#118) Update CHANGELOG.md (logstash-plugins#114) add zstd compression type (logstash-plugins#112)
Zstandard compression type is supported by the Kafka client already embedded in this plugin.
This PR allows the "zstd" compression type to be used in the Logstash Kafka output.