[libbeat] Go Benchmarks comparing compress/gzip and klauspost/compress

[khushijain21]

Results

We send data to the _bulk endpoint and benchmark the results with and without the https://github.com/klauspost/compress library.

Results with varying compression level and event size

Time per operation

Test	without_klauspost.txt	with_klauspost.txt	Improvement
Publish/5_events_with_compression_level_1-12	91.52µ ± 11%	76.34µ ± 2%	-16.58%
Publish/5_events_with_compression_level_4-12	126.78µ ± 8%	82.20µ ± 0%	-35.16%
Publish/5_events_with_compression_level_7-12	118.2µ ± 1%	109.5µ ± 0%	-7.34%
Publish/5_events_with_compression_level_9-12	131.1µ ± 8%	147.3µ ± 3%	+12.36%
Publish/50_events_with_compression_level_1-12	453.8µ ± 5%	347.3µ ± 2%	-23.47%
Publish/50_events_with_compression_level_4-12	676.3µ ± 5%	422.7µ ± 2%	-37.50%
Publish/50_events_with_compression_level_7-12	704.8µ ± 2%	592.2µ ± 1%	-15.98%
Publish/50_events_with_compression_level_9-12	878.8µ ± 3%	991.6µ ± 2%	+12.84%
Publish/500_events_with_compression_level_1-12	435.4µ ± 1%	343.6µ ± 2%	-21.09%
Publish/500_events_with_compression_level_4-12	636.7µ ± 1%	428.6µ ± 12%	-32.68%
Publish/500_events_with_compression_level_7-12	681.7µ ± 1%	601.5µ ± 1%	-11.77%
Publish/500_events_with_compression_level_9-12	855.8µ ± 2%	992.4µ ± 2%	+15.96%
Geomean	364.7µ	309.3µ	-15.17%

---

Bytes per operation

Test	without_klauspost.txt	with_klauspost.txt	Improvement
Publish/5_events_with_compression_level_1-12	18.78Ki ± 0%	18.74Ki ± 0%	-0.20%
Publish/5_events_with_compression_level_4-12	18.83Ki ± 0%	18.68Ki ± 0%	-0.79%
Publish/5_events_with_compression_level_7-12	18.83Ki ± 0%	18.75Ki ± 0%	-0.41%
Publish/5_events_with_compression_level_9-12	18.79Ki ± 0%	18.82Ki ± 0%	~
Publish/50_events_with_compression_level_1-12	88.73Ki ± 0%	88.46Ki ± 0%	-0.31%
Publish/50_events_with_compression_level_4-12	88.27Ki ± 0%	88.32Ki ± 0%	~
Publish/50_events_with_compression_level_7-12	88.31Ki ± 0%	88.51Ki ± 0%	+0.23%
Publish/50_events_with_compression_level_9-12	88.37Ki ± 0%	88.83Ki ± 0%	+0.52%
Publish/500_events_with_compression_level_1-12	88.39Ki ± 0%	88.44Ki ± 0%	~
Publish/500_events_with_compression_level_4-12	88.30Ki ± 0%	88.47Ki ± 0%	+0.19%
Publish/500_events_with_compression_level_7-12	88.28Ki ± 0%	88.55Ki ± 0%	+0.31%
Publish/500_events_with_compression_level_9-12	88.27Ki ± 0%	88.86Ki ± 0%	+0.66%
Geomean	52.76Ki	52.78Ki	+0.04%

---

Summary

With the ' klauspost ' library, there is a significant speedup across tests for most compression levels with varying event sizes. But with compression level 9, it is slower - resulting in a performance drop of around 12-16%.

This PR also replaces compress/gzip to klauspost/compress/gzip

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Relates #38599

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @khushijain21? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[rdner]

@khushijain21 I think we need a better representation of the results in the description. Would be nice to have a more readable table / graphs and a summary / conclusion at the end.

[rdner]

LGTM but the linter issue needs to be fixed and there are a few comments to address.

[rdner]

@khushijain21 from the table looks like the compression level 4 always wins. I wonder what's the payload size difference between compress/gzip and klauspost/compress in this case?

I think it's important to compare encoded sizes, otherwise if klauspost/compress ~35% faster but the compressed payload is 50% larger we would slow down on networking more than we win on the CPU.

[rdner]

Looks good to me.
We just need to fix the changelog formatting, there is an extra empty line before the new entry and the missing empty line after the new entry.