Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libbeat: optimize asset data decoding #42180

Merged
merged 1 commit into from
Jan 2, 2025

Conversation

mauri870
Copy link
Member

@mauri870 mauri870 commented Dec 30, 2024

Proposed commit message

While working on #41888 I was benchmarking the filebeatreceiver
CreateLogs factory and noticed that the asset decoding in libbeat
dominates the cpu and memory profile of the receiver creation.

This behavior is expected since asset decoding is intended to occur at
startup. However, it's still worthwhile to optimize it if possible.

Some time ago I worked on iobuf.ReadAll at
elastic/elastic-agent-libs#229, an optimized version of io.ReadAll
that has a better growth algorithm—based on bytes.Buffer—and
benefits from the io.ReaderFrom optimization. The choice of when to
use this is very picky, as using it with a reader that is not a
io.ReaderFrom can be slower than the standard io.ReadAll. For this
case we are certain of the reader implementation, so we can use it.
Benchmark results show that it is 5% faster and uses 17% less memory.

goos: linux
goarch: amd64
pkg: github.com/elastic/beats/v7/x-pack/filebeat/fbreceiver
cpu: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
           │   old.txt   │              new.txt               │
           │   sec/op    │   sec/op     vs base               │
Factory-32   9.506m ± 3%   8.962m ± 2%  -5.73% (p=0.000 n=10)

           │   old.txt    │               new.txt                │
           │     B/op     │     B/op      vs base                │
Factory-32   12.89Mi ± 0%   10.68Mi ± 0%  -17.20% (p=0.000 n=10)

           │   old.txt   │              new.txt               │
           │  allocs/op  │  allocs/op   vs base               │
Factory-32   5.083k ± 0%   4.944k ± 0%  -2.74% (p=0.000 n=10)

After these fixes the profiles are still dominated by the asset
decoding, but I guess that is expected, at least it is a bit faster now.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@mauri870 mauri870 self-assigned this Dec 30, 2024
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 30, 2024
Copy link
Contributor

mergify bot commented Dec 30, 2024

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @mauri870? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

Copy link
Contributor

mergify bot commented Dec 30, 2024

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

@mergify mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Dec 30, 2024
@mauri870 mauri870 added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Dec 30, 2024
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 30, 2024
While working on elastic#41888 I was benchmarking the filebeatreceiver
CreateLogs factory and noticed that the asset decoding in libbeat
dominates the cpu and memory profile of the receiver creation.

This behavior is expected since asset decoding is intended to occur at
startup. However, it's still worthwhile to optimize it if possible.

Some time ago I worked on `iobuf.ReadAll` at
elastic/elastic-agent-libs#229, an optimized version of `io.ReadAll`
that has a better growth algorithm—based on bytes.Buffer—and
benefits from the `io.ReaderFrom` optimization. The choice of when to
use this is very picky, as using it with a reader that is not a
`io.ReaderFrom` can be slower than the standard `io.ReadAll`. For this
case we are certain of the reader implementation, so we can use it.
Benchmark results show that it is 5% faster and uses 17% less memory.

After these fixes the profiles are still dominated by the asset
decoding, but I guess that is expected, at least it is a bit faster now.
@mauri870 mauri870 force-pushed the libbeat-asset-decode-data branch from 8f4fbb5 to dc216f1 Compare December 31, 2024 13:34
@mauri870
Copy link
Member Author

/test

@mauri870 mauri870 marked this pull request as ready for review January 1, 2025 15:12
@mauri870 mauri870 requested a review from a team as a code owner January 1, 2025 15:12
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@mauri870 mauri870 merged commit 3d1bdcf into elastic:main Jan 2, 2025
141 checks passed
mergify bot pushed a commit that referenced this pull request Jan 2, 2025
While working on #41888 I was benchmarking the filebeatreceiver
CreateLogs factory and noticed that the asset decoding in libbeat
dominates the cpu and memory profile of the receiver creation.

This behavior is expected since asset decoding is intended to occur at
startup. However, it's still worthwhile to optimize it if possible.

Some time ago I worked on `iobuf.ReadAll` at
elastic/elastic-agent-libs#229, an optimized version of `io.ReadAll`
that has a better growth algorithm—based on bytes.Buffer—and
benefits from the `io.ReaderFrom` optimization. The choice of when to
use this is very picky, as using it with a reader that is not a
`io.ReaderFrom` can be slower than the standard `io.ReadAll`. For this
case we are certain of the reader implementation, so we can use it.
Benchmark results show that it is 5% faster and uses 17% less memory.

After these fixes the profiles are still dominated by the asset
decoding, but I guess that is expected, at least it is a bit faster now.

(cherry picked from commit 3d1bdcf)
@Kavindu-Dodan
Copy link
Contributor

It seems this introduced a build issue. I get vet: fbreceiver/receiver_test.go:133:26: NewFactory().CreateLogsReceiver undefined (type receiver.Factory has no field or method CreateLogsReceiver) 1 from recent builds.

Footnotes

  1. https://github.com/elastic/beats/actions/runs/12588586152/job/35086765916?pr=41817#step:8:216

@mauri870
Copy link
Member Author

mauri870 commented Jan 2, 2025

It seems this introduced a build issue.

Yes, that is fixed in #42199.

mauri870 added a commit that referenced this pull request Jan 6, 2025
* libbeat: optimize asset data decoding (#42180)

While working on #41888 I was benchmarking the filebeatreceiver
CreateLogs factory and noticed that the asset decoding in libbeat
dominates the cpu and memory profile of the receiver creation.

This behavior is expected since asset decoding is intended to occur at
startup. However, it's still worthwhile to optimize it if possible.

Some time ago I worked on `iobuf.ReadAll` at
elastic/elastic-agent-libs#229, an optimized version of `io.ReadAll`
that has a better growth algorithm—based on bytes.Buffer—and
benefits from the `io.ReaderFrom` optimization. The choice of when to
use this is very picky, as using it with a reader that is not a
`io.ReaderFrom` can be slower than the standard `io.ReadAll`. For this
case we are certain of the reader implementation, so we can use it.
Benchmark results show that it is 5% faster and uses 17% less memory.

After these fixes the profiles are still dominated by the asset
decoding, but I guess that is expected, at least it is a bit faster now.

(cherry picked from commit 3d1bdcf)

* s/CreateLogsReceiver/CreateLogs/

---------

Co-authored-by: Mauri de Souza Meneguzzo <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-8.x Automated backport to the 8.x branch with mergify cleanup Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants