-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
libbeat: optimize asset data decoding #42180
Conversation
This pull request does not have a backport label.
To fixup this pull request, you need to add the backport labels for the needed
|
|
While working on elastic#41888 I was benchmarking the filebeatreceiver CreateLogs factory and noticed that the asset decoding in libbeat dominates the cpu and memory profile of the receiver creation. This behavior is expected since asset decoding is intended to occur at startup. However, it's still worthwhile to optimize it if possible. Some time ago I worked on `iobuf.ReadAll` at elastic/elastic-agent-libs#229, an optimized version of `io.ReadAll` that has a better growth algorithm—based on bytes.Buffer—and benefits from the `io.ReaderFrom` optimization. The choice of when to use this is very picky, as using it with a reader that is not a `io.ReaderFrom` can be slower than the standard `io.ReadAll`. For this case we are certain of the reader implementation, so we can use it. Benchmark results show that it is 5% faster and uses 17% less memory. After these fixes the profiles are still dominated by the asset decoding, but I guess that is expected, at least it is a bit faster now.
8f4fbb5
to
dc216f1
Compare
/test |
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane) |
While working on #41888 I was benchmarking the filebeatreceiver CreateLogs factory and noticed that the asset decoding in libbeat dominates the cpu and memory profile of the receiver creation. This behavior is expected since asset decoding is intended to occur at startup. However, it's still worthwhile to optimize it if possible. Some time ago I worked on `iobuf.ReadAll` at elastic/elastic-agent-libs#229, an optimized version of `io.ReadAll` that has a better growth algorithm—based on bytes.Buffer—and benefits from the `io.ReaderFrom` optimization. The choice of when to use this is very picky, as using it with a reader that is not a `io.ReaderFrom` can be slower than the standard `io.ReadAll`. For this case we are certain of the reader implementation, so we can use it. Benchmark results show that it is 5% faster and uses 17% less memory. After these fixes the profiles are still dominated by the asset decoding, but I guess that is expected, at least it is a bit faster now. (cherry picked from commit 3d1bdcf)
It seems this introduced a build issue. I get Footnotes |
Yes, that is fixed in #42199. |
* libbeat: optimize asset data decoding (#42180) While working on #41888 I was benchmarking the filebeatreceiver CreateLogs factory and noticed that the asset decoding in libbeat dominates the cpu and memory profile of the receiver creation. This behavior is expected since asset decoding is intended to occur at startup. However, it's still worthwhile to optimize it if possible. Some time ago I worked on `iobuf.ReadAll` at elastic/elastic-agent-libs#229, an optimized version of `io.ReadAll` that has a better growth algorithm—based on bytes.Buffer—and benefits from the `io.ReaderFrom` optimization. The choice of when to use this is very picky, as using it with a reader that is not a `io.ReaderFrom` can be slower than the standard `io.ReadAll`. For this case we are certain of the reader implementation, so we can use it. Benchmark results show that it is 5% faster and uses 17% less memory. After these fixes the profiles are still dominated by the asset decoding, but I guess that is expected, at least it is a bit faster now. (cherry picked from commit 3d1bdcf) * s/CreateLogsReceiver/CreateLogs/ --------- Co-authored-by: Mauri de Souza Meneguzzo <[email protected]>
Proposed commit message
While working on #41888 I was benchmarking the filebeatreceiver
CreateLogs factory and noticed that the asset decoding in libbeat
dominates the cpu and memory profile of the receiver creation.
This behavior is expected since asset decoding is intended to occur at
startup. However, it's still worthwhile to optimize it if possible.
Some time ago I worked on
iobuf.ReadAll
atelastic/elastic-agent-libs#229, an optimized version of
io.ReadAll
that has a better growth algorithm—based on bytes.Buffer—and
benefits from the
io.ReaderFrom
optimization. The choice of when touse this is very picky, as using it with a reader that is not a
io.ReaderFrom
can be slower than the standardio.ReadAll
. For thiscase we are certain of the reader implementation, so we can use it.
Benchmark results show that it is 5% faster and uses 17% less memory.
After these fixes the profiles are still dominated by the asset
decoding, but I guess that is expected, at least it is a bit faster now.
Checklist
CHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.