Remove 'onFilteredOut' and 'onDroppedOnPublish' callback logs #36399
Conversation
botelastic
bot
added
the
needs_team
The 'onFilteredOut' and 'onDroppedOnPublish' callbacks debug logs can cause an infinity loop of log entries being logged over and over again when running under Elastic-Agent. Those log entries are removed to prevent this from happening in the future. The monitoring Filebeat uses a drop processor to drop its own logs, however if log level is debug the `onFilteredOut` callback debug log will log all dropped events, because it is done by the monitoring Filebeat, it will be picked up, dropped and logged again, creating this infinity loop.
belimawr
force-pushed
the
remove-on-filtered-events-from-logs
branch
from
099c2ff to
3e9e8e0
Compare
|
This pull request does not have a backport label.
To fixup this pull request, you need to add the backport labels for the needed
|
botelastic
bot
removed
the
needs_team
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
💚 Flaky test reportTests succeeded. 🤖 GitHub commentsExpand to view the GitHub comments
To re-run your PR in the CI, just comment with:
|
| @@ -203,12 +203,10 @@ func (c *client) onPublished() { | |||
| } | |||
|
|
|||
| func (c *client) onFilteredOut(e beat.Event) { | |||
| c.logger.Debugf("Pipeline client receives callback 'onFilteredOut' for event: %+v", e) | |||
There was a problem hiding this comment.
I'm not sure I agree with this change. I remember this log line helped in a few SDHs. Is there a way to log it once? The infinite loop you mentioned in the description – is it expected behaviour? Because this change only removes observability of that loop not the loop itself, right?
There was a problem hiding this comment.
The fact that we observe every filtered out event is the root cause of the loop I described. Indeed it only happens if Filebeat is collecting it own logs, which is a side effect of the Elastic-Agent aggregating all Beats logs into its own log file.
Another option I see is to check if Filebeat is running under Elastic-Agent before issuing those debug logs. I'm not a huge fan of having different behaviours on this part of the code base based on whether the Beat is running under Elastic-Agent or not, but that is still a valid option.
What do you think?
There was a problem hiding this comment.
Do we count this callbacks somewhere in the periodic metrics? If so, we can remove the log messages, since I see complaints from other users about the flood of these logs.
There was a problem hiding this comment.
Not that I'm aware of. Not as callbacks called, we do have some metrics for events dropped, I'm not sure about filtered out.
There was a problem hiding this comment.
Okay, at least we have it from dropped. LGTM.
|
Could we backport this to 8.9 and 8.10? |
belimawr
added
backport-v8.9.0
I don't see any reason not to backport ;) |
The 'onFilteredOut' and 'onDroppedOnPublish' callbacks debug logs can cause an infinity loop of log entries being logged over and over again when running under Elastic-Agent. Those log entries are removed to prevent this from happening in the future. The monitoring Filebeat uses a drop processor to drop its own logs, however if log level is debug the `onFilteredOut` callback debug log will log all dropped events, because it is done by the monitoring Filebeat, it will be picked up, dropped and logged again, creating this infinity loop. (cherry picked from commit dde4079)
The 'onFilteredOut' and 'onDroppedOnPublish' callbacks debug logs can cause an infinity loop of log entries being logged over and over again when running under Elastic-Agent. Those log entries are removed to prevent this from happening in the future. The monitoring Filebeat uses a drop processor to drop its own logs, however if log level is debug the `onFilteredOut` callback debug log will log all dropped events, because it is done by the monitoring Filebeat, it will be picked up, dropped and logged again, creating this infinity loop. (cherry picked from commit dde4079)
#36487) The 'onFilteredOut' and 'onDroppedOnPublish' callbacks debug logs can cause an infinity loop of log entries being logged over and over again when running under Elastic-Agent. Those log entries are removed to prevent this from happening in the future. The monitoring Filebeat uses a drop processor to drop its own logs, however if log level is debug the `onFilteredOut` callback debug log will log all dropped events, because it is done by the monitoring Filebeat, it will be picked up, dropped and logged again, creating this infinity loop. (cherry picked from commit dde4079) Co-authored-by: Tiago Queiroz <[email protected]>
…c#36399) The 'onFilteredOut' and 'onDroppedOnPublish' callbacks debug logs can cause an infinity loop of log entries being logged over and over again when running under Elastic-Agent. Those log entries are removed to prevent this from happening in the future. The monitoring Filebeat uses a drop processor to drop its own logs, however if log level is debug the `onFilteredOut` callback debug log will log all dropped events, because it is done by the monitoring Filebeat, it will be picked up, dropped and logged again, creating this infinity loop.
Proposed commit message
The 'onFilteredOut' and 'onDroppedOnPublish' callbacks debug logs can cause an infinity loop of log entries being logged over and over again when running under Elastic-Agent. Those log entries are removed to prevent this from happening in the future.
The monitoring Filebeat uses a drop processor to drop its own logs, however if log level is debug the
onFilteredOutcallback debug log will log all dropped events, because it is done by the monitoring Filebeat, it will be picked up, dropped and logged again, creating this infinity loop.Checklist
- [ ] I have commented my code, particularly in hard-to-understand areas- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration files- [ ] I have added tests that prove my fix is effective or that my feature worksCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.## Author's Checklist## How to test this PR locallyRelated issues
## Use cases## Screenshots## Logs