Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop non-matching logs inside elasticsearch filebeat module 8.0 pipelines #30428

Closed
matschaffer opened this issue Feb 16, 2022 · 1 comment · Fixed by #30440
Closed

Drop non-matching logs inside elasticsearch filebeat module 8.0 pipelines #30428

matschaffer opened this issue Feb 16, 2022 · 1 comment · Fixed by #30440
Assignees
Labels
Feature:Stack Monitoring Filebeat Filebeat Team:Integrations Label for the Integrations team

Comments

@matschaffer
Copy link
Contributor

#30018 added support for ES 8.0's ECS formatted logs.

Since the logs are ECS format we opted for minimal processing and removed the drop found in the 7.x log processors.

Unfortunately in cases like kubernetes, the combination of multiple file sets (server, audit, etc...) and using a single mixed stream (stdout) means we end up double-ingesting the same messages across multiple pipelines and storing the duplicates.

To work around this we can add a drop to the 8.0 pipelines for the elasticsearch module.

@matschaffer matschaffer added Filebeat Filebeat Team:Integrations Label for the Integrations team Feature:Stack Monitoring labels Feb 16, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Stack Monitoring Filebeat Filebeat Team:Integrations Label for the Integrations team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants