Add drop and explicit tests to avoid duplicate ingest of elasticsearch logs

[matschaffer]

What does this PR do?

Adds a "drop" to the elasticsearch pipeline as well as explicit "mixed" test logs to ensure we won't attempt to ingest logs across mismatched pipelines.

Why is it important?

Without it, the elasticsearch slowlog pipeline will attempt to ingest all 5 file sets shipped by the elasticsearch filebeat module.

Additionally the test cases help guard against removal of the drop processors.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

See https://www.elastic.co/guide/en/beats/devguide/master/filebeat-modules-devguide.html#_test for setup instructions. Specify TESTING_FILEBEAT_MODULES=elasticsearch to test the elasticsearch module directly.

Related issues

Fixes #30428
Related #30164
Related #16540

Use cases

In Elasticsearch log4j2.properties config is defined that server, deprecation, slowlog and audit logs are written to Console:

grep -R "Console" log4j2.properties
appender.rolling.type = Console
appender.deprecation_rolling.type = Console
appender.index_search_slowlog_rolling.type = Console
appender.index_indexing_slowlog_rolling.type = Console
appender.audit_rolling.type = Console

On the kubernetes node, where elasticsearch container is running, all those logs will be in one file:

/var/log/containers/__elasticsearch-.log

Filebeat pod has the whole folder /var/log mounted and reads log files from /var/log/containers/.

The elasticsearch module has 5 filesets which will lead to reading the kubernetes log 5 times and shipping every message to each pipeline.

This works around the issue of duplicate log storage by dropping at the top of the pipeline.

[elasticmachine]

Pinging @elastic/stack-monitoring (Stack monitoring)

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-02-21T01:30:45.265+0000

• Duration: 99 min 59 sec

Test stats 🧪

Test	Results
Failed	0
Passed	8673
Skipped	1122
Total	9795

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[matschaffer]

/test

[matschaffer]

The docs failure seems to be unrelated:

14:33:31 INFO:build_docs:To github.com:elastic/built-docs
14:33:31 INFO:build_docs: ! [rejected]                  beats_30440 -> beats_30440 (fetch first)

If the above test doesn't fix it, I'll merge main on Monday.

[dedemorton]

@elasticmachine run elasticsearch-ci/docs

[klacabane]

Nice set of tests :)

[tetianakravchenko]

Thank you for adding tests!
I've tried those pipeline adjustment as well as audit pipeline changes from #30164 on k8s environment - all works, no duplication

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 30428-drop-non-matching-es-logs upstream/30428-drop-non-matching-es-logs
git merge upstream/main
git push upstream 30428-drop-non-matching-es-logs