Adding functionality to death letter index instead of dropping events.

[mjmbischoff]

What does this PR do?

Allows the user to specify the policy (/action) of how to deal with non-indexable events. We default to drop dropping the event which is current behavior. Alternatively one can specify death_letter_index to index into the specified index name

output.elasticsearch:
    non_indexable_policy.dead_letter_index:
        index: "dead-letter-index"

output.elasticsearch:
    non_indexable_policy.drop: ~

Why is it important?

Currently when there's a mapping conflict events are dropped, this is undesirable for a lot of customers. While filebeat technically doesn't violate it's guarantee that events are delivered at least once, it does cause the event to be lost. Which is both undesirable and unexpected behavior for most customers.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

See integration test,
Configure filebeat with the ES ouput and specifying the action and index
Then creating a mapping conflict:

{ "bar": 0 }
{ "bar": "bar0" }

Related issues

Use cases

When a mapping conflict occurs the message is kept and indexed into an alternative index. This can then later be re-ingested after either the data or the index (the alias points to) is changed.

Other remarks

• The index currently must reside on the same ES cluster. Changing this, if desired, would be a separate PR.
• The format we use to store the events on the death letter index is message fields (containing the original message with escaped json), error.type (http code) and error.message (description)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-08-02T15:42:58.813+0000

• Duration: 134 min 48 sec

• Commit: 0304845

Test stats 🧪

Test	Results
Failed	0
Passed	49421
Skipped	5318
Total	54739

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	49421
Skipped	5318
Total	54739

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[kvch]

All in all, great PR with good tests. I have a few nits that are not required to be addressed if you do not want to. However, the configuration should be reworked a bit to be more user friendly.

[mjmbischoff]

dead letter vs death letter 😂

Made the changes to the config setuo, also reduced some more parameters by making additional functions a function of client
Added tests for the config.
If our CI is happy tomorrow, I'll have a look at the docs.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b deathletter-index upstream/deathletter-index
git merge upstream/master
git push upstream deathletter-index

[mjmbischoff]

@kvch think it's all good now

[kvch]

Please update the PR description with the new configuration format. Also, as it is a new feature, please make it beta with cfgwarn.Beta and beta[] in the documentation.

Otherwise, the PR LGTM.

[kvch]

Awesome \o/

[kvch]

Once you fix the lint errors, you can merge it. Thanks for the PR!