[Fleet] Add a pipeline processor to all the ingest_pipeline installed by fleet

[nchaulet]

Description

Related to #133740

To allow users to write custom ingest pipeline, we add a new processor at the end at the end of the fleet install ingest_pipeline that will call the user custom ingest pipeline if it exists.

Release notes

That PR will enable user to write custom ingest pipeline for Fleet installed datastream.

Tests and how to manually test?

This features is tested with some unit tests and a new API integration test that index data with and without custom pipeline

If you want to manually test it you can use the custom log integration

1. check that the pipeline processor is correctly added to the datastream ingest pipeline

2. create a custom pipeline logs-log.log@custom (could be done in the ingest pipeline UI)

3. send data to the datastream and the custom ingest pipeline is called

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

I ran through the whole process locally and everything works as expected. The code is sensible and has tests so no complaints there. One very minor nitpick on a method name but otherwise LGTM! Good luck with all the test failures 😅

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 35978b8
• Storybooks Preview
• Webpack Bundle Analyzer

Metrics [docs]

✅ unchanged

History

• 💛 Build #51410 was flaky 9abce63
• 💔 Build #51396 failed 324eb1f
• 💔 Build #51374 failed 93a8ca6

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @nchaulet