[8.12] [Fleet] Update Fleet's custom ingest pipeline names to avoid collisions + add descriptions to each pipeline (#175448) #175547
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ns + add descriptions to each pipeline (elastic#175448) ## Summary Closes elastic#175254 Ref elastic#168019 Ref elastic#170270 In 8.12.0, Fleet unintentionally shipped a breaking change in elastic#170270 for APM users who make use of a custom `traces-apm` data stream. If a user had previously defined this ingest pipeline to customize documents ingested for the `traces-apm` data stream (defined [here](https://github.com/elastic/integrations/blob/9a36183f0bd12e39a957d2f7bd65f3de4ee685b1/packages/apm/data_stream/traces/manifest.yml#L2-L3), then they would unexpectedly see that pipeline called when documents were ingested to the `traces-apm.rum` and `traces-apm.sampled` datastreams as well. This PR addresses this collision by adding a `.package` suffix to the "package level" ingest pipeline introduced in 8.12.0. So, in 8.12.0 a processor would be defined as such on the `traces-apm.rum` or `traces-apm.sampled` ingest pipeline ``` { "pipeline": { "name": "traces-apm@custom", "ignore_missing_pipeline": true, } }, ``` This PR replaces the pipeline with one that looks as follows: ``` { "pipeline": { "name": "traces-apm.package@custom", "ignore_missing_pipeline": true, "description": "[Fleet] Pipeline for all data streams of type `traces` defined by the `apm` integration" } }, ``` **To be clear: this is a breaking change if you have defined the `traces-apm@custom` integration on 8.12. In 8.12.1, it will no longer be called for documents ingested to the `traces-apm`, `traces-apm.rum`, or `traces-apm.sampled` data streams. You will need to rename your pipeline to `traces-apm.package@custom` to preserve this behavior.** This change also applies to `logs-elastic_agent.*` ingest pipelines. See [this comment](elastic#175254 (comment)) for more information. There is still technically room for a collision, though it's unlikely, if the data stream name is `package`. This will be handled by a package spec validation proposed in elastic/package-spec#699. --------- Co-authored-by: Kibana Machine <[email protected]> (cherry picked from commit 9fe5a66)
botelastic
bot
added
the
Team:Fleet
|
Pinging @elastic/fleet (Team:Fleet) |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
💛 Build succeeded, but was flaky
Failed CI StepsTest Failures
Metrics [docs]
To update your PR or re-run it, just comment with: cc @kpollich |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto8.12:Questions ?
Please refer to the Backport tool documentation