[Fleet] Make input IDs unique in agent policy yaml

[jen-huang]

Summary

Resolves #125844. This PR makes input IDs in the final generated agent policy unique by building it from the parent package policy ID + input type + the associated policy template name (if any).

When a package policy is built, its streams are grouped by input type+policy template, so that combination should be unique (and stable) on each package policy.

This screenshot shows an example of unique IDs being generated for AWS billing and Cloudtrail inputs:

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 46f1826
• Storybooks Preview
• Webpack Bundle Analyzer

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	110.2KB	110.3KB	+55.0B

History

• 💔 Build #29165 failed 39ff528
• 💔 Build #29010 failed 4e093ba

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jen-huang

[nchaulet]

LGTM 🚀

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.1	Backport failed because of merge conflicts You might need to backport the following PRs to 8.1: - [Security Solution] Unskip alerts related Cypress tests (#127187) - [demo env] Skip docker cloud build (#127459) - unskipping a11y painless lab test (#127082)
❌	7.17	Backport failed because of merge conflicts You might need to backport the following PRs to 7.17: - unskipping a11y painless lab test (#127082)
❌	8.0	Backport failed because of merge conflicts You might need to backport the following PRs to 8.0: - unskipping a11y painless lab test (#127082)

Manual backport

To create the backport manually run:

node scripts/backport --pr 127343

Questions ?

Please refer to the Backport tool documentation

[kevinlog]

@ph @jen-huang before backporting and releasing this, we need to resolve this downstream effect: #127570

More details on what we're seeing on our side in this this comment: #127570 (comment)

Essentially, the Endpoint is now adding the endpoint- prefix to all policy IDs which is breaking some of our UI. Let's figure out the right way to resolve this before it goes out.

cc @ferullo @paul-tavares

[nchaulet]

@kevinlog, currently we do not persist that id, but if we persist that id on the package policy saved object, you will be able to query the package policy saved object with the id to retrieve the package policy id, it is something that will work for you?

We can expose a service getPackagePolicyForInputId() and the package policy api can support a query param ?input_id= and we will take care to support old input id too without the prefix.

[ph]

@kevinlog Thanks for pointing this out, sorry for not having communicated that clearly with everyone. For the Agent, we will merge the changes because they currently affect Filebeat. We haven't changed the logic on the endpoint configuration. We are still giving the complete input block.

@nchaulet @jen-huang I haven't considered that, but would the following be true:

When a user upgrades they will still be using the old ids for input that aren't unique?

[nchaulet]

When a user upgrades they will still be using the old ids for input that aren't unique?

Yes until they do something that trigger a policy change, (changing anything in the policy or a global change to outputs or settings)

[ph]

@kvch @belimawr would that be a problem with the filestream input ?

[paul-tavares]

@nchaulet

Re: using input_id to query fleet's package policies

I'm not sure yet of the full impact of the changes to the id on our side, so its hard to tell if what you suggest would work. FYI - I think we also use these IDs in some cases to query Agent Policies using KQL. We might also have impacts to our server code.

I would also like to request that the actual Package Policy ID be included in the YAML if the id is no longer going to be actual Package policy id. We have some other info about the package policy already (revision, name, type), so including the ID will help us ensure we are able to cross reference data back to fleet policies:

- id: endpoint-066fdf94-a6bc-4185-b27d-0245f96e3ab9
    name: Security
    revision: 4
    type: endpoint
    use_output: default
    meta:
      package:
        name: endpoint
        version: 1.5.0

We can then maybe see if we can get Endpoint to return that back to us in their messages and we can start to use it again.

[paul-tavares]

Question:

Could this id remain unchanged and a new property be added that store the unique ID that is needed by you all?

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 127343 or prevent reminders by adding the backport:skip label.

[belimawr]

@kvch @belimawr would that be a problem with the filestream input ?

The input IDs changing from a non-unique to a unique one? That will duplicate the events once.

If a filestream input that did not have an ID set gets an ID, we can migrate the data in the registry and not duplicate events.

My PR #30717 focus on adding IDs to filestream inputs that did not have one. Other cases have not been covered there.

[amolnater-qasource]

Hi @joshdover
We have revalidated this on latest 8.2 Snapshot and found it working fine now.

• Unique ids are provided to integrations in agent policy yaml.
• Even after doing changes unique id remains same.

Screenshots:

Build details:

VERSION: 8.2.0
BUILD: 51328
COMMIT: 2a93e80574f9c1052c541bc4083a440a623899e8

Hence marking this as QA:Validated.
Please let us know if anything else is required from our end.
Thanks