[Fleet] Add input settings at the agent policy level

[mostlyjason]

We have several use cases for adding input settings at the agent policy level:

1. Adding ignore_older to prevent stability issues resulting from sending large quantities of historical logs (from an ER). This also needs a default value to provide a good experience to first-time users.
2. [Fleet - Elastic Agent] Add additional fields to Agent policies, output. #99653
3. https://github.com/elastic/ingest-dev/issues/2442

These are generally advanced settings that most users will not need to modify. Advanced users are likely comfortable applying settings as YAML blocks. It'd also allow us to solve for many use cases at once. We could convert these to specific UI elements in the future to make the UX easier. We'd likely need a way to differentiate the settings for each input type, such as by having a YAML block for each type. It'd be nice offer the ability to use variables for more dynamic behavior.

Questions:

1. Are there any input settings we anticipate the need to deprecate or change in the future, such as with agent architecture V2?
2. Should we offer any settings as a UI feature? Note that not all settings are available globally.
3. What should the default value for ignore_older be? 1 hour would prevent unlimited data, while still allowing dashboards to populate for a nice getting started experience.
4. Should we offer similar options at the integration, input or stream level? Could these be controlled via conditions as an alternative to duplicating configuration?
5. How can users preconfigure this for IaC use cases? Can they apply settings to multiple input types without duplication?

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[mostlyjason]

The highest priority is the setting to ignore old logs since its needed for stability. We'll also want to review the set of global input settings to define the scope for this issue. @nimarezainia discussed decoupling processors if necessary until we have better definition for V2 architecture.

[juliaElastic]

@elastic/elastic-agent-control-plane Hey, could you let us know the answer to this question?

Are there any input settings we anticipate the need to deprecate or change in the future, such as with agent architecture V2?

[jlind23]

ping @ph @cmacknz

[ph]

@juliaElastic I don't know that answer and the Agent wouldn't know it too, since we will push down the validation to the input.

Also, I don't think we have audited the input settings to see if we can deprecate some.

[cmacknz]

I don't think we have audited the input settings to see if we can deprecate some.

Correct, we haven't done this. Is the intent here that we have the opportunity to avoid exposing input/beat settings in agent policies that we don't want to maintain in the future?

[mostlyjason]

@cmacknz yes that is the intent. This will be the first time we document input settings under elastic agent, so it could be an opportunity to audit and simplify.

[nimarezainia]

@cmacknz yes that is the intent. This will be the first time we document input settings under elastic agent, so it could be an opportunity to audit and simplify.

We (I) have to create the full set of input parameters and we collectively audit. Some may no longer be applicable.

[cstegm]

Hi @mostlyjason,

Not sure if this is on you Use Case list, so:
In my case i just want to add fields, labels or tags to a specific groups of agents to identify them later easily.

Example i want to add a label - something like "application: webserver" to my agents that have the webservers policy.

By seeing this: Maybe it is also a good idea to include the policy name/id as default to the data to make it easier to debug when an integration has an issue?

[mostlyjason]

@cstegm yes this use case is covered by number 2 in the list above. You could add "application: webserver" or the policy name as fields at the agent policy level. Fields are basically key-value pairs that you can set to whatever you like, and they get added to the event data. We are also considering the ability to set fields at the integration or input level, since each host can have more than one application running.