[Fleet - Elastic Agent] Add additional fields to Agent policies, output.

[scottdfedorov]

@mostlyjason Asked me to open a separate issue for suggestion of the ability to add fields, KV pairs to Agents. I misunderstood #76844 thinking it to be the same topic.

The Custom logs integration allows additional "Custom configuration" that allows adding some additional items, like adding additional fields to be added to the output of the Agents.

This feature does not appear on the Agent itself or on (most? of) of the individual integrations.

There are many use cases, but the central use case for us is to be able to "group" various things. I think the case most likely to be common to most/all users is in Observability.

For example, we have multiple customers, each with their own sets of servers.

• Customer A
  • Database Server
  • Comm Server
  • App Server
• Customer B
  • Database Server
  • Comm Server
  • App Server
• Customer C
  • Database Server
  • Comm Server
  • App Server

We'd like to be able to add to each policy the additional metadata here like

fields_under_root: true
fields:
  labels.customer: CustomerA
  labels.server_type: CustomerDatabase

We'd use environment variables on the host, but then with those values, we could see metrics for all Database servers, or all Comm Servers, or all Servers for CustomerB, or any other 'type' of server', shared or not.

Jason asked if we'd need other processors as well. I'm sure some users have use cases where they'd benefit from additional processors, but we just need to add constant fields to the output. We handle all processing in either Ingest Node or Logstash (depending on the content).

Let me know if this makes sense or if I can add additional info.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[mostlyjason]

@sorantis @ruflin thoughts on adding custom confirmations more generally to all integrations as an advanced option? The alternative is building UI elements for all the configuration options and it may be some time before we have bandwidth. The YAML is not elegant, but it would unblock users and its available today.

[ruflin]

++ on the custom yaml. We could either add it to all packages or have it by default in the UI.

[nimarezainia]

@mostlyjason, @ruflin and @ph woyuldn't this issue be resolved if we support Global processors?

[ruflin]

It would potentially solve part of the issue where the user wants to add data to all integrations. It does not solve the issue where only some additional data should be added to one integration.

[ph]

exactly, you need a processor both at the input and the output level. In the case of inputs, there is a decision to take concerning the ordering of the processor. Integrations authors can add processors for specific input, so if a user of the system wants to add custom processor or in the above case add new fields we should decide how they get merged with the existing processors defined by the integration

Probably after?

[ntnn]

Is there an update on this issue?