[Fleet] Create consistent framework for Fleet settings

[kpollich]

Fleet has many "settings" objects that share similarities in their usage, UI/UX treatment, and persistence implementations. e.g.

• Fleet Server Hosts
• Outputs
• Agent Binary Downloads
• Proxies
• (soon) Hostname behavior (FQDN vs default)

We present these settings on Fleet's /settings page, and it's clear how similar they all are from the UI here:

Because we have such a common use case of "add a setting to Fleet to change the behavior of agent", it would be wise to develop a consistent "framework" implementation for these settings. For instance, driving the UI shown above from a config file rather than having to spin up a bunch of boilerplate API endpoints, saved object definitions, and React components would make us a lot more flexible with settings in the future, and would even allow for the agent team to introduce new settings on their own without involvement from Fleet UI engineers.

Here's an example @joshdover proposed for the FQDN field represented by YML:

- name: Host name
  type: dropdown
  options:
    - title: Host
      value: false
    - title: FQDN
      value: true
  policy_field: providers.host.use_fqdn

We'd need to develop a schema for these settings, and some docs for how to introduce new ones. To prove out the implementation, we'd want to migrate our existing settings implementations to the new config-based approach as well. This would involve a somewhat extensive saved object migration to convert from discrete saved objects to a singular fleet-settings-record saved object definition or something along those lines.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[joshdover]

@cmacknz here's the issue I mentioned earlier today. We could use something like this to unblock a lot of the agent changes we need to make and allow the teams to move more independently.

@amitkanfer @jlind23 @jen-huang what do you all think about prioritizing this sooner in the Fleet & Integrations roadmap? There may be a number of tasks (planned and not yet discovered) that could be mostly "automated" by a generic settings framework like this. We could start with rolling this out only with support in the API first, and then expand it to include generating UI.

[nchaulet]

I think we may want to differentiate what settings should be generalized in a framework, for example outputs have more domain rules, (some integrations like fleet server, apm, do not support all the output types, ...)

But it will be great to have a framework for settings like FQDN that are simple key:value per agent policy

[amitkanfer]

There may be a number of tasks (planned and not yet discovered)

like?

[kpollich]

Shipper settings come to mind. Any other policy-level flags/settings/enums we want to support would fall under this implementation as well.

Here's an example I found where ignore_older would become a policy-level setting: #122892. Processors could also be handled by this framework.

Essentially we'd be building a configuration-first way to add 1:1 policy fields to Fleet UI without having to write code. So perhaps something like this could become possible:

# x-pack/plugins/fleet/policy_settings.yml
- name: Processors
  type: yml
  policy_field: processors
  schema: # Some validation rules, etc 

[kpollich]

@nchaulet's point about anything with domain logic and complex requirements like outputs needing to be excluded here is important. I don't think we could successfully build up a configuration-first data model that supports the Kafka output for example. Just look at the topic filter UI we're building for an example of where things become too difficult to capture in plain YML. Basic policy fields should be wrapped up in this consistent "framework" though.

[cmacknz]

like?

Fleet currently doesn't expose the entire agent policy. There are good reasons for this in some cases, but not in others. We don't expose all of the monitoring configurations, but it would be extremely valuable to expose many of them in some way: https://github.com/elastic/elastic-agent/blob/7506b3d89477c8643845c632b3820a9b55ca0e27/elastic-agent.reference.yml#L114-L150

There are many development options in the agent policy that we cannot set easily for Fleet managed agents. Partially this is because they are only known the the agent engineers. Having an easy way for agent engineers to extend the Fleet UI to add these simple settings would be beneficial. Otherwise every time we need something exposed in the Fleet UI it automatically becomes a cross-team project, which adds significant friction to the development process.

For another example, now that we have feature flags we may find uses of them that are not tied to obvious user facing features. Having a way for engineers to easily extend this for experimentation and adding safe guards in new releases without requiring time from the Fleet UI team would make this much easier to use and more likely to be used for this purpose.

[amitkanfer]

Understood. Why not extending the feature flags framework with an option to specify whether or not the setting is shown or hidden from the UI?
In my past life I had a similar framework, where the engineer/user could set a flag in the browser's local storage, refresh the page, and then a few more feature flags showed on the screen. I fully understand the need, let's just aim for the 20% that will solve 80% of the problem

[joshdover]

Why not extending the feature flags framework with an option to specify whether or not the setting is shown or hidden from the UI?
In my past life I had a similar framework, where the engineer/user could set a flag in the browser's local storage, refresh the page, and then a few more feature flags showed on the screen. I fully understand the need, let's just aim for the 20% that will solve 80% of the problem

Yeah I think this comes down to having a central source of truth for what can be in an agent policy. We have no such thing today that is consumable from both our UI code and Agent code. I think this is similar to what I proposed for this yaml format, but would be separate from Kibana.

I'd recommend we explore a json-schema for the agent policy and then use that to drive all parts of the architecture that need to consume it.

[cmacknz]

+1 to json-schema for describing the structure of an agent policy. We've needed a proper schema for policies for a long time, it has just never been our most important problem.

[joshdover]

So there's a pre-req here to begin defining an Agent policy schema that can be leveraged in several places.

Given the short-term roadmap and team capacity, we would need to be very sure this would be a time and maintenance savings. @kpollich or @jen-huang would you be able to see how much of the Fleet backlog could be eliminated by something like this and what effort we could save? Anything around policy or output settings are likely candidates.

[kpollich]

Closed as we're tracking this in an internal issue for now