[Fleet]: Simplify policy preconfiguration

[axw]

Describe the feature:

Simplify Fleet policy preconfiguration variables such that they can more easily be used in environment variables, such as in docker-compose.

Users should not need to customise kibana.yml to preconfigure a policy. However, at the moment, this is really the only practical way to do so. Preconfiguration of APM and Fleet Server looks something like this:

xpack.fleet.packages:
  - name: fleet_server
    version: latest
  - name: apm
    version: latest
xpack.fleet.agentPolicies:
  - name: Fleet Server
    id: fleet-server-policy
    is_default_fleet_server: true
    is_managed: false
    namespace: default
    package_policies:
      - name: fleet_server
        id: default-fleet-server
        package:
          name: fleet_server
  - name: APM
    id: apm-policy
    is_managed: false
    namespace: default
    package_policies:
      - name: apm
        id: apm
        package:
          name: apm
        inputs:
          - type: apm
            vars:
              - name: host
                value: :8200
              - name: secret_token
                value: abc123

Setting all of that via environment variables is currently impractical.

We have a couple of broad options, which are not mutually exclusive:

• Simplify what we have. For example, we could infer the packages to install from the preconfigured policies.
• Define more specific configuration variables for stack-aligned Elastic products (namely apm-server and fleet-server), e.g. xpack.apm.defaultAgentPolicy which would be APM-specific. APM supports only a single input, so this would allow us to minimise the APM portion of the above preconfiguration to something like xpack.am.defaultAgentPolicy: name:apm-policy vars:host=':8200',secret_token='abc123'.

Some additional thoughts from @joshdover:

• We should try to avoid adding new configuration keys if possible and instead try to make the existing ones have better defaults, so you don’t have to specify as many options to just get something basic working
• We should consider automatically assuming xpack.fleet.packages based on the packages used in xpack.fleet.agentPolicies
• We shouldn’t do anything special for Docker using env vars. We should add kibana.yml settings that can be used in other use cases and make them available as env vars just like other Kibana configuration
• Maybe package policy templates should be able to define “dev-only” / “quick start" default values for such use cases instead of special behavior or configs for APM and Fleet Server

Describe a specific use case for the feature:

Users who want to try out the Elastic APM product, or otherwise run Elastic APM, in a small development (laptop) environment with docker-compose.

It should be possible and straightforward to define all required Elastic components (Elasticsearch, Kibana, and Elastic Agent) in docker-compose.yml, alongside the user's APM-instrumented services. Having a working APM product should be possible by just running docker-compose up -d in a clean environment. This would:

1. Start Elasticsearch, Kibana, and Elastic Agent
2. Install the apm and fleet_server integration packages
3. Create an agent policy with the fleet_server and apm integrations, with the APM host and secret token injected
4. Bootstrap Elastic Agent/Fleet Server using the superuser username/password and above mentioned policy ID

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[juliaElastic]

there is a similar enhancement request created here: #124030

[joshdover]

Let's continue the discussion in #124030