[Fleet] Simplified preconfigured agent policies

[juliaElastic]

In #108456 we remove Default policies and integrations, which impacts a few dependent projects, that in turn have to preconfigure their agent policies.
https://github.com/elastic/observability-test-environments/issues/2030
elastic/elastic-package#655
elastic/e2e-testing#2039

These preconfigured agent policies are repeated in a few places, and it would be easier to offer a simplified preconfiguration to avoid copying the long agent policy config.

Current preconfiguration:

xpack.fleet.packages:
  - name: system
    version: latest
  - name: elastic_agent
    version: latest
  - name: fleet_server
    version: latest
xpack.fleet.agentPolicies:
  - name: Agent policy 1
    id: agent-policy-1
    namespace: default
    monitoring_enabled:
      - logs
      - metrics
    package_policies:
      - name: system-1
        id: default-system
        package:
          name: system
  - name: Fleet Server policy
    id: fleet-server-policy
    namespace: default
    package_policies:
      - name: Fleet Server
        package:
          name: fleet_server

Proposed preconfiguration:

xpack.fleet.agentPolicies:
  - name: Agent policy 1
    id: agent-policy-1
    add_packages: system, elastic_agent 
  - name: Fleet Server policy
    id: fleet-server-policy
    add_packages: fleet_server

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[juliaElastic]

Copying here the use case proposed by @kuisathaverat:
To start the APM server now you use, and Elastic Agent and the APM integration, it is true that the current default policy does not have the APM integration. To have a way to create a policy for an integration with the default values in a single config step simplifies the configuration. It is not directly related to remove the default policies but it could reuse the work in #124030
The use case could be something like:

GIVEN a user wants to connect an APM agent
AND an Elastic stack cluster up and running
AND a Fleet Server up and running
WHEN adding an APM package to Kibana creates a APM policy with the default values

xpack.fleet.packages:
    - name: elastic_agent
      version: latest
    - name: apm
      version: latest
      create_default: 
        name: APM default policy
        id: apm-default-policy

THEN a new policy is created with id apm-default-policy ready to use with an Elastic Agent
I am thinking on environments like our test environments that are configured as code for everything.

[kuisathaverat]

the approach suggested by @nchaulet at #108456 (comment) is also a good one.

[axw]

@juliaElastic this looks great. A couple of questions.

Would it be reasonable to make the agent policy name optional, and have it default to the policy ID?

Let's say I want a single agent policy with fleet_server and apm packages. The fleet_server package can have defaults, while we want to set some vars for apm. How might that look?

I can imagine Fleet being able to handle something like this:

xpack.fleet.agentPolicies:
  - id: fleet_and_apm
    add_packages:
      - fleet_server
      - name: apm
        inputs:
          -  vars:
                host: ':8200'
                secret_token: abc123

This would require:

• add_packages would need to support an array of either package names, or a package policy
• package policies in this context could be minimal: just the name, and inputs (APM only has one input type, so I omitted it in the example)
• vars could be a simple YAML object (k=v), or list of (name, type, value, frozen) tuples.

As mentioned in #128338, what I'm hoping to do is simplify preconfiguration via environment variables. The above would be something like:

XPACK_FLEET_AGENT_POLICIES={'xpack.fleet.agentPolicies': [{'id': 'fleet_and_apm', 'add_packages': ['fleet_server', {'name': 'apm', 'inputs': [{'vars': {'host': ':8200', 'secret_token': 'abc123'}}]}]}]}

Still not as terse as I'd like, but certainly better than what's possible today. Can we do better?

[juliaElastic]

@axw Thanks for the suggestions.

I agree with Josh's comments here: first it would be best to try to simplify the existing config by making fields optional and set defaults (e.g. no need to specify packages that are already mentioned in package_policies), instead of coming up with new config options.

[GeorgeGkinis]

Please consider using the same yaml configuration for the policies retreived by api/fleet/agent_policies/ so that we can create policies in the UI and then be able to paste them into out deployment under xpack.fleet.agentPolicies

We are really struggling to get our policies yaml's right, add them to git and deploy them in ECK.

Kibana complains about the format beeing incorrect.
Related to #128314

[WildDogOne]

I agree with @GeorgeGkinis it would be nice to have a way to export into YAML. Or make the documentation more clear. For example installing the package elastic defend is a huge pain right now.