[Fleet] Use fleet server indices for enrollment keys and to list agents with a feature flag

[nchaulet]

Summary

Part of #87372

Introduce a feature flag to start using fleet server indice xpack.fleet.agents.fleetServerEnabled: true

If this flag is enabled we will use a different service that read and write for .fleet-enrollment-key instead of saved object.

This is a draft PR to explore what a feature flag for the development of Kibana with Fleet Server will look like

How to test that PR?

With the feature flag off

if the feature flag is off xpack.fleet.agents.fleetServerEnabled: false you should be able to create a policy, enroll agents, unenroll, all the actions as before, (this PR should not have any impact)

With the feature flag on

with the feature flag on xpack.fleet.agents.fleetServerEnabled: true you should be able to enroll agent, and rollup a policy via fleet-server, (turning the flag off again is not supported)

You should respect the following order (for now):

• Start fleet-server
• Create a custom kibana system user with the right permissions (see below)
• start kibana with flag and the custom kibana user
• Setup fleet and agent in Kibana set the kibana url to the fleet server url (by default http://localhost:8000)
• enroll an agent to fleet server, you can trigger a policy change the other actions are not yet working.

Migration

During the plugin start we will transform all of the existing saved object to .fleet-* document, I discussed this kibana platform and it should work for us.

Indices creation

In this PR we rely on fleet server creating the indice, (in the future the indice should be created by an ES plugin)

So the first step is to run fleet server.

Kibana system user

The kibana_system will need to have access to .fleet* indices.

In this PR I used a custom user as the kibana system user created like

POST /_security/role/kibana_fleet_system
{
   "cluster" : [
      "all"
    ],
    "indices" : [
      {
        "names" : [
          ".fleet*"
        ],
        "privileges" : [
          "all"
        ]
      }
    ]
}



POST /_security/user/kibana_fleet_system
{
  "password" : "changeme",
  "roles" : [ "kibana_system", "kibana_fleet_system" ]
}

then in kibana.dev.yml

elasticsearch.username: 'kibana_fleet_system'
elasticsearch.password: 'changeme'

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[jen-huang]

Didn't review test files or security_solution files. Left questions and comments mostly regarding the ES agents CRUD service.

[nchaulet]

Yes it's normal you cannot enroll an agent to kibana for now, you need to enroll the agent to fleet server. I will add the support for the new indices and Kibana enroll in a future PR the whole migration plan is here #87372

[jen-huang]

@nchaulet How can I enroll into Fleet server? Can you update the PR description with more testing steps around what is expected behavior?

[nchaulet]

@jen-huang thanks for taking some time on that PR, and sorry for the missing expectations, I udpated the PR description and created a small video on what to expect when testing with the flag set to on.

What to test on that PR:

• nothing break when the flag is off
• you can enroll and agent via Fleet server when the flag is on.

fleet-server.mp4

[jen-huang]

Thanks for the updated instructions and video. I ran through the Fleet server enabled scenario again today and am seeing all of the behavior described. My agent has logs and I see data streams come in too. The only issue I see is that the status bar doesn't reflect the status of the agents, but I think that might be due to the current limitation of "can trigger a policy change the other actions are not yet working"?

Tomorrow I will run another test with Fleet server disabled and give the code another look too.

[nchaulet]

@jen-huang yes the agent status with Fleet server are not yet working if it's okay with you I will fix it in a following PR as there is already a lot of change here

[nchaulet]

@kevinlog I would love a review from endpoint here, mostly checking if the flag set to false it's not breaking anything

[jen-huang]

@nchaulet Yeah no worries, just wanted to confirm that it's a known limitation right now.

[kevinlog]

@nchaulet

I checked it out and tested with some basic Endpoint flows with xpack.fleet.agents.fleetServerEnabled: false, Endpoints deploy correctly and I'm still able to change policies, etc.

From that perspective, the change looks good from our end - I'll leave the review of the actual code to Fleet devs.

[jen-huang]

Tested locally with and without Fleet server enabled. With Fleet server, ran into known limitations but overall things work as described. Without Fleet server, things worked normally and I confirmed that .fleet indices were not created.

Left some code comments but not blocking. Thanks for your patience during my reviews of this and good work on getting this first major piece completed!

[jen-huang]

not a blocker but would be good to have tests for this

would agent documents need to be migrated too?

[nchaulet]

Yes agents would need to be migrated too, I am probably going to address that and add tests for that in a following PR as this PR is already complex

[jen-huang]

why do we wait for licensing before starting migration?

[nchaulet]

we need licensing to be available so the SO client is correctly initialized. it probably need a comment here

[nchaulet]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 533e506

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	377.0KB	377.9KB	+860.0B

History

• 💔 Build #100158 failed 18293e9
• 💚 Build #100102 succeeded 74e89bc
• 💚 Build #99597 succeeded cbafae0
• 💚 Build #99466 succeeded b888fe2
• 💔 Build #99398 failed 60eab63

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream