Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fleet] Setup fleet server indices in Kibana without packages #90658

Merged
merged 14 commits into from
Feb 16, 2021

Conversation

nchaulet
Copy link
Member

@nchaulet nchaulet commented Feb 8, 2021

Summary

Related to #87372

Replace the setup of fleet server indices by the integration package, by creating them using the kibana_system during the plugin start.

Details

Foreach index we create an index like this .fleet-agents_1 and an alias to allow for destructive migrations in the future if we need them.

For each index we create a hash of the mapping and we update the index only if the hash do not match.

The kibana_system role as all access to .fleet-* so it can create the indices.

How to test

Starting from a fresh ES and kibana with this flag

xpack.fleet.agents.fleetServerEnabled: true

You should see the .fleet indices created.

@nchaulet nchaulet added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Feature:Fleet Fleet team's agent central management project v7.12.0 labels Feb 8, 2021
@nchaulet nchaulet self-assigned this Feb 8, 2021
@nchaulet nchaulet added Team:Fleet Team label for Observability Data Collection Fleet team and removed Feature:Fleet Fleet team's agent central management project labels Feb 8, 2021
@nchaulet nchaulet marked this pull request as ready for review February 8, 2021 19:33
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@nchaulet nchaulet requested review from a team and kobelb February 8, 2021 19:33
@nchaulet
Copy link
Member Author

nchaulet commented Feb 9, 2021

@elasticmachine merge upstream

@nchaulet nchaulet requested review from jfsiii and jen-huang February 9, 2021 15:39
@nchaulet
Copy link
Member Author

nchaulet commented Feb 9, 2021

@elasticmachine merge upstream

@nchaulet
Copy link
Member Author

@elasticmachine merge upstream

@nchaulet nchaulet requested review from a team and removed request for a team February 10, 2021 20:03
Copy link
Contributor

@kobelb kobelb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making these changes @nchaulet, they will address my concerns expressed in #89251

I didn't look super closely at the code, but it doesn't look like we're performing any retries in the case of failure. This might be something we want to add incase there's a transient failure.

'Extra setup is needed to be able to use central management for agent, please visit the Fleet app in Kibana.'
);
}
if (appContextService.getConfig()?.agents?.fleetServerEnabled) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should have another condition to check on if central management is enabled yet? setting up central management === setting up Fleet server, according to the code here. what do you think? of course, this would mean we trigger setup after user clicks to enable central management too.

Copy link
Member Author

@nchaulet nchaulet Feb 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's okay to create the indices even if central management is not enabled later, and doing it here outside of central management setup allow us to have a seamless migration

x-pack/plugins/fleet/server/services/fleet_server/index.ts Outdated Show resolved Hide resolved
x-pack/plugins/fleet/server/services/fleet_server/index.ts Outdated Show resolved Hide resolved
Comment on lines +26 to +27
['.fleet-policies', ESFleetPoliciesIndex],
['.fleet-policies-leader', ESFleetPoliciesLeaderIndex],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"policies" here is a bit confusing as we already have the concept of agent and integration policies within fleet. can you help me understand the purpose of these indices?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will try to explain what are these indices, first Fleet Server do not know about kibana saved object so all the data used by fleet server come from these indices:

.fleet-policies contains a full policy as it's going to be distributed to the agent (except the API keys)

.fleet-policies-leader it's not used in Kibana is used to have a policy coordinator per policy in Fleet server for future features.

Copy link
Contributor

@skh skh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this works as described. After starting Kibana with xpack.fleet.agents.fleetServerEnabled: true, and without triggering the Fleet setup endpoints, I see the following indices and aliases being created.

Indices

.fleet-actions_1
.fleet-policies_1
.fleet-servers_1
.fleet-policies-leader_1
.fleet-enrollment-api-keys_1
.fleet-agents_1

Aliases

.fleet-servers                .fleet-servers_1                - - - -
.fleet-agents                 .fleet-agents_1                 - - - -
.fleet-policies               .fleet-policies_1               - - - -
.fleet-actions                .fleet-actions_1                - - - -
.fleet-enrollment-api-keys    .fleet-enrollment-api-keys_1    - - - -
.fleet-policies-leader        .fleet-policies-leader_1        - - - -

Is this expected behavior? Would it be possible to document in the initial PR descriptions what these indices are for?

@nchaulet
Copy link
Member Author

@skh yes creating the indices and the aliases is the expected behavior this will allow us to run destructive migrations in the future if needed

I will try to describe what indice is used for what

.fleet-servers                 Contains all the existing fleet server
.fleet-agents                  Contains all the agent this replace the existing SO agent
.fleet-policies                Contains the policy as it will be send to the agent (similar to the SO action we have today)
.fleet-actions                 Contains agent action like UNENROLL, UPGRADE, ...
.fleet-enrollment-api-keys     Contains the enrollment API keys replace the existing SO enrollment api key
.fleet-policies-leader         Contains the fleet server leader to support feature like having one input run only by one agent in the future

@skh skh self-requested a review February 16, 2021 14:46
Copy link
Contributor

@skh skh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nchaulet Thanks for the additional explanation.

Copy link
Contributor

@jen-huang jen-huang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes and additional explanation 🚀

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
fleet 755.2KB 755.2KB +1.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
fleet 367.2KB 367.5KB +307.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.12.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants