From e78c4ab5f14c3d27d890a4ddc518cd3ab9e9f578 Mon Sep 17 00:00:00 2001 From: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Date: Tue, 7 Sep 2021 09:56:56 -0400 Subject: [PATCH] Add dev_docs for Fleet data model (#109476) (#111388) Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com> --- x-pack/plugins/fleet/dev_docs/data_model.md | 212 ++++++++++++++++++++ 1 file changed, 212 insertions(+) create mode 100644 x-pack/plugins/fleet/dev_docs/data_model.md diff --git a/x-pack/plugins/fleet/dev_docs/data_model.md b/x-pack/plugins/fleet/dev_docs/data_model.md new file mode 100644 index 0000000000000..ec9fa031d09d3 --- /dev/null +++ b/x-pack/plugins/fleet/dev_docs/data_model.md @@ -0,0 +1,212 @@ +# Fleet Data Model + +The Fleet plugin has 3 sources of data that it reads and writes to, these large categories are: +- **Package Registry**: read-only data source for retrieving packages published by Elastic +- **`.fleet-*` Indices**: read & write data source for interacting with Elastic Agent policies, actions, and enrollment tokens +- **Saved Objects**: read & write data source for storing installed packages, configured policies, outputs, and other settings + +## Package Registry + +The package registry hosts all of the packages available for installation by Fleet. The Fleet plugin in Kibana interacts +with the registry exclusively through read-only JSON APIs for listing, searching, and download packages. Read more about +the available APIs in the [package-registry repository](https://github.com/elastic/package-registry). + +By default, the Fleet plugin will use Elastic's nightly 'snapshot' registry on the `master` branch, the 'staging' +registry on Kibana nightly snapshot builds, and the 'prod' registry for release builds. The registry that will be used +can be configured by setting the `xpack.fleet.registryUrl` in the `kibana.yml` file. + +The code that integrates with this registry API is contained in the +[`x-pack/plugins/fleet/server/services/epm/registry`](../server/services/epm/registry) directory. + +## `.fleet-*` Indices + +For any data that needs to be accessible by Fleet Service instances to push updates to, we write and read data +directly to a handful of `.fleet-` Elasticsearch indices. Fleet Server instances are configured with an API key that +has access only to these indices. + +In prior alpha versions of Fleet, this data was also stored in Saved Objects because Elastic Agent instances were +communicating directly with Kibana for policy updates. Once Fleet Server was introduced, that data was migrated to these +Elasticsearch indices to be readable by Fleet Server. + +### `.fleet-agents` index + +Each document in this index tracks an individual Elastic Agent's enrollment in the Fleet, which policy it is current +assigned to, its check in status, which packages are currently installed, and other metadata about the Agent. + +All of the code that interacts with this index is currently located in +[`x-pack/plugins/fleet/server/services/agents/crud.ts`](../server/services/agents/crud.ts) and the schema of these +documents is maintained by the `FleetServerAgent` TypeScript interface. + +Prior to Fleet Server, this data was stored in the `fleet-agents` Saved Object type which is now obsolete. + +### `.fleet-actions` index + +Each document in this index represents an action that was initiated by a user and needs to be processed by Fleet Server +and sent to any agents that it applies to. Actions can apply to one or more agents. There are different types of actions +that can be created such as policy changes, unenrollments, upgrades, etc. See the `AgentActionType` type for a complete +list. + +The total schema for actions is represented by the `FleetServerAgentAction` type. + +### `.fleet-actions-results` + +### `.fleet-servers` + +### `.fleet-artifacts` + +### `.fleet-entrollment-api-keys` + +### `.fleet-policies` + +### `.fleet-policies-leader` + +## Saved Object types + +The Fleet plugin leverages several Saved Object types to track metadata on install packages, agent policies, and more. +This document is intended to outline what each type is for, the primary places it's accessed from in the codebase, and +any caveats regarding the history of that saved object type. + +At this point in time, all types are currently: +- `hidden: false` +- `namespaceType: agnostic` +- `management.importableAndExportable: false` + +### `ingest_manager_settings` + +- Constant in code: `GLOBAL_SETTINGS_SAVED_OBJECT_TYPE` +- Introduced in ? +- Migrations: 7.10.0, 7.13.0 +- [Code Link](../server/saved_objects/index.ts#57) + +Tracks the Fleet server host addresses and whether or not the cluster has been shown the "add data" and +"fleet migration" notices in the UI. + +Can be accessed via the APIs exposed in the [server's settings service](../server/services/settings.ts). + + +### `ingest-agent-policies` + +- Constant in code: `AGENT_POLICY_SAVED_OBJECT_TYPE` +- Introduced in ? +- [Code Link](../server/saved_objects/index.ts#136) +- Migrations: 7.10.0, 7.12.0 +- References to other objects: + - `package_policies` - array of IDs that point to the specific integration instances for this agent policy (`ingest-package-policies`) + +The overall policy for a group of agents. Each policy consists of specific integration configurations for a group of +enrolled agents. + +### `ingest-package-policies` + +- Constant in code: `PACKAGE_POLICY_SAVED_OBJECT_TYPE` +- Introduced in ? +- [Code Link](../server/saved_objects/index.ts#212) +- Migrations: 7.10.0, 7.11.0, 7.12.0, 7.13.0, 7.14.0, 7.15.0 +- References to other objects: + - `policy_id` - ID that points to an agent policy (`ingest-agent-policies`) + - `output_id` - ID that points to an output (`ingest-outputs`) + +Contains the configuration for a specific instance of a package integration as configured for an agent policy. + +### `ingest-outputs` + +- Constant in code: `OUTPUT_SAVED_OBJECT_TYPE` +- Introduced in ? +- [Code Link](../server/saved_objects/index.ts#190) +- Migrations: 7.13.0 + +Contains configuration for ingest outputs that can be shared across multiple `ingest-package-policies`. Currently the UI +only exposes a single Elasticsearch output that will be used for all package policies, but in the future this may be +used for other types of outputs like separate monitoring clusters, Logstash, etc. + +### `epm-packages` + +- Constant in code: `PACKAGES_SAVED_OBJECT_TYPE` +- Introduced in ? +- [Code Link](../server/saved_objects/index.ts#279) +- Migrations: 7.14.0, 7.14.1 +- References to other objects: + - `installed_es` - array of assets installed into Elasticsearch + - `installed_es.id` - ID in Elasticsearch of an asset (eg. `logs-system.application-1.1.2`) + - `installed_es.type` - type of Elasticsearch asset (eg. `ingest_pipeline`) + - `installed_kibana` - array of assets that were installed into Kibana + - `installed_kibana.id` - Saved Object ID (eg. `system-01c54730-fee6-11e9-8405-516218e3d268`) + - `installed_kibana.type` - Saved Object type name (eg. `dashboard`) + - One caveat with this array is that the IDs are currently space-specific so if a package's assets were installed in + one space, they may not be visible in other spaces. We also do not keep track of which space these assets were + installed into. + - `package_assets` - array of original file contents of the package as it was installed + - `package_assets.id` - Saved Object ID for a `epm-package-assets` type + - `package_assets.type` - Saved Object type for the asset. As of now, only `epm-packages-assets` are supported. + +Contains metadata on an installed integration package including references to all assets installed in Kibana and +Elasticsearch. This allows for easy cleanup when a package is removed or upgraded. + +### `epm-packages-assets` + +- Constant in code: `ASSETS_SAVED_OBJECT_TYPE` +- Introduced in ? +- [Code Link](../server/saved_objects/index.ts#328) +- Migrations: +- References to other objects: + +Contains the raw file contents of a package, where each document represents one file from the original package. Storing +these as Saved Objects allows Fleet to install package contents when the package registry is down or unavailable. Also +allows for installing packages that were uploaded manually and are not from a package registry. The `asset_path` field +represents the relative file path of the file from the package contents +(eg. `system-1.1.2/data_stream/application/agent/stream/httpjson.yml.hbs`). + +### `fleet-preconfiguration-deletion-record` + +- Constant in code: `PRECONFIGURATION_DELETION_RECORD_SAVED_OBJECT_TYPE` +- Introduced in ? +- [Code Link](../server/saved_objects/index.ts#328) +- Migrations: +- References to other objects: + - `id` - references the policy ID from the preconfiguration API + +Used as "tombstone record" to indicate that a package that was installed by default through preconfiguration was +explicitly deleted by user. Used to avoid recreating a preconfiguration policy that a user explicitly does not want. + +### `fleet-agents` + +**DEPRECATED in favor of `.fleet-agents` index.** + +- Constant in code: `AGENT_SAVED_OBJECT_TYPE` +- Introduced in ? +- [Code Link](../server/saved_objects/index.ts#76) +- Migrations: 7.10.0, 7.12.0 +- References to other objects: + - `policy_id` - ID that points to the policy (`ingest-agent-policies`) this agent is assigned to. + - `access_api_key_id` + - `default_api_key_id` + +Tracks an individual Elastic Agent's enrollment in the Fleet, which policy it is current assigned to, its check in +status, which packages are currently installed, and other metadata about the Agent. + +### `fleet-agent-actions` + +**DEPRECATED in favor of `.fleet-agent-actions` index.** + +- Constant in code: `AGENT_ACTION_SAVED_OBJECT_TYPE` +- Introduced in ? +- [Code Link](../server/saved_objects/index.ts#113) +- Migrations: 7.10.0 +- References to other objects: + - `agent_id` - ID that points to the agent for this action (`fleet-agents`) + - `policy_id`- ID that points to the policy for this action (`ingest-agent-policies`) + + +### `fleet-enrollment-api-keys` + +**DEPRECATED in favor of `.fleet-enrollment-api-keys` index.** + +- Constant in code: `ENROLLMENT_API_KEYS_SAVED_OBJECT_TYPE` +- Introduced in ? +- [Code Link](../server/saved_objects/index.ts#166) +- Migrations: 7.10.0 +- References to other objects: + - `api_key_id` + - `policy_id` - ID that points to an agent policy (`ingest-agent-policies`) + +Contains an enrollment key that can be used to enroll a new agent in a specific agent policy. \ No newline at end of file