diff --git a/docs/developer/architecture/development-plugin-saved-objects.asciidoc b/docs/developer/architecture/development-plugin-saved-objects.asciidoc new file mode 100644 index 0000000000000..0d31f5d90f668 --- /dev/null +++ b/docs/developer/architecture/development-plugin-saved-objects.asciidoc @@ -0,0 +1,311 @@ +[[development-plugin-saved-objects]] +== Using Saved Objects + +Saved Objects allow {kib} plugins to use {es} like a primary +database. Think of it as an Object Document Mapper for {es}. Once a +plugin has registered one or more Saved Object types, the Saved Objects client +can be used to query or perform create, read, update and delete operations on +each type. + +By using Saved Objects your plugin can take advantage of the following +features: + +* Migrations can evolve your document's schema by transforming documents and +ensuring that the field mappings on the index are always up to date. +* a <> is automatically exposed for each type (unless +`hidden=true` is specified). +* a Saved Objects client that can be used from both the server and the browser. +* Users can import or export Saved Objects using the Saved Objects management +UI or the Saved Objects import/export API. +* By declaring `references`, an object's entire reference graph will be +exported. This makes it easy for users to export e.g. a `dashboard` object and +have all the `visualization` objects required to display the dashboard +included in the export. +* When the X-Pack security and spaces plugins are enabled these transparently +provide RBAC access control and the ability to organize Saved Objects into +spaces. + +This document contains developer guidelines and best-practices for plugins +wanting to use Saved Objects. + +=== Registering a Saved Object type +Saved object type definitions should be defined in their own `my_plugin/server/saved_objects` directory. + +The folder should contain a file per type, named after the snake_case name of the type, and an `index.ts` file exporting all the types. + +.src/plugins/my_plugin/server/saved_objects/dashboard_visualization.ts +[source,typescript] +---- +import { SavedObjectsType } from 'src/core/server'; + +export const dashboardVisualization: SavedObjectsType = { + name: 'dashboard_visualization', // <1> + hidden: false, + namespaceType: 'single', + mappings: { + dynamic: false, + properties: { + description: { + type: 'text', + }, + hits: { + type: 'integer', + }, + }, + }, + migrations: { + '1.0.0': migratedashboardVisualizationToV1, + '2.0.0': migratedashboardVisualizationToV2, + }, +}; +---- +<1> Since the name of a Saved Object type forms part of the url path for the +public Saved Objects HTTP API, these should follow our API URL path convention +and always be written as snake case. + +.src/plugins/my_plugin/server/saved_objects/index.ts +[source,typescript] +---- +export { dashboardVisualization } from './dashboard_visualization'; +export { dashboard } from './dashboard'; +---- + +.src/plugins/my_plugin/server/plugin.ts +[source,typescript] +---- +import { dashboard, dashboardVisualization } from './saved_objects'; + +export class MyPlugin implements Plugin { + setup({ savedObjects }) { + savedObjects.registerType(dashboard); + savedObjects.registerType(dashboardVisualization); + } +} +---- + +=== Mappings +Each Saved Object type can define it's own {es} field mappings. +Because multiple Saved Object types can share the same index, mappings defined +by a type will be nested under a top-level field that matches the type name. + +For example, the mappings defined by the `dashboard_visualization` Saved +Object type: + +.src/plugins/my_plugin/server/saved_objects/dashboard_visualization.ts +[source,typescript] +---- +import { SavedObjectsType } from 'src/core/server'; + +export const dashboardVisualization: SavedObjectsType = { + name: 'dashboard_visualization', + ... + mappings: { + properties: { + dynamic: false, + description: { + type: 'text', + }, + hits: { + type: 'integer', + }, + }, + }, + migrations: { ... }, +}; +---- + +Will result in the following mappings being applied to the `.kibana` index: +[source,json] +---- +{ + "mappings": { + "dynamic": "strict", + "properties": { + ... + "dashboard_vizualization": { + "dynamic": false, + "properties": { + "description": { + "type": "text", + }, + "hits": { + "type": "integer", + }, + }, + } + } + } +} +---- + +Do not use field mappings like you would use data types for the columns of a +SQL database. Instead, field mappings are analogous to a SQL index. Only +specify field mappings for the fields you wish to search on or query. By +specifying `dynamic: false` in any level of your mappings, {es} will +accept and store any other fields even if they are not specified in your mappings. + +Since {es} has a default limit of 1000 fields per index, plugins +should carefully consider the fields they add to the mappings. Similarly, +Saved Object types should never use `dynamic: true` as this can cause an +arbitrary amount of fields to be added to the `.kibana` index. + +=== References +When a Saved Object declares `references` to other Saved Objects, the +Saved Objects Export API will automatically export the target object with all +of it's references. This makes it easy for users to export the entire +reference graph of an object. + +If a Saved Object can't be used on it's own, that is, it needs other objects +to exist for a feature to function correctly, that Saved Object should declare +references to all the objects it requires. For example, a `dashboard` +object might have panels for several `visualization` objects. When these +`visualization` objects don't exist, the dashboard cannot be rendered +correctly. The `dashboard` object should declare references to all it's +visualizations. + +However, `visualization` objects can continue to be rendered or embedded into +other dashboards even if the `dashboard` it was originally embedded into +doesn't exist. As a result, `visualization` objects should not declare +references to `dashboard` objects. + +For each referenced object, an `id`, `type` and `name` are added to the +`references` array: + +[source, typescript] +---- +router.get( + { path: '/some-path', validate: false }, + async (context, req, res) => { + const object = await context.core.savedObjects.client.create( + 'dashboard', + { + title: 'my dashboard', + panels: [ + { visualization: 'vis1' }, // <1> + ], + indexPattern: 'indexPattern1' + }, + { references: [ + { id: '...', type: 'visualization', name: 'vis1' }, + { id: '...', type: 'index_pattern', name: 'indexPattern1' }, + ] + } + ) + ... + } +); +---- +<1> Note how `dashboard.panels[0].visualization` stores the `name` property of +the reference (not the `id` directly) to be able to uniquely identify this +reference. This guarantees that the id the reference points to always remains +up to date. If a visualization `id` was directly stored in +`dashboard.panels[0].visualization` there is a risk that this `id` gets +updated without updating the reference in the references array. + +==== Writing Migrations + +Saved Objects support schema changes between Kibana versions, which we call +migrations. Migrations are applied when a Kibana installation is upgraded from +one version to the next, when exports are imported via the Saved Objects +Management UI, or when a new object is created via the HTTP API. + +Each Saved Object type may define migrations for its schema. Migrations are +specified by the Kibana version number, receive an input document, and must +return the fully migrated document to be persisted to Elasticsearch. + +Let's say we want to define two migrations: +- In version 1.1.0, we want to drop the `subtitle` field and append it to the + title +- In version 1.4.0, we want to add a new `id` field to every panel with a newly + generated UUID. + +First, the current `mappings` should always reflect the latest or "target" +schema. Next, we should define a migration function for each step in the schema +evolution: + +src/plugins/my_plugin/server/saved_objects/dashboard_visualization.ts +[source,typescript] +---- +import { SavedObjectsType, SavedObjectMigrationFn } from 'src/core/server'; +import uuid from 'uuid'; + +interface DashboardVisualizationPre110 { + title: string; + subtitle: string; + panels: Array<{}>; +} +interface DashboardVisualization110 { + title: string; + panels: Array<{}>; +} + +interface DashboardVisualization140 { + title: string; + panels: Array<{ id: string }>; +} + +const migrateDashboardVisualization110: SavedObjectMigrationFn< + DashboardVisualizationPre110, // <1> + DashboardVisualization110 +> = (doc) => { + const { subtitle, ...attributesWithoutSubtitle } = doc.attributes; + return { + ...doc, // <2> + attributes: { + ...attributesWithoutSubtitle, + title: `${doc.attributes.title} - ${doc.attributes.subtitle}`, + }, + }; +}; + +const migrateDashboardVisualization140: SavedObjectMigrationFn< + DashboardVisualization110, + DashboardVisualization140 +> = (doc) => { + const outPanels = doc.attributes.panels?.map((panel) => { + return { ...panel, id: uuid.v4() }; + }); + return { + ...doc, + attributes: { + ...doc.attributes, + panels: outPanels, + }, + }; +}; + +export const dashboardVisualization: SavedObjectsType = { + name: 'dashboard_visualization', // <1> + /** ... */ + migrations: { + // Takes a pre 1.1.0 doc, and converts it to 1.1.0 + '1.1.0': migrateDashboardVisualization110, + + // Takes a 1.1.0 doc, and converts it to 1.4.0 + '1.4.0': migrateDashboardVisualization140, // <3> + }, +}; +---- +<1> It is useful to define an interface for each version of the schema. This +allows TypeScript to ensure that you are properly handling the input and output +types correctly as the schema evolves. +<2> Returning a shallow copy is necessary to avoid type errors when using +different types for the input and output shape. +<3> Migrations do not have to be defined for every version. The version number +of a migration must always be the earliest Kibana version in which this +migration was released. So if you are creating a migration which will be +part of the v7.10.0 release, but will also be backported and released as +v7.9.3, the migration version should be: 7.9.3. + +Migrations should be written defensively, an exception in a migration function +will prevent a Kibana upgrade from succeeding and will cause downtime for our +users. Having said that, if a document is encountered that is not in the +expected shape, migrations are encouraged to throw an exception to abort the +upgrade. In most scenarios, it is better to fail an upgrade than to silently +ignore a corrupt document which can cause unexpected behaviour at some future +point in time. + +It is critical that you have extensive tests to ensure that migrations behave +as expected with all possible input documents. Given how simple it is to test +all the branch conditions in a migration function and the high impact of a bug +in this code, there's really no reason not to aim for 100% test code coverage. \ No newline at end of file diff --git a/docs/developer/architecture/index.asciidoc b/docs/developer/architecture/index.asciidoc index ac25fe003df08..dc15b90b69d1a 100644 --- a/docs/developer/architecture/index.asciidoc +++ b/docs/developer/architecture/index.asciidoc @@ -15,12 +15,16 @@ A few services also automatically generate api documentation which can be browse A few notable services are called out below. * <> +* <> * <> * <> +include::security/index.asciidoc[leveloffset=+1] + +include::development-plugin-saved-objects.asciidoc[leveloffset=+1] + include::add-data-tutorials.asciidoc[leveloffset=+1] include::development-visualize-index.asciidoc[leveloffset=+1] -include::security/index.asciidoc[leveloffset=+1]