First stab at developer-focussed saved objects docs (#71430)

* First stab at developer-focussed saved objects docs

* Don't introduce spelling mistakes

* Add docs for SO migrations

* Link to HTTP API documentation

* Grammar fixes

* Rendering fixes

* Migrations should be tested, remove nested migration docs for now

* Drop subtitle field in migration, add notes about migration version, behaviour for corrupt documents and emphasize testing

Co-authored-by: Josh Dover <[email protected]>

docs/developer/architecture/development-plugin-saved-objects.asciidoc

@@ -0,0 +1,311 @@
+[[development-plugin-saved-objects]]
+== Using Saved Objects
+
+Saved Objects allow {kib} plugins to use {es} like a primary
+database. Think of it as an Object Document Mapper for {es}. Once a
+plugin has registered one or more Saved Object types, the Saved Objects client
+can be used to query or perform create, read, update and delete operations on
+each type.
+
+By using Saved Objects your plugin can take advantage of the following
+features:
+
+* Migrations can evolve your document's schema by transforming documents and
+ensuring that the field mappings on the index are always up to date.
+* a <<saved-objects-api,HTTP API>> is automatically exposed for each type (unless
+`hidden=true` is specified).
+* a Saved Objects client that can be used from both the server and the browser.
+* Users can import or export Saved Objects using the Saved Objects management
+UI or the Saved Objects import/export API.
+* By declaring `references`, an object's entire reference graph will be
+exported. This makes it easy for users to export e.g. a `dashboard` object and
+have all the `visualization` objects required to display the dashboard
+included in the export.
+* When the X-Pack security and spaces plugins are enabled these transparently
+provide RBAC access control and the ability to organize Saved Objects into
+spaces.
+
+This document contains developer guidelines and best-practices for plugins
+wanting to use Saved Objects.
+
+=== Registering a Saved Object type
+Saved object type definitions should be defined in their own `my_plugin/server/saved_objects` directory.
+
+The folder should contain a file per type, named after the snake_case name of the type, and an `index.ts` file exporting all the types.
+
+.src/plugins/my_plugin/server/saved_objects/dashboard_visualization.ts
+[source,typescript]
+----
+import { SavedObjectsType } from 'src/core/server';
+
+export const dashboardVisualization: SavedObjectsType = {
+  name: 'dashboard_visualization', // <1>
+  hidden: false,
+  namespaceType: 'single',
+  mappings: {
+    dynamic: false,
+    properties: {
+      description: {
+        type: 'text',
+      },
+      hits: {
+        type: 'integer',
+      },
+    },
+  },
+  migrations: {
+    '1.0.0': migratedashboardVisualizationToV1,
+    '2.0.0': migratedashboardVisualizationToV2,
+  },
+};
+----
+<1> Since the name of a Saved Object type forms part of the url path for the
+public Saved Objects HTTP API, these should follow our API URL path convention
+and always be written as snake case.
+
+.src/plugins/my_plugin/server/saved_objects/index.ts
+[source,typescript]
+----
+export { dashboardVisualization } from './dashboard_visualization';
+export { dashboard } from './dashboard'; 
+----
+
+.src/plugins/my_plugin/server/plugin.ts
+[source,typescript]
+----
+import { dashboard, dashboardVisualization } from './saved_objects';
+
+export class MyPlugin implements Plugin {
+  setup({ savedObjects }) {
+    savedObjects.registerType(dashboard);
+    savedObjects.registerType(dashboardVisualization);
+  }
+}
+----
+
+=== Mappings
+Each Saved Object type can define it's own {es} field mappings.
+Because multiple Saved Object types can share the same index, mappings defined
+by a type will be nested under a top-level field that matches the type name.
+
+For example, the mappings defined by the `dashboard_visualization` Saved
+Object type:
+
+.src/plugins/my_plugin/server/saved_objects/dashboard_visualization.ts
+[source,typescript]
+----
+import { SavedObjectsType } from 'src/core/server';
+
+export const dashboardVisualization: SavedObjectsType = {
+  name: 'dashboard_visualization',
+  ...
+  mappings: {
+    properties: {
+      dynamic: false,
+      description: {
+        type: 'text',
+      },
+      hits: {
+        type: 'integer',
+      },
+    },
+  },
+  migrations: { ... },
+};
+----
+
+Will result in the following mappings being applied to the `.kibana` index:
+[source,json]
+----
+{
+  "mappings": {
+    "dynamic": "strict",
+    "properties": {
+      ...
+      "dashboard_vizualization": {
+        "dynamic": false,
+        "properties": {
+          "description": {
+            "type": "text",
+          },
+          "hits": {
+            "type": "integer",
+          },
+        },
+      }
+    }
+  }
+}
+----
+
+Do not use field mappings like you would use data types for the columns of a
+SQL database. Instead, field mappings are analogous to a SQL index. Only
+specify field mappings for the fields you wish to search on or query. By
+specifying `dynamic: false` in any level of your mappings, {es} will
+accept and store any other fields even if they are not specified in your mappings.
+
+Since {es} has a default limit of 1000 fields per index, plugins
+should carefully consider the fields they add to the mappings. Similarly,
+Saved Object types should never use `dynamic: true` as this can cause an
+arbitrary amount of fields to be added to the `.kibana` index.
+
+=== References
+When a Saved Object declares `references` to other Saved Objects, the
+Saved Objects Export API will automatically export the target object with all
+of it's references. This makes it easy for users to export the entire
+reference graph of an object. 
+
+If a Saved Object can't be used on it's own, that is, it needs other objects
+to exist for a feature to function correctly, that Saved Object should declare
+references to all the objects it requires. For example, a `dashboard`
+object might have panels for several `visualization` objects. When these
+`visualization` objects don't exist, the dashboard cannot be rendered
+correctly. The `dashboard` object should declare references to all it's
+visualizations.
+
+However, `visualization` objects can continue to be rendered or embedded into
+other dashboards even if the `dashboard` it was originally embedded into
+doesn't exist. As a result, `visualization` objects should not declare
+references to `dashboard` objects.
+
+For each referenced object, an `id`, `type` and `name` are added to the
+`references` array:
+
+[source, typescript]
+----
+router.get(
+  { path: '/some-path', validate: false },
+  async (context, req, res) => {
+    const object = await context.core.savedObjects.client.create(
+      'dashboard',
+      {
+        title: 'my dashboard',
+        panels: [
+          { visualization: 'vis1' }, // <1>
+        ],
+        indexPattern: 'indexPattern1'
+      },
+      { references: [
+          { id: '...', type: 'visualization', name: 'vis1' },
+          { id: '...', type: 'index_pattern', name: 'indexPattern1' },
+        ]
+      }
+    )
+    ...
+  }
+);
+----
+<1> Note how `dashboard.panels[0].visualization` stores the `name` property of
+the reference (not the `id` directly) to be able to uniquely identify this
+reference. This guarantees that the id the reference points to always remains
+up to date. If a visualization `id` was directly stored in
+`dashboard.panels[0].visualization` there is a risk that this `id` gets
+updated without updating the reference in the references array.
+
+==== Writing Migrations
+
+Saved Objects support schema changes between Kibana versions, which we call
+migrations. Migrations are applied when a Kibana installation is upgraded from
+one version to the next, when exports are imported via the Saved Objects
+Management UI, or when a new object is created via the HTTP API.
+
+Each Saved Object type may define migrations for its schema. Migrations are
+specified by the Kibana version number, receive an input document, and must
+return the fully migrated document to be persisted to Elasticsearch.
+
+Let's say we want to define two migrations:
+- In version 1.1.0, we want to drop the `subtitle` field and append it to the
+  title
+- In version 1.4.0, we want to add a new `id` field to every panel with a newly
+  generated UUID.
+
+First, the current `mappings` should always reflect the latest or "target"
+schema. Next, we should define a migration function for each step in the schema
+evolution:
+
+src/plugins/my_plugin/server/saved_objects/dashboard_visualization.ts
+[source,typescript]
+----
+import { SavedObjectsType, SavedObjectMigrationFn } from 'src/core/server';
+import uuid from 'uuid';
+
+interface DashboardVisualizationPre110 {
+  title: string;
+  subtitle: string;
+  panels: Array<{}>;
+}
+interface DashboardVisualization110 {
+  title: string;
+  panels: Array<{}>;
+}
+
+interface DashboardVisualization140 {
+  title: string;
+  panels: Array<{ id: string }>;
+}
+
+const migrateDashboardVisualization110: SavedObjectMigrationFn<
+  DashboardVisualizationPre110, // <1>
+  DashboardVisualization110
+> = (doc) => {
+  const { subtitle, ...attributesWithoutSubtitle } = doc.attributes;
+  return {
+    ...doc, // <2>
+    attributes: {
+      ...attributesWithoutSubtitle,
+      title: `${doc.attributes.title} - ${doc.attributes.subtitle}`,
+    },
+  };
+};
+
+const migrateDashboardVisualization140: SavedObjectMigrationFn<
+  DashboardVisualization110,
+  DashboardVisualization140
+> = (doc) => {
+  const outPanels = doc.attributes.panels?.map((panel) => {
+    return { ...panel, id: uuid.v4() };
+  });
+  return {
+    ...doc,
+    attributes: {
+      ...doc.attributes,
+      panels: outPanels,
+    },
+  };
+};
+
+export const dashboardVisualization: SavedObjectsType = {
+  name: 'dashboard_visualization', // <1>
+  /** ... */
+  migrations: {
+    // Takes a pre 1.1.0 doc, and converts it to 1.1.0
+    '1.1.0': migrateDashboardVisualization110,
+
+    // Takes a 1.1.0 doc, and converts it to 1.4.0
+    '1.4.0': migrateDashboardVisualization140,  // <3>
+  },
+};
+----
+<1> It is useful to define an interface for each version of the schema. This
+allows TypeScript to ensure that you are properly handling the input and output
+types correctly as the schema evolves.
+<2> Returning a shallow copy is necessary to avoid type errors when using
+different types for the input and output shape.
+<3> Migrations do not have to be defined for every version. The version number
+of a migration must always be the earliest Kibana version in which this
+migration was released. So if you are creating a migration which will be
+part of the v7.10.0 release, but will also be backported and released as
+v7.9.3, the migration version should be: 7.9.3.
+
+Migrations should be written defensively, an exception in a migration function
+will prevent a Kibana upgrade from succeeding and will cause downtime for our
+users. Having said that, if a document is encountered that is not in the
+expected shape, migrations are encouraged to throw an exception to abort the
+upgrade. In most scenarios, it is better to fail an upgrade than to silently
+ignore a corrupt document which can cause unexpected behaviour at some future
+point in time.
+
+It is critical that you have extensive tests to ensure that migrations behave
+as expected with all possible input documents. Given how simple it is to test
+all the branch conditions in a migration function and the high impact of a bug
+in this code, there's really no reason not to aim for 100% test code coverage.

docs/developer/architecture/index.asciidoc

@@ -15,12 +15,16 @@ A few services also automatically generate api documentation which can be browse
 A few notable services are called out below.
 
 * <<development-security>>
+* <<development-plugin-saved-objects>>
 * <<add-data-tutorials>>
 * <<development-visualize-index>>
 
+include::security/index.asciidoc[leveloffset=+1]
+
+include::development-plugin-saved-objects.asciidoc[leveloffset=+1]
+
 include::add-data-tutorials.asciidoc[leveloffset=+1]
 
 include::development-visualize-index.asciidoc[leveloffset=+1]
 
-include::security/index.asciidoc[leveloffset=+1]