Merge branch 'master' into feat/disableable-plugins

.buildkite/pipelines/hourly.yml

@@ -1,3 +1,5 @@
+env:
+  REPORT_FAILED_TESTS_TO_GITHUB: 'true'
 steps:
   - command: .buildkite/scripts/lifecycle/pre_build.sh
     label: Pre-Build

dev_docs/key_concepts/persistable_state.mdx

@@ -0,0 +1,83 @@
+---
+id: kibDevDocsPersistableStateIntro
+slug: /kibana-dev-docs/persistable-state-intro
+title: Persistable State
+summary: Persitable state is a key concept to understand when building a Kibana plugin.
+date: 2021-02-02
+tags: ['kibana','dev', 'contributor', 'api docs']
+---
+
+ “Persistable state” is developer-defined state that supports being persisted by a plugin other than the one defining it. Persistable State needs to be serializable and the owner can/should provide utilities to migrate it, extract and inject any <DocLink id="kibDevDocsSavedObjectsIntro" section="references" text="references to Saved Objects"/> it may contain, as well as telemetry collection utilities.
+
+## Exposing state that can be persisted
+
+Any plugin that exposes state that another plugin might persist should implement <DocLink id="kibKibanaUtilsPluginApi " section="def-common.PersistableStateService" text="`PersistableStateService`"/> interface on their `setup` contract. This will allow plugins persisting the state to easily access migrations and other utilities.
+
+Example: Data plugin allows you to generate filters. Those filters can be persisted by applications in their saved 
+objects or in the URL. In order to allow apps to migrate the filters in case the structure changes in the future, the Data plugin implements `PersistableStateService` on <DocLink id="kibDataQueryPluginApi " section="def-public.FilterManager" text="`data.query.filterManager`"/>.
+
+note: There is currently no obvious way for a plugin to know which state is safe to persist. The developer must manually look for a matching `PersistableStateService`, or ad-hoc provided migration utilities (as is the case with Rule Type Parameters).
+In the future, we hope to make it easier for producers of state to understand when they need to write a migration with changes, and also make it easier for consumers of such state, to understand whether it is safe to persist.
+
+## Exposing state that can be persisted but is not owned by plugin exposing it (registry)
+
+Any plugin that owns collection of items (registry) whose state/configuration can be persisted should implement `PersistableStateService`
+interface on their `setup` contract and each item in the collection should implement <DocLink id="kibKibanaUtilsPluginApi" section="def-common.PersistableStateDefinition" text="`PersistableStateDefinition`"/> interface.
+
+Example: Embeddable plugin owns the registry of embeddable factories to which other plugins can register new embeddable factories. Dashboard plugin
+stores a bunch of embeddable panels input in its saved object and URL. Embeddable plugin setup contract implements `PersistableStateService`
+interface and each `EmbeddableFactory`  needs to implement `PersistableStateDefinition` interface.
+
+Embeddable plugin exposes this interfaces:
+```
+// EmbeddableInput implements Serializable
+
+export interface EmbeddableRegistryDefinition extends PersistableStateDefinition<EmbeddableInput> {
+  id: string;
+  ...
+}
+
+export interface EmbeddableSetup extends PersistableStateService<EmbeddableInput>;
+```
+
+Note: if your plugin doesn't expose the state (it is the only one storing state), the plugin doesn't need to implement the `PersistableStateService` interface.
+If the state your plugin is storing can be provided by other plugins (your plugin owns a registry) items in that registry still need to implement `PersistableStateDefinition` interface.
+
+## Storing persistable state as part of saved object
+
+Any plugin that stores any persistable state as part of their saved object should make sure that its saved object migration 
+and reference extraction and injection methods correctly use the matching `PersistableStateService` implementation for the state they are storing.
+
+Take a look at [example saved object](https://github.com/elastic/kibana/blob/master/examples/embeddable_examples/server/searchable_list_saved_object.ts#L32) which stores an embeddable state. Note how the `migrations`, `extractReferences` and `injectReferences` are defined.
+
+## Storing persistable state as part of URL
+
+When storing persistable state as part of URL you must make sure your URL is versioned. When loading the state `migrateToLatest` method
+of `PersistableStateService` should be called, which will migrate the state from its original version to latest.
+
+note: Currently there is no recommended way on how to store version in url and its up to every application to decide on how to implement that.
+
+## Available state operations
+
+### Extraction/Injection of References
+
+In order to support import and export, and space-sharing capabilities, Saved Objects need to explicitly list any references they contain to other Saved Objects.
+To support persisting your state in saved objects owned by another plugin, the <DocLink id="kibKibanaUtilsPluginApi" section="def-common.PersistableState.extract" text="`extract`"/> and <DocLink id="kibKibanaUtilsPluginApi" section="def-common.PersistableState.inject" text="`inject`"/> methods of Persistable State interface should be implemented.
+
+<DocLink id="kibDevTutorialSavedObject" section="references" text="Learn how to define Saved Object references"/>
+
+[See example embeddable providing extract/inject functions](https://github.com/elastic/kibana/blob/master/examples/embeddable_examples/public/migrations/migrations_embeddable_factory.ts)
+
+### Migrations and Backward compatibility
+
+As your plugin evolves, you may need to change your state in a breaking way. If that happens, you should write a migration to upgrade the state that existed prior to the change.
+
+<DocLink id="kibDevTutorialSavedObject" section="migrations" text="How to write a migration"/>.
+
+[See an example saved object storing embeddable state implementing saved object migration function](https://github.com/elastic/kibana/blob/master/examples/embeddable_examples/server/searchable_list_saved_object.ts)
+
+[See example embeddable providing migration functions](https://github.com/elastic/kibana/blob/master/examples/embeddable_examples/public/migrations/migrations_embeddable_factory.ts)
+
+## Telemetry
+
+You might want to collect statistics about how your state is used. If that is the case you should implement the telemetry method of Persistable State interface. 

examples/embeddable_examples/kibana.json

@@ -9,7 +9,7 @@
     "githubTeam": "kibana-app-services"
   },
   "description": "Example app that shows how to register custom embeddables",
-  "requiredPlugins": ["embeddable", "uiActions", "savedObjects", "dashboard"],
+  "requiredPlugins": ["embeddable", "uiActions", "savedObjects", "dashboard", "kibanaUtils"],
   "optionalPlugins": [],
   "extraPublicDirs": ["public/todo", "public/hello_world", "public/todo/todo_ref_embeddable"],
   "requiredBundles": ["kibanaReact"]

examples/embeddable_examples/public/migrations/migrations_embeddable_factory.ts

@@ -7,6 +7,7 @@
  */
 
 import { i18n } from '@kbn/i18n';
+import { EmbeddableStateWithType } from '../../../../src/plugins/embeddable/common';
 import {
   IContainer,
   EmbeddableInput,
@@ -35,6 +36,16 @@ export class SimpleEmbeddableFactoryDefinition
     '7.3.0': migration730,
   };
 
+  public extract(state: EmbeddableStateWithType) {
+    // this embeddable does not store references to other saved objects
+    return { state, references: [] };
+  }
+
+  public inject(state: EmbeddableStateWithType) {
+    // this embeddable does not store references to other saved objects
+    return state;
+  }
+
   /**
    * In our simple example, we let everyone have permissions to edit this. Most
    * embeddables should check the UI Capabilities service to be sure of

examples/embeddable_examples/server/merge_migration_function_maps.ts

@@ -0,0 +1,25 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { mergeWith } from 'lodash';
+import type { SerializableRecord } from '@kbn/utility-types';
+import { MigrateFunctionsObject, MigrateFunction } from '../../../src/plugins/kibana_utils/common';
+
+export const mergeMigrationFunctionMaps = (
+  obj1: MigrateFunctionsObject,
+  obj2: MigrateFunctionsObject
+) => {
+  const customizer = (objValue: MigrateFunction, srcValue: MigrateFunction) => {
+    if (!srcValue || !objValue) {
+      return srcValue || objValue;
+    }
+    return (state: SerializableRecord) => objValue(srcValue(state));
+  };
+
+  return mergeWith({ ...obj1 }, obj2, customizer);
+};

examples/embeddable_examples/server/searchable_list_saved_object.ts

@@ -9,9 +9,12 @@
 import { mapValues } from 'lodash';
 import { SavedObjectsType, SavedObjectUnsanitizedDoc } from 'kibana/server';
 import { EmbeddableSetup } from '../../../src/plugins/embeddable/server';
+// NOTE: this should rather be imported from 'plugins/kibana_utils/server' but examples at the moment don't
+// allow static imports from plugins so this code was duplicated
+import { mergeMigrationFunctionMaps } from './merge_migration_function_maps';
 
 export const searchableListSavedObject = (embeddable: EmbeddableSetup) => {
-  return {
+  const searchableListSO: SavedObjectsType = {
     name: 'searchableList',
     hidden: false,
     namespaceType: 'single',
@@ -30,14 +33,22 @@ export const searchableListSavedObject = (embeddable: EmbeddableSetup) => {
       },
     },
     migrations: () => {
-      // we assume all the migration will be done by embeddables service and that saved object holds no extra state besides that of searchable list embeddable input\
-      // if saved object would hold additional information we would need to merge the response from embeddables.getAllMigrations with our custom migrations.
-      return mapValues(embeddable.getAllMigrations(), (migrate) => {
+      // there are no migrations defined for the saved object at the moment, possibly they would be added in the future
+      const searchableListSavedObjectMigrations = {};
+
+      // we don't know if embeddables have any migrations defined so we need to fetch them and map the received functions so we pass
+      // them the correct input and that we correctly map the response
+      const embeddableMigrations = mapValues(embeddable.getAllMigrations(), (migrate) => {
         return (state: SavedObjectUnsanitizedDoc) => ({
           ...state,
           attributes: migrate(state.attributes),
         });
       });
+
+      // we merge our and embeddable migrations and return
+      return mergeMigrationFunctionMaps(searchableListSavedObjectMigrations, embeddableMigrations);
     },
-  } as SavedObjectsType;
+  };
+
+  return searchableListSO;
 };

packages/kbn-test/src/failed_tests_reporter/run_failed_tests_reporter_cli.ts

@@ -37,30 +37,31 @@ export function runFailedTestsReporterCli() {
       }
 
       if (updateGithub) {
-        let branch: string | undefined = '';
+        let branch: string = '';
         let isPr = false;
 
         if (process.env.BUILDKITE === 'true') {
-          branch = process.env.BUILDKITE_BRANCH;
+          branch = process.env.BUILDKITE_BRANCH || '';
           isPr = process.env.BUILDKITE_PULL_REQUEST === 'true';
+          updateGithub = process.env.REPORT_FAILED_TESTS_TO_GITHUB === 'true';
         } else {
           // JOB_NAME is formatted as `elastic+kibana+7.x` in some places and `elastic+kibana+7.x/JOB=kibana-intake,node=immutable` in others
           const jobNameSplit = (process.env.JOB_NAME || '').split(/\+|\//);
-          branch = jobNameSplit.length >= 3 ? jobNameSplit[2] : process.env.GIT_BRANCH;
+          branch = jobNameSplit.length >= 3 ? jobNameSplit[2] : process.env.GIT_BRANCH || '';
           isPr = !!process.env.ghprbPullId;
+
+          const isMasterOrVersion = branch === 'master' || branch.match(/^\d+\.(x|\d+)$/);
+          if (!isMasterOrVersion || isPr) {
+            log.info('Failure issues only created on master/version branch jobs');
+            updateGithub = false;
+          }
         }
 
         if (!branch) {
           throw createFailError(
             'Unable to determine originating branch from job name or other environment variables'
           );
         }
-
-        const isMasterOrVersion = branch === 'master' || branch.match(/^\d+\.(x|\d+)$/);
-        if (!isMasterOrVersion || isPr) {
-          log.info('Failure issues only created on master/version branch jobs');
-          updateGithub = false;
-        }
       }
 
       const githubApi = new GithubApi({

src/core/public/doc_links/doc_links_service.ts

@@ -241,6 +241,7 @@ export class DocLinksService {
           kibanaSearchSettings: `${KIBANA_DOCS}advanced-options.html#kibana-search-settings`,
           visualizationSettings: `${KIBANA_DOCS}advanced-options.html#kibana-visualization-settings`,
           timelionSettings: `${KIBANA_DOCS}advanced-options.html#kibana-timelion-settings`,
+          savedObjectsApiList: `${KIBANA_DOCS}saved-objects-api.html#saved-objects-api`,
         },
         ml: {
           guide: `${ELASTIC_WEBSITE_URL}guide/en/machine-learning/${DOC_LINK_VERSION}/index.html`,

src/core/server/http/router/socket.test.ts

@@ -92,7 +92,7 @@ describe('KibanaSocket', () => {
       });
       const socket = new KibanaSocket(tlsSocket);
 
-      expect(socket.renegotiate({})).resolves.toBe(result);
+      await expect(socket.renegotiate({})).rejects.toBe(result);
       expect(spy).toBeCalledTimes(1);
     });
 

src/core/server/saved_objects/migrationsv2/README.md

@@ -36,7 +36,7 @@
   - [REINDEX_SOURCE_TO_TEMP_READ](#reindex_source_to_temp_read)
     - [Next action](#next-action-11)
     - [New control state](#new-control-state-11)
-  - [REINDEX_SOURCE_TO_TEMP_INDEX](#reindex_source_to_temp_index)
+  - [REINDEX_SOURCE_TO_TEMP_TRANSFORM](#REINDEX_SOURCE_TO_TEMP_TRANSFORM)
     - [Next action](#next-action-12)
     - [New control state](#new-control-state-12)
   - [REINDEX_SOURCE_TO_TEMP_INDEX_BULK](#reindex_source_to_temp_index_bulk)
@@ -284,11 +284,11 @@ Read the next batch of outdated documents from the source index by using search
 
 ### New control state
 1. If the batch contained > 0 documents
-  → `REINDEX_SOURCE_TO_TEMP_INDEX`
+  → `REINDEX_SOURCE_TO_TEMP_TRANSFORM`
 2. If there are no more documents returned
   → `REINDEX_SOURCE_TO_TEMP_CLOSE_PIT`
 
-## REINDEX_SOURCE_TO_TEMP_INDEX
+## REINDEX_SOURCE_TO_TEMP_TRANSFORM
 ### Next action
 `transformRawDocs`
 
@@ -357,7 +357,7 @@ documents.
 If another instance has a disabled plugin it will reindex that plugin's
 documents without transforming them. Because this instance doesn't know which
 plugins were disabled by the instance that performed the
-`REINDEX_SOURCE_TO_TEMP_INDEX` step, we need to search for outdated documents
+`REINDEX_SOURCE_TO_TEMP_TRANSFORM` step, we need to search for outdated documents
 and transform them to ensure that everything is up to date.
 
 ### New control state

src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.ts

@@ -13,7 +13,7 @@ import type { ElasticsearchClient } from '../../elasticsearch';
 import { getErrorMessage, getRequestDebugMeta } from '../../elasticsearch';
 import { Model, Next, stateActionMachine } from './state_action_machine';
 import { cleanup } from './migrations_state_machine_cleanup';
-import { ReindexSourceToTempIndex, ReindexSourceToTempIndexBulk, State } from './types';
+import { ReindexSourceToTempTransform, ReindexSourceToTempIndexBulk, State } from './types';
 import { SavedObjectsRawDoc } from '../serialization';
 
 interface StateTransitionLogMeta extends LogMeta {
@@ -115,7 +115,9 @@ export async function migrationStateActionMachine({
         const redactedNewState = {
           ...newState,
           ...{
-            outdatedDocuments: ((newState as ReindexSourceToTempIndex).outdatedDocuments ?? []).map(
+            outdatedDocuments: (
+              (newState as ReindexSourceToTempTransform).outdatedDocuments ?? []
+            ).map(
               (doc) =>
                 ({
                   _id: doc._id,