Restructure SavedObject types internal representation (#56378) (#56953)

* adapt types and tests to prepare for new NP api

* rename and export public types

* update generated doc

* first implementation of registerMappings

* adapt es archiver to convert legacy mappings

* update generated doc

* fix more tests

* add unit tests

* add legacy-compat unit test

* add documentation and examples

* Introduce SavedObjectTypeRegistry and SavedObjectType types

* add and fix tests

* expose createSerializer API and fix usages

* remove registerMappings API, add internal registerType

* revert changes to migration guide

* adapt ES-archiver migrator creation

* export serializer-related types

* update generated doc

* add and use convertTypesToLegacySchema

* remove / move to internal some mapping types

* fix forEach closure context

* add missing docs

* fix core path

* some nits

* fix so_mixin tests

* fix integration tests

* fix integration tests for real

* add documentation on serializer + restructure files and types

* nit

* add and use the ISavedObjectTypeRegistry interface

* Add documentation, deprecates migrationLogger#warning

* better typing for SavedObjectsRawDoc._source

* nits

* update generated doc

* remove exposition of SavedObjectsTypeMappingDefinitions, update doc

* creates so internal contracts mocks

* improve documentation

docs/development/core/server/kibana-plugin-server.md

@@ -25,6 +25,7 @@ The plugin integrates with the core system via lifecycle events: `setup`<!-- -->
 |  [SavedObjectsClient](./kibana-plugin-server.savedobjectsclient.md) |  |
 |  [SavedObjectsErrorHelpers](./kibana-plugin-server.savedobjectserrorhelpers.md) |  |
 |  [SavedObjectsRepository](./kibana-plugin-server.savedobjectsrepository.md) |  |
+|  [SavedObjectsSerializer](./kibana-plugin-server.savedobjectsserializer.md) | A serializer that can be used to manually convert [raw](./kibana-plugin-server.savedobjectsrawdoc.md) or [sanitized](./kibana-plugin-server.savedobjectsanitizeddoc.md) documents to the other kind. |
 |  [ScopedClusterClient](./kibana-plugin-server.scopedclusterclient.md) | Serves the same purpose as "normal" <code>ClusterClient</code> but exposes additional <code>callAsCurrentUser</code> method that doesn't use credentials of the Kibana internal user (as <code>callAsInternalUser</code> does) to request Elasticsearch API, but rather passes HTTP headers extracted from the current user request to the API.<!-- -->See [ScopedClusterClient](./kibana-plugin-server.scopedclusterclient.md)<!-- -->. |
 
 ## Enumerations
@@ -106,6 +107,7 @@ The plugin integrates with the core system via lifecycle events: `setup`<!-- -->
 |  [RouteValidatorOptions](./kibana-plugin-server.routevalidatoroptions.md) | Additional options for the RouteValidator class to modify its default behaviour. |
 |  [SavedObject](./kibana-plugin-server.savedobject.md) |  |
 |  [SavedObjectAttributes](./kibana-plugin-server.savedobjectattributes.md) | The data for a Saved Object is stored as an object in the <code>attributes</code> property. |
+|  [SavedObjectMigrationMap](./kibana-plugin-server.savedobjectmigrationmap.md) | A map of [migration functions](./kibana-plugin-server.savedobjectmigrationfn.md) to be used for a given type. The map's keys must be valid semver versions.<!-- -->For a given document, only migrations with a higher version number than that of the document will be applied. Migrations are executed in order, starting from the lowest version and ending with the highest one. |
 |  [SavedObjectReference](./kibana-plugin-server.savedobjectreference.md) | A reference to another saved object. |
 |  [SavedObjectsBaseOptions](./kibana-plugin-server.savedobjectsbaseoptions.md) |  |
 |  [SavedObjectsBulkCreateObject](./kibana-plugin-server.savedobjectsbulkcreateobject.md) |  |
@@ -116,6 +118,8 @@ The plugin integrates with the core system via lifecycle events: `setup`<!-- -->
 |  [SavedObjectsBulkUpdateResponse](./kibana-plugin-server.savedobjectsbulkupdateresponse.md) |  |
 |  [SavedObjectsClientProviderOptions](./kibana-plugin-server.savedobjectsclientprovideroptions.md) | Options to control the creation of the Saved Objects Client. |
 |  [SavedObjectsClientWrapperOptions](./kibana-plugin-server.savedobjectsclientwrapperoptions.md) | Options passed to each SavedObjectsClientWrapperFactory to aid in creating the wrapper instance. |
+|  [SavedObjectsComplexFieldMapping](./kibana-plugin-server.savedobjectscomplexfieldmapping.md) | See [SavedObjectsFieldMapping](./kibana-plugin-server.savedobjectsfieldmapping.md) for documentation. |
+|  [SavedObjectsCoreFieldMapping](./kibana-plugin-server.savedobjectscorefieldmapping.md) | See [SavedObjectsFieldMapping](./kibana-plugin-server.savedobjectsfieldmapping.md) for documentation. |
 |  [SavedObjectsCreateOptions](./kibana-plugin-server.savedobjectscreateoptions.md) |  |
 |  [SavedObjectsDeleteByNamespaceOptions](./kibana-plugin-server.savedobjectsdeletebynamespaceoptions.md) |  |
 |  [SavedObjectsDeleteOptions](./kibana-plugin-server.savedobjectsdeleteoptions.md) |  |
@@ -132,13 +136,16 @@ The plugin integrates with the core system via lifecycle events: `setup`<!-- -->
 |  [SavedObjectsImportUnknownError](./kibana-plugin-server.savedobjectsimportunknownerror.md) | Represents a failure to import due to an unknown reason. |
 |  [SavedObjectsImportUnsupportedTypeError](./kibana-plugin-server.savedobjectsimportunsupportedtypeerror.md) | Represents a failure to import due to having an unsupported saved object type. |
 |  [SavedObjectsIncrementCounterOptions](./kibana-plugin-server.savedobjectsincrementcounteroptions.md) |  |
+|  [SavedObjectsMappingProperties](./kibana-plugin-server.savedobjectsmappingproperties.md) | Describe the fields of a [saved object type](./kibana-plugin-server.savedobjectstypemappingdefinition.md)<!-- -->. |
 |  [SavedObjectsMigrationLogger](./kibana-plugin-server.savedobjectsmigrationlogger.md) |  |
 |  [SavedObjectsMigrationVersion](./kibana-plugin-server.savedobjectsmigrationversion.md) | Information about the migrations that have been applied to this SavedObject. When Kibana starts up, KibanaMigrator detects outdated documents and migrates them based on this value. For each migration that has been applied, the plugin's name is used as a key and the latest migration version as the value. |
 |  [SavedObjectsRawDoc](./kibana-plugin-server.savedobjectsrawdoc.md) | A raw document as represented directly in the saved object index. |
 |  [SavedObjectsRepositoryFactory](./kibana-plugin-server.savedobjectsrepositoryfactory.md) | Factory provided when invoking a [client factory provider](./kibana-plugin-server.savedobjectsclientfactoryprovider.md) See [SavedObjectsServiceSetup.setClientFactoryProvider](./kibana-plugin-server.savedobjectsservicesetup.setclientfactoryprovider.md) |
 |  [SavedObjectsResolveImportErrorsOptions](./kibana-plugin-server.savedobjectsresolveimporterrorsoptions.md) | Options to control the "resolve import" operation. |
 |  [SavedObjectsServiceSetup](./kibana-plugin-server.savedobjectsservicesetup.md) | Saved Objects is Kibana's data persistence mechanism allowing plugins to use Elasticsearch for storing and querying state. The SavedObjectsServiceSetup API exposes methods for creating and registering Saved Object client wrappers. |
 |  [SavedObjectsServiceStart](./kibana-plugin-server.savedobjectsservicestart.md) | Saved Objects is Kibana's data persisentence mechanism allowing plugins to use Elasticsearch for storing and querying state. The SavedObjectsServiceStart API provides a scoped Saved Objects client for interacting with Saved Objects. |
+|  [SavedObjectsType](./kibana-plugin-server.savedobjectstype.md) |  |
+|  [SavedObjectsTypeMappingDefinition](./kibana-plugin-server.savedobjectstypemappingdefinition.md) | Describe a saved object type mapping. |
 |  [SavedObjectsUpdateOptions](./kibana-plugin-server.savedobjectsupdateoptions.md) |  |
 |  [SavedObjectsUpdateResponse](./kibana-plugin-server.savedobjectsupdateresponse.md) |  |
 |  [SessionCookieValidationResult](./kibana-plugin-server.sessioncookievalidationresult.md) | Return type from a function to validate cookie contents. |
@@ -218,10 +225,13 @@ The plugin integrates with the core system via lifecycle events: `setup`<!-- -->
 |  [RouteValidatorFullConfig](./kibana-plugin-server.routevalidatorfullconfig.md) | Route validations config and options merged into one object |
 |  [SavedObjectAttribute](./kibana-plugin-server.savedobjectattribute.md) | Type definition for a Saved Object attribute value |
 |  [SavedObjectAttributeSingle](./kibana-plugin-server.savedobjectattributesingle.md) | Don't use this type, it's simply a helper type for [SavedObjectAttribute](./kibana-plugin-server.savedobjectattribute.md) |
+|  [SavedObjectMigrationFn](./kibana-plugin-server.savedobjectmigrationfn.md) | A migration function defined for a [saved objects type](./kibana-plugin-server.savedobjectstype.md) used to migrate it's  |
+|  [SavedObjectSanitizedDoc](./kibana-plugin-server.savedobjectsanitizeddoc.md) |  |
 |  [SavedObjectsClientContract](./kibana-plugin-server.savedobjectsclientcontract.md) | Saved Objects is Kibana's data persisentence mechanism allowing plugins to use Elasticsearch for storing plugin state.<!-- -->\#\# SavedObjectsClient errors<!-- -->Since the SavedObjectsClient has its hands in everything we are a little paranoid about the way we present errors back to to application code. Ideally, all errors will be either:<!-- -->1. Caused by bad implementation (ie. undefined is not a function) and as such unpredictable 2. An error that has been classified and decorated appropriately by the decorators in [SavedObjectsErrorHelpers](./kibana-plugin-server.savedobjectserrorhelpers.md)<!-- -->Type 1 errors are inevitable, but since all expected/handle-able errors should be Type 2 the <code>isXYZError()</code> helpers exposed at <code>SavedObjectsErrorHelpers</code> should be used to understand and manage error responses from the <code>SavedObjectsClient</code>.<!-- -->Type 2 errors are decorated versions of the source error, so if the elasticsearch client threw an error it will be decorated based on its type. That means that rather than looking for <code>error.body.error.type</code> or doing substring checks on <code>error.body.error.reason</code>, just use the helpers to understand the meaning of the error:<!-- -->\`\`\`<!-- -->js if (SavedObjectsErrorHelpers.isNotFoundError(error)) { // handle 404 }<!-- -->if (SavedObjectsErrorHelpers.isNotAuthorizedError(error)) { // 401 handling should be automatic, but in case you wanted to know }<!-- -->// always rethrow the error unless you handle it throw error; \`\`\`<!-- -->\#\#\# 404s from missing index<!-- -->From the perspective of application code and APIs the SavedObjectsClient is a black box that persists objects. One of the internal details that users have no control over is that we use an elasticsearch index for persistance and that index might be missing.<!-- -->At the time of writing we are in the process of transitioning away from the operating assumption that the SavedObjects index is always available. Part of this transition is handling errors resulting from an index missing. These used to trigger a 500 error in most cases, and in others cause 404s with different error messages.<!-- -->From my (Spencer) perspective, a 404 from the SavedObjectsApi is a 404; The object the request/call was targeting could not be found. This is why \#14141 takes special care to ensure that 404 errors are generic and don't distinguish between index missing or document missing.<!-- -->\#\#\# 503s from missing index<!-- -->Unlike all other methods, create requests are supposed to succeed even when the Kibana index does not exist because it will be automatically created by elasticsearch. When that is not the case it is because Elasticsearch's <code>action.auto_create_index</code> setting prevents it from being created automatically so we throw a special 503 with the intention of informing the user that their Elasticsearch settings need to be updated.<!-- -->See [SavedObjectsClient](./kibana-plugin-server.savedobjectsclient.md) See [SavedObjectsErrorHelpers](./kibana-plugin-server.savedobjectserrorhelpers.md) |
 |  [SavedObjectsClientFactory](./kibana-plugin-server.savedobjectsclientfactory.md) | Describes the factory used to create instances of the Saved Objects Client. |
 |  [SavedObjectsClientFactoryProvider](./kibana-plugin-server.savedobjectsclientfactoryprovider.md) | Provider to invoke to retrieve a [SavedObjectsClientFactory](./kibana-plugin-server.savedobjectsclientfactory.md)<!-- -->. |
 |  [SavedObjectsClientWrapperFactory](./kibana-plugin-server.savedobjectsclientwrapperfactory.md) | Describes the factory used to create instances of Saved Objects Client Wrappers. |
+|  [SavedObjectsFieldMapping](./kibana-plugin-server.savedobjectsfieldmapping.md) | Describe a [saved object type mapping](./kibana-plugin-server.savedobjectstypemappingdefinition.md) field.<!-- -->Please refer to [elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html) For the mapping documentation |
 |  [ScopeableRequest](./kibana-plugin-server.scopeablerequest.md) | A user credentials container. It accommodates the necessary auth credentials to impersonate the current user.<!-- -->See [KibanaRequest](./kibana-plugin-server.kibanarequest.md)<!-- -->. |
 |  [SharedGlobalConfig](./kibana-plugin-server.sharedglobalconfig.md) |  |
 |  [StringValidation](./kibana-plugin-server.stringvalidation.md) | Allows regex objects or a regex string |

docs/development/core/server/kibana-plugin-server.savedobjectmigrationfn.md

@@ -0,0 +1,13 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectMigrationFn](./kibana-plugin-server.savedobjectmigrationfn.md)
+
+## SavedObjectMigrationFn type
+
+A migration function defined for a [saved objects type](./kibana-plugin-server.savedobjectstype.md) used to migrate it's 
+
+<b>Signature:</b>
+
+```typescript
+export declare type SavedObjectMigrationFn = (doc: SavedObjectUnsanitizedDoc, log: SavedObjectsMigrationLogger) => SavedObjectUnsanitizedDoc;
+```

docs/development/core/server/kibana-plugin-server.savedobjectmigrationmap.md

@@ -0,0 +1,27 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectMigrationMap](./kibana-plugin-server.savedobjectmigrationmap.md)
+
+## SavedObjectMigrationMap interface
+
+A map of [migration functions](./kibana-plugin-server.savedobjectmigrationfn.md) to be used for a given type. The map's keys must be valid semver versions.
+
+For a given document, only migrations with a higher version number than that of the document will be applied. Migrations are executed in order, starting from the lowest version and ending with the highest one.
+
+<b>Signature:</b>
+
+```typescript
+export interface SavedObjectMigrationMap 
+```
+
+## Example
+
+
+```typescript
+const migrations: SavedObjectMigrationMap = {
+  '1.0.0': migrateToV1,
+  '2.1.0': migrateToV21
+}
+
+```
+

docs/development/core/server/kibana-plugin-server.savedobjectsanitizeddoc.md

@@ -0,0 +1,12 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectSanitizedDoc](./kibana-plugin-server.savedobjectsanitizeddoc.md)
+
+## SavedObjectSanitizedDoc type
+
+
+<b>Signature:</b>
+
+```typescript
+export declare type SavedObjectSanitizedDoc = SavedObjectDoc & Referencable;
+```

docs/development/core/server/kibana-plugin-server.savedobjectscomplexfieldmapping.dynamic.md

@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsComplexFieldMapping](./kibana-plugin-server.savedobjectscomplexfieldmapping.md) &gt; [dynamic](./kibana-plugin-server.savedobjectscomplexfieldmapping.dynamic.md)
+
+## SavedObjectsComplexFieldMapping.dynamic property
+
+<b>Signature:</b>
+
+```typescript
+dynamic?: string;
+```

docs/development/core/server/kibana-plugin-server.savedobjectscomplexfieldmapping.md

@@ -0,0 +1,22 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsComplexFieldMapping](./kibana-plugin-server.savedobjectscomplexfieldmapping.md)
+
+## SavedObjectsComplexFieldMapping interface
+
+See [SavedObjectsFieldMapping](./kibana-plugin-server.savedobjectsfieldmapping.md) for documentation.
+
+<b>Signature:</b>
+
+```typescript
+export interface SavedObjectsComplexFieldMapping 
+```
+
+## Properties
+
+|  Property | Type | Description |
+|  --- | --- | --- |
+|  [dynamic](./kibana-plugin-server.savedobjectscomplexfieldmapping.dynamic.md) | <code>string</code> |  |
+|  [properties](./kibana-plugin-server.savedobjectscomplexfieldmapping.properties.md) | <code>SavedObjectsMappingProperties</code> |  |
+|  [type](./kibana-plugin-server.savedobjectscomplexfieldmapping.type.md) | <code>string</code> |  |
+

docs/development/core/server/kibana-plugin-server.savedobjectscomplexfieldmapping.properties.md

@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsComplexFieldMapping](./kibana-plugin-server.savedobjectscomplexfieldmapping.md) &gt; [properties](./kibana-plugin-server.savedobjectscomplexfieldmapping.properties.md)
+
+## SavedObjectsComplexFieldMapping.properties property
+
+<b>Signature:</b>
+
+```typescript
+properties: SavedObjectsMappingProperties;
+```

docs/development/core/server/kibana-plugin-server.savedobjectscomplexfieldmapping.type.md

@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsComplexFieldMapping](./kibana-plugin-server.savedobjectscomplexfieldmapping.md) &gt; [type](./kibana-plugin-server.savedobjectscomplexfieldmapping.type.md)
+
+## SavedObjectsComplexFieldMapping.type property
+
+<b>Signature:</b>
+
+```typescript
+type?: string;
+```

docs/development/core/server/kibana-plugin-server.savedobjectscorefieldmapping.enabled.md

@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsCoreFieldMapping](./kibana-plugin-server.savedobjectscorefieldmapping.md) &gt; [enabled](./kibana-plugin-server.savedobjectscorefieldmapping.enabled.md)
+
+## SavedObjectsCoreFieldMapping.enabled property
+
+<b>Signature:</b>
+
+```typescript
+enabled?: boolean;
+```

docs/development/core/server/kibana-plugin-server.savedobjectscorefieldmapping.fields.md

@@ -0,0 +1,15 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsCoreFieldMapping](./kibana-plugin-server.savedobjectscorefieldmapping.md) &gt; [fields](./kibana-plugin-server.savedobjectscorefieldmapping.fields.md)
+
+## SavedObjectsCoreFieldMapping.fields property
+
+<b>Signature:</b>
+
+```typescript
+fields?: {
+        [subfield: string]: {
+            type: string;
+        };
+    };
+```

docs/development/core/server/kibana-plugin-server.savedobjectscorefieldmapping.index.md

@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsCoreFieldMapping](./kibana-plugin-server.savedobjectscorefieldmapping.md) &gt; [index](./kibana-plugin-server.savedobjectscorefieldmapping.index.md)
+
+## SavedObjectsCoreFieldMapping.index property
+
+<b>Signature:</b>
+
+```typescript
+index?: boolean;
+```

docs/development/core/server/kibana-plugin-server.savedobjectscorefieldmapping.md

@@ -0,0 +1,23 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsCoreFieldMapping](./kibana-plugin-server.savedobjectscorefieldmapping.md)
+
+## SavedObjectsCoreFieldMapping interface
+
+See [SavedObjectsFieldMapping](./kibana-plugin-server.savedobjectsfieldmapping.md) for documentation.
+
+<b>Signature:</b>
+
+```typescript
+export interface SavedObjectsCoreFieldMapping 
+```
+
+## Properties
+
+|  Property | Type | Description |
+|  --- | --- | --- |
+|  [enabled](./kibana-plugin-server.savedobjectscorefieldmapping.enabled.md) | <code>boolean</code> |  |
+|  [fields](./kibana-plugin-server.savedobjectscorefieldmapping.fields.md) | <code>{</code><br/><code>        [subfield: string]: {</code><br/><code>            type: string;</code><br/><code>        };</code><br/><code>    }</code> |  |
+|  [index](./kibana-plugin-server.savedobjectscorefieldmapping.index.md) | <code>boolean</code> |  |
+|  [type](./kibana-plugin-server.savedobjectscorefieldmapping.type.md) | <code>string</code> |  |
+

docs/development/core/server/kibana-plugin-server.savedobjectscorefieldmapping.type.md

@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsCoreFieldMapping](./kibana-plugin-server.savedobjectscorefieldmapping.md) &gt; [type](./kibana-plugin-server.savedobjectscorefieldmapping.type.md)
+
+## SavedObjectsCoreFieldMapping.type property
+
+<b>Signature:</b>
+
+```typescript
+type: string;
+```

docs/development/core/server/kibana-plugin-server.savedobjectsfieldmapping.md

@@ -0,0 +1,15 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsFieldMapping](./kibana-plugin-server.savedobjectsfieldmapping.md)
+
+## SavedObjectsFieldMapping type
+
+Describe a [saved object type mapping](./kibana-plugin-server.savedobjectstypemappingdefinition.md) field.
+
+Please refer to [elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html) For the mapping documentation
+
+<b>Signature:</b>
+
+```typescript
+export declare type SavedObjectsFieldMapping = SavedObjectsCoreFieldMapping | SavedObjectsComplexFieldMapping;
+```