Adapt the ZDT migration algorithm to support v2 migrations #155981

pgayvallet · 2023-04-27T09:24:36Z

Summary

Adapt the zdt migration algorithm to run the v2 migrations in addition to the model version transformations.

The intent is to be able to use the current migration system in a zero-downtime upgrade friendly-ish way

Technicals

#148656 was a precondition, as the zdt algo requires that mapping changes are compatible (given we keep the same index).

Technically, it means we're now storing the virtualVersion instead of the modelVersion in the index's meta (mappingVersions and docVersions), to allow keeping track of mixed stack and model versions per type.

Note that switching to model versioning is still a one way, non-revertible action. Once using model versioning (by specifying switchToModelVersionAt on your type), there is no going back.

…pport-v2-compat

pgayvallet

Self-review

pgayvallet · 2023-05-01T10:37:33Z

packages/core/saved-objects/core-saved-objects-base-server-internal/src/mappings/types.ts

+/** @internal */
+export type IndexMappingMeta = V2AlgoIndexMappingMeta & ZdtAlgoIndexMappingMeta;


I got tired of having to check which properties are used by each algo everytime I read the type, so it's now split (still using an union though, otherwise we would need to manually specify which one we're using everytime)

pgayvallet · 2023-05-01T11:44:10Z

...jects/core-saved-objects-migration-server-internal/src/zdt/utils/outdated_documents_query.ts

+        const virtualVersion = virtualVersions[type.name];
        return {
          bool: {
-            must: [
-              { term: { type: type.name } },
-              {
-                bool: {
-                  should: [
-                    {
-                      bool: {
-                        must: { exists: { field: 'migrationVersion' } },
-                        must_not: { term: { [`migrationVersion.${type.name}`]: virtualVersion } },
-                      },
-                    },
-                    {
-                      bool: {
-                        must_not: [
-                          { exists: { field: 'migrationVersion' } },
-                          { term: { typeMigrationVersion: virtualVersion } },
-                        ],
-                      },
-                    },
-                  ],
-                },
-              },
-            ],
+            must: [{ term: { type: type.name } }],
+            must_not: [{ term: { typeMigrationVersion: virtualVersion } }],


The query was copied from the v2 algo, but we effectively don't need to support the old migrationVersion.{type} syntax given that zdt will be introduced this change, so I cleaned that up.

pgayvallet · 2023-05-01T12:05:47Z

...e/server/integration_tests/saved_objects/migrations/zdt_v2_compat/jest.integration.config.js

+  roots: ['<rootDir>/src/core/server/integration_tests/saved_objects/migrations/zdt_v2_compat'],
+  // must override to match all test given there is no `integration_tests` subfolder


I added a dedicated suite for the v2 compat tests. The reasoning was that we may eventually want to get rid of the support later, so having dedicated test suites would make the removal easier.

I've been thinking the same thing: Isolate our newest changes in new CI groups.

elasticmachine · 2023-05-01T12:06:49Z

Pinging @elastic/kibana-core (Team:Core)

jloleysens

Great work @pgayvallet , a lot of comments stem from missing some parts of the context in which these changes were made, but I think I could piece most of them together. FWIW, perhaps some of my questions can translate into doc comments that could help other devs grok how the different parts interact.

jloleysens · 2023-05-02T09:33:19Z

.../saved-objects/core-saved-objects-base-server-internal/src/model_version/version_map.test.ts

+            migrations: {
+              '7.17.2': dummyMigration,
+              '8.6.0': dummyMigration,
+            },
+            modelVersions: {
+              1: dummyModelVersion(),


Just want to make sure I understand this correctly, but what is the plan moving forward for these two sets of values? Will consumers specify when they are switching to model versions and then we will assume all "virtual" version migrations should be run first after which the "model versions" take over?

Will consumers specify when they are switching to model versions

Correct, via switchToModelVersionAt

kibana/packages/core/saved-objects/core-saved-objects-server/src/saved_objects_type.ts

Line 230 in 7136039

switchToModelVersionAt?: string;

jloleysens · 2023-05-02T09:34:28Z

.../core/saved-objects/core-saved-objects-base-server-internal/src/model_version/version_map.ts

+
+/**
+ * Returns the current virtual version for the given type.
+ * If will either be the latest model version if the type


Suggested change

* If will either be the latest model version if the type

* It will either be the latest model version if the type

jloleysens · 2023-05-02T09:35:35Z

.../core/saved-objects/core-saved-objects-base-server-internal/src/model_version/version_map.ts


 export type ModelVersionMap = Record<string, number>;
+export type VirtualVersion = string;


Could we add a doc comment to this type explaining what VirtualVersion is, perhaps in relation to ModelVersion?

jloleysens · 2023-05-02T09:37:25Z

...jects/core-saved-objects-migration-server-internal/src/zdt/actions/update_index_meta.test.ts

    const meta: IndexMappingMeta = {
      mappingVersions: {
-        foo: 1,
-        bar: 1,
+        foo: '10.1.0',
+        bar: '10.1.0',
      },
    };


Exciting (and a little scary) times! Will we, at some point in time, potentially have mixed state of VirtualVersions and "regular" stack versions in this map?

Feel like the "missing link" for my understanding is the intended co-existence of these values in the future and what form that will take specifically.

OK I think I got the answer to my own q when reading this test:

https://github.com/elastic/kibana/blob/8a2e3e147fd2fdf2373932b3127c4e3fa5c2606b/packages/core/saved-objects/core-saved-objects-migration-server-internal/src/zdt/utils/outdated_documents_query.test.ts

VirtualVersion is a "special" major combined with model version as minor - fully decoupled from stack (or so it seems).

Yeah, virtual versions is a compatibility format between stack versions and model versions, making the assumption that we will no longer be using stack versioning before this arbitrary 10 version, that serves as a delimiter / identifier for model versions (10.{modelVersion}.0)

…pport-v2-compat

kibana-ci · 2023-05-02T11:38:56Z

💚 Build Succeeded

Buildkite Build
Commit: 36cb6c6

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/core-saved-objects-base-server-internal`	52	55	+3

Unknown metric groups

API count

id	before	after	diff
`@kbn/core-saved-objects-base-server-internal`	72	79	+7

ESLint disabled line counts

id	before	after	diff
`enterpriseSearch`	19	21	+2
`securitySolution`	399	402	+3
total			+5

Total ESLint disabled count

id	before	after	diff
`enterpriseSearch`	20	22	+2
`securitySolution`	479	482	+3
total			+5

History

💛 Build #124480 was flaky 8a2e3e1
💛 Build #124473 was flaky 5441475
💔 Build #123930 failed 9163bd4
💔 Build #123825 failed f3f37ed
💔 Build #123805 failed 1bca7a3

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…55981) ## Summary Adapt the `zdt` migration algorithm to run the v2 migrations in addition to the model version transformations. The intent is to be able to use the current migration system in a zero-downtime upgrade friendly-ish way ### Technicals elastic#148656 was a precondition, as the zdt algo requires that mapping changes are compatible (given we keep the same index). Technically, it means we're now storing the `virtualVersion` instead of the `modelVersion` in the index's meta (`mappingVersions` and `docVersions`), to allow keeping track of mixed stack and model versions per type. Note that switching to model versioning is still a one way, non-revertible action. Once using model versioning (by specifying `switchToModelVersionAt` on your type), there is no going back.

pgayvallet added 3 commits April 27, 2023 11:23

start adapting the version utils

1bca7a3

fix/adapt zdt algo utils

f3f37ed

adapt more usages and tests

9163bd4

pgayvallet added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc release_note:skip Skip the PR/issue when compiling release notes Feature:Migrations v8.9.0 labels Apr 27, 2023

pgayvallet added 4 commits May 1, 2023 09:12

adapt integration tests

c91ce1e

Merge remote-tracking branch 'upstream/main' into kbn-xxx-zdt-algo-su…

5441475

…pport-v2-compat

add tests for zdt-v2-compat

7eb03fa

add compat integration tests

8a2e3e1

pgayvallet commented May 1, 2023

View reviewed changes

pgayvallet marked this pull request as ready for review May 1, 2023 12:06

pgayvallet requested a review from a team as a code owner May 1, 2023 12:06

jloleysens approved these changes May 2, 2023

View reviewed changes

pgayvallet added 2 commits May 2, 2023 12:22

Merge remote-tracking branch 'upstream/main' into kbn-xxx-zdt-algo-su…

54f96f3

…pport-v2-compat

review NITs

36cb6c6

pgayvallet enabled auto-merge (squash) May 2, 2023 10:32

pgayvallet merged commit 9cddb60 into elastic:main May 2, 2023

kibanamachine added the backport:skip This commit does not require backporting label May 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adapt the ZDT migration algorithm to support v2 migrations #155981

Adapt the ZDT migration algorithm to support v2 migrations #155981

pgayvallet commented Apr 27, 2023 •

edited

Loading

pgayvallet left a comment

pgayvallet May 1, 2023

pgayvallet May 1, 2023

pgayvallet May 1, 2023

TinaHeiligers May 2, 2023

elasticmachine commented May 1, 2023

jloleysens left a comment

jloleysens May 2, 2023

pgayvallet May 2, 2023

jloleysens May 2, 2023

jloleysens May 2, 2023

jloleysens May 2, 2023

jloleysens May 2, 2023

jloleysens May 2, 2023

pgayvallet May 2, 2023

kibana-ci commented May 2, 2023

API count

ESLint disabled line counts

Total ESLint disabled count

		/** @internal */
		export type IndexMappingMeta = V2AlgoIndexMappingMeta & ZdtAlgoIndexMappingMeta;

		roots: ['<rootDir>/src/core/server/integration_tests/saved_objects/migrations/zdt_v2_compat'],
		// must override to match all test given there is no `integration_tests` subfolder

	* If will either be the latest model version if the type
	* It will either be the latest model version if the type


		export type ModelVersionMap = Record<string, number>;
		export type VirtualVersion = string;

Adapt the ZDT migration algorithm to support v2 migrations #155981

Adapt the ZDT migration algorithm to support v2 migrations #155981

Conversation

pgayvallet commented Apr 27, 2023 • edited Loading

Summary

Technicals

pgayvallet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented May 1, 2023

jloleysens left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibana-ci commented May 2, 2023

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

API count

ESLint disabled line counts

Total ESLint disabled count

History

pgayvallet commented Apr 27, 2023 •

edited

Loading