Only migrate an index if necessary

[rudolf]

In order to support different Kibana instances having different plugins enabled, the v2 migration algorithm had to perform a migration for every version, because even if instance 1 doesn't think any migrations are necessary, instance 2 might have a plugin which defines a migration for this version.

Given that we no longer support unknown saved object types #101351 I don't think there are any reasons that we need to run the migration on every upgrade. If the mappings have stayed the same, we could just add new version specific aliases to the existing index. This will improve the performance of migrations in the majority of cases,

but for some edge case configurations it could cause unexpected downtime once a plugin gets enabled

If a Kibana deployment has:

• a disabled plugin
• the plugin has never created any saved objects
• the plugin has a migration function which hasn't been run

Then, as soon as the plugin is enabled, Kibana will see that the indices are no longer up to date and will therefore cause downtime while performing a full migration because of the migration function/mapping changes in the newly enabled plugin.

This means we could have an alias like .kibana_8.0.0 pointing to the index .kibana_7.16.0_001 which might seem wrong. We would have to educate users/support that this could be the expected behaviour. Alternatively we could clone the index into .kibana_8.0.0_001 but this seems expensive. Another alternative is to obfuscate the version by hashing the version number. Though at the end of the day this is internal implementation details that users shouldn't have to even know about so just keeping the existing index name with a version-specific alias seems simpler.

This would be the first step towards scaling migration performance by having an index per saved object type #104081 It would also give immediate benefits since we typically migrate the task manager index much less frequently than the .kibana index.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[rudolf]

An additional benefit of checking the mappings would be that we don't need to run the UPDATE_TARGET_MAPPINGS step on every restart. While this step is usually fairly fast, we have had some reports that clusters where the .kibana index is really large like > 10GB this could take a significant amount of time (> 7 minutes).

[rudolf]

For reference...

Here is where we calculate the md5 hash of every plugins mapping fields: https://github.com/elastic/kibana/blob/main/packages/core/saved-objects/core-saved-objects-migration-server-internal/src/core/build_active_mappings.ts#L27

And we have an existing diffMappings function to test if mappings are different in an incompatible way. This was used in v1 migrations but we didn't yet plug it into v2 migrations
https://github.com/elastic/kibana/blob/main/packages/core/saved-objects/core-saved-objects-migration-server-internal/src/core/build_active_mappings.ts#L49

[rudolf]

We will have to remember that upgrades from 7.x to 8.x rewrite the _id of saved object types that become namespaceType: multiple-isolated. In theory, all saved objects that wanted to become multiple-isolated would have done so with the 8.0 release but there are no technical controls to guarantee this.

We would either have to add a check so that we still kick off a full migration if a type becomes multiple-isolated, but such a check would likely only work when upgrading to the next release. E.g. we can add the check in 8.8 and then when someone upgrades from 8.8 to 8.x we could safely skip the reindexing but this significantly increases the time to creating value for users.

Alternatively, we could audit all 8.x releases and confirm that there has been no changes to namespaceType: 'multiple-isolated'. We could then add a technical control to prevent further changes (or rely on our existing automatic code reviews for migration-related changes?). This would allow us to assume any upgrade from 8.0+ can safely skip the reindexing if the mappings didn't change.

[pgayvallet]

We could then add a technical control to prevent further changes (or rely on our existing automatic code reviews for migration-related changes?). This would allow us to assume any upgrade from 8.0+ can safely skip the reindexing if the mappings didn't change.

We know we will need to prevent changing more existing types to be shareable for other features (as we won't be able to support id rewriting), so I'd say checking / blocking SavedObjectType.convertToMultiNamespaceTypeVersion from being higher than an arbitrary version would be acceptable?

[rudolf]

so I'd say checking / blocking SavedObjectType.convertToMultiNamespaceTypeVersion from being higher than an arbitrary version would be acceptable?

Yes, that's a really nice way to add a technical control 😁

[pgayvallet]

This means we could have an alias like .kibana_8.0.0 pointing to the index .kibana_7.16.0_001 which might seem wrong. We would have to educate users/support that this could be the expected behaviour.

I know this probably implies more changes, but given we kinda already know we're planning on decoupling model version from Kibana version, would it make sense to go further and change the index naming to something not including the version number? We will likely need to do it later anyway, so now may be the correct option. WDTY @rudolf?

(Hard not to troll here, but we could, idk, use a number-based versioning instead?)

[rudolf]

Hard not to troll here, but we could, idk, use a number-based versioning instead?

That would just be absurd 👎

[rudolf]

Unless we complete #147237 at the same time we will probably have to stop assuming there's just one index name per version to handle the edge cases. Because you could be on v8.8.0 so you have the index .kibana_8.8.0_001 but then you enable a plugin that defines new mappings so we need a new index but we're still on v8.8.0.

Because enabling a new plugin would always only add new fields it would always be a compatible mapping change so #147237 kinda solves this problem.

Edit: in this scenario we'd detect that a version migration was already completed and update the existing index with the mappings of the new plugin.

One useful feature of the version number in the index is that we block upgrades if that version number is newer. So if we remove it from the index we would have to put the Kibana version also into the mappings _meta.