Improve migrations performance by using distinct indices per SO type

[pgayvallet]

This is a follow-up on [https://elasticco.atlassian.net/browse/KBNA-7838|https://elasticco.atlassian.net/browse/KBNA-7838|smart-link]

In this phase we propose to split some saved object types out of the .kibana index into separate indices

[https://github.com//issues/104081|https://github.com//issues/104081|smart-link]
While prior work can reduce downtime for some upgrades, if just one saved object type defines a migration all saved objects still need to be migrated. E.g. there might be no cases migrations defined but there is a dashboard migration which then requires us to migrate all 1m cases. This change would mean we would only migrate the 1m cases if there is a cases migration defined. While this reduces the average downtime per upgrade it does introduce unpredictability for users where some upgrades are fast and others cause 10minutes of downtime.

We already thought a lot about splitting the kibana index into multiple ones, either one index by type, or potentially one index per solution (#90817). If we were to go into that direction, we could potentially increase the migration speed significantly, by only performing the migration process on the indices containing documents that need to be migrated.

As an example, let’s say that we got 3 SO types, foo, bar and dolly, respectively in the .kibana_foo_8_0_0, .kibana_bar_8_0_0 and .kibana_dolly_8_0_0 indices

During the migration from v8.0.0 to v8.2.1, we detect that we only have migrations to apply to the foo type. We could then only apply the migration to the .kibana_foo_8_0_0 index, and just clone the other indices to their new version.

We could even go further: If we keep inside an internal index the current version (name) of each index, we could potentially even avoid the cloning of the .kibana_bar_8_0_0 and .kibana_dolly_8_0_0 indices, and just have the next version use the exact same indices.

This could drastically reduce the average migration time, especially if we choose to have one index per type (as opposed to one index per grouping of types)

Note that this would have the additional advantage of handling the problematic of the increasing number of registered fields (and dodging once and for all the Damocles sword of hitting the limit), see #70471

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[rudolf]

We should remember that the core usage collector for saved objects currently only collects saved objects per type for the "default saved objects index" https://github.com/elastic/kibana/blob/main/src/plugins/kibana_usage_collection/server/plugin.ts#L126

So we would need to refactor this collector to check the saved objects registry during start to find all the indices of all the types.

[rudolf]

Some initial thoughts around the design:

There's several options to accomplish this. Some things could easily be done in one state machine by having the action target all indices at once, like the REINDEX_* step. But this becomes complex when different indices require different actions, e.g. some indices need to be created for the first time, others need to be migrated from a legacy version and yet others might already have an index with compatible mappings.

So it feels simplest to keep separate state machines per index but allow each state machine to block before progressing to the next step until all other state machines are past a certain step.

[pgayvallet]

Folllowing our discussions from yesterday, especially what @gsoldevila proposed, here's my brainstorm on the kind of trade-offs and optimization we could make:

If we look at the use cases, I think we have two main situations:

1. we're migrating all types of a given index to the same index
2. we're migrating some types of a given index to the same index and some other types from the said index to a non-existing index

Now, the main question I have is: Do we want/need to allow doing more than that?

E.g would we want to support something like

• during 8.8, I move case-foo from .kibana to .kibana-cases (which is a new index I would be creating during this 8.8 migration)
• during 8.9, I move case-bar from .kibana to .kibana-cases (which already exists)

My point being, if we define that:

1. the action of creating a new index and moving types to it as 'atomical' and bound to a release (so basically you can't move types to an index that already exists)
2. you can only move from one source index to a new index (e.g we can't move types from both .kibana and .kibana_task_manager to .kibana_foo)

I think it would significantly lower the needs in term of synchronization, as I think we could, as @gsoldevila suggested, have most of the work be performed in 'isolation' from a single migrator.

I'm not speaking in term of technicals here (the migration could instantiate other migrators or something similar) but in term of responsibilities / concerns here, as each migrator wouldn't need to sync or interact with the others

wdyt all? does that make sense?

[gsoldevila]

If we look at the use cases, I think we have two main situations:

1. we're migrating all types of a given index to the same index
2. we're migrating some types of a given index to the same index and some other types from the said index to a non-existing index

If we think about how all that fits into serverless, none of them are going to take place in the near future anymore.
We won't reindex, and we won't move SO types around to different indices either. So in the end, we'll have a much simpler state machine once serverless is widespread.

I was imagining the "after-split" functioning mode as a sort of v3 migrator that would bring us very close to serverless. A simplified one that only knows about updating mappings in place.

With that in mind, I define 2 main scenarios:

1. We have a new deployment, or one that it's already in v3 (split) state. In this scenario we start one v3 migrator per "solution" index (or whatever SO type grouping we decide).
2. We have a v1 or v2 deployment. In this scenario, I'd tweak the v2 migrator to perform the split, using a single migrator in 'isolation'. This migrator would create all indices of the split, and fill each of them with the corresponding documents. This would be a one off, expensive operation. After the migration to v3 is finished, consecutive restarts will always use v3 logic, and no further breaking changes or splitting would be allowed.

This way, we can use the "splitting into separate indices" as an opportunity to take us a step closer to serverless. We'd also isolate the v3 logic in a separate "component", easier to maintain and update.

I think this also fits well with Pierre's suggestion of having moves bound to a release. After giving it some thought, Pierre's proposal is more flexible as it would allow different moves between indices in the future, which we could turn to in case of facing issues like mapping explosions. Nevertheless, that would come at the cost of some downtime, and we would also be keeping some (a priori) unnecessary complexity in the code.

[rudolf]

Spoke to @gsoldevila and @jloleysens. We agreed that reducing the code complexity of the migrator (especially the model) would be good. At a high level we could split up v1/v2/v3 migrations into their own state machines. It would also be useful to have a state machine of state machines so that the INIT and CHECK_MIGRATION_VERSION could benefit from all the retry logic and logging of the state action machine. It could also help to split out some parts of e.g. a v2 state machine into sub-state machines.

(illustration from @gsoldevila)

When we do such a refactoring is debatable. If type splitting doesn't require a lot of risky changes to the state machine that would add a lot of additional complexity it feels like we should push for reducing downtime first and then refactor once that's released.

When it comes to what a v3 migrator might look like this is a basic way to visualise the proposal. A single migrator instance that does it's actions for all the indices it manages e.g. creates 3 temp indices, clones 3 temp indices performs the MARK_VERSION_INDEX_READY for 3 indices etc.

Let's call independant migratiors proposal (1) and this proposal (2). Both approaches require injecting the typeRegistry so that REINDEX_SOURCE_TO_TEMP_INDEX_BULK can create bulkWrite operations with the correct destination index for each document.

In (1) the coordination happens between the migrators in (2) the coordination happens within a migrator. But in both cases the coordination is only necessary inside a single Kibana instance, we're not talking about distributed synchronisation across instances.

I think (1) would require much less code changes though. We would leave all actions untouched but introduce two new steps READY_TO_REINDEX_SYNC and DONE_REINDEXING_SYNC. We would also have to inject a function into the migrators that allows them to query the controlState of other migrators this would require some changes but since it doesn't impact the algorithm it's low risk changes. And then in the *_SYNC steps we would wait indefinitely for all migrators to reach this state. To increase visibility we would want to model this as retries every 60s like READY_TO_REINDEX_SYNC -> READY_TO_REINDEX_SYNC so that we can see the log activity instead of it looking like nothing is happening.

The previous illustrations were kinda mixing the state machine controlState and the writing of documents into indexes which made it harder to understand, so I updated that to hopefully make that clearer

I also agree with the previous comments that we don't need to design for a flexibly splitting strategy that can change if we decide to split types differently in the future. This is out of scope and if we can simplify the implementation without it that'd be fine. But I think it's possible without any further change (illustrated here with the orange arrow which moves a document out of the cases index and back to the .kibana index.

[rudolf]

We also discussed why a clone is necessary even if it's a newly split index because it avoids lost deletes when theres multiple instances.