-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow for additive mappings update without creating a new version index #147237
Comments
Pinging @elastic/kibana-core (Team:Core) |
Dropping an apparently compatible but not really compatible use case: #147341 adds a compatible change (it adds a field to the mappings). However, a reindex is required so sorting by |
In #145604 when Kibana starts up and detects the mappings of the index doesn't match what it expects, it will update the mappings, but also run an Assuming that a new field added to the mappings necessarily requires reading and updating all documents in that index is quite expensive. We could optimise it somewhat by only targeting documents of the type of the saved object that had its mappings update instead of updating all documents. E.g. if a dashboard changes its mappings in a compatible way, we need an |
…ex (#149326) Fixes [#147237](#147237) Based on the same principle as [#147371](#147371), the goal of this PR is to **avoid reindexing if possible**. This time, the idea is to check whether the new mappings are still compatible with the ones stored in ES. To to so, we attempt to update the mappings in place in the existing index, introducing a new `CHECK_COMPATIBLE_MAPPINGS` step: * If the update operation fails, we assume the mappings are NOT compatible, and we continue with the normal reindexing flow. * If the update operation succeeds, we assume the mappings ARE compatible, and we skip reindexing, just like [#147371](#147371) does. ![image](https://user-images.githubusercontent.com/25349407/216979882-9fe9f034-b521-4171-b85d-50be6a13e179.png) --------- Co-authored-by: kibanamachine <[email protected]>
…ex (elastic#149326) Fixes [elastic#147237](elastic#147237) Based on the same principle as [elastic#147371](elastic#147371), the goal of this PR is to **avoid reindexing if possible**. This time, the idea is to check whether the new mappings are still compatible with the ones stored in ES. To to so, we attempt to update the mappings in place in the existing index, introducing a new `CHECK_COMPATIBLE_MAPPINGS` step: * If the update operation fails, we assume the mappings are NOT compatible, and we continue with the normal reindexing flow. * If the update operation succeeds, we assume the mappings ARE compatible, and we skip reindexing, just like [elastic#147371](elastic#147371) does. ![image](https://user-images.githubusercontent.com/25349407/216979882-9fe9f034-b521-4171-b85d-50be6a13e179.png) --------- Co-authored-by: kibanamachine <[email protected]>
In #124946 we will only migrate saved objects to a new version specific index if the mappings changed, otherwise we keep the existing index and only transform outdated documents.
An additional optimization would be to only create a new version specific index if the mappings change is incompatible with the existing index. For instance, adding a new field or a
ignore_above
to akeyword
field can be done in place. Whereas, removing a field or changing a text field type to a numeric one isn't.The behaviour for when a mappings change is compatible or not isn't always obvious. I can't remember the specifics but I have been surprised in the past that some changes I thought should be compatible weren't. So except for when a field is removed, Kibana should not be trying to determine whether a change is compatible, it should just send the update request to Elasticsearch. If the update succeeds we can proceed without a new index, if not, we need a new version specific index.
The text was updated successfully, but these errors were encountered: