Split the .kibana saved objects index into multiple indices #154888
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gsoldevila
added
release_note:enhancement
Team:Core
This was referenced Apr 13, 2023
Merged
Merged
2 tasks
rudolf
added
the
Epic:ScaleMigrations
…TypesMap` (#154892) Part of #154888 Some mappings.json use a special `$KIBANA_PACKAGE_VERSION` variable. ES archiver uses this variable to generate saved object indices with names that match the current stack version. Note that this a bit brittle, cause incompatible mappings changes would break Kibana. However, it allows skipping reindexing and saving some precious CI time when running tests. With the introduction of the .kibana split, the mappings `_meta` information now contain a new property, called `indexTypesMap`, which holds a list of the SO types that are part of each index. If the list is not present, the migration logic assumes that the .kibana index must be split (and its documents redistributed), and it forces a reindex. The goal of this PR is to enrich the `mappings.json` with that extra property, so that the migrator logic will not trigger a reindex. - Thus, these mappings must be updated whenever existing types are moved to a separate index, or new types are added. - Ideally, we should also update the corresponding `data.json` files, to place each SO document to the appropriate index.
…ltiple SO indices (#154884) Part of #154888 There are a bunch of FTR and integration tests that interact with the SO indices directly, assuming there are only two SO indices: `.kibana` and `.kibana_task_manager`. The goal of this PR is to factorise that knowledge and make these tests use a list of SO indices instead.
…154882) Part of #154888 ES archiver logic assumes that there are only two SO indices: '.kibana' and '.kibana_task_manager'. This PR updates ES archiver so that it works with a list of SO indices instead. --------- Co-authored-by: pgayvallet <[email protected]>
gsoldevila
pushed a commit
that referenced
this pull request
Apr 18, 2023
…#155035) ## Summary As part of #154888, we need to stop making direct requests to the index `.kibana`, and use the SO Clients instead. This PR changes the utility `getSavedObjectsCount` to use aggregations in the SO client. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### For maintainers - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) I'm pointing to `main` because it's an improvement we needed anyway. --------- Co-authored-by: kibanamachine <[email protected]>
2 tasks
… to use the correct index (#155155) ### **this PR is targeting the `dot-kibana-split` feature branch and not `main`** ## Summary Part of #154888 In #154888, we're going to split the `.kibana` savedObject index into multiple ones. For this reason, calls to `core.savedObjects.getKibanaIndex` will not necessarily return the correct value (e.g types that were moved out of this index) This PR introduces the following SO APIs: - `getDefaultIndex` - `getIndexForType` - `getIndicesForTypes` - `getAllIndices` And adapt plugins code to replace usages of `core.savedObjects.getKibanaIndex` with the proper alternative
….kibana_alerting_cases` (#154897) ## Summary **This is targeting a feature branch** Part of #154888 This PR aims at moving the following saved object types to a new `.kibana_alerting_cases ` system index: - action - action_task_params - alert - api_key_pending_invalidation - cases - cases-comments - cases-configure - cases-connector-mappings - cases-telemetry - cases-user-actions - connector_token - rules-settings --------- Co-authored-by: Christos Nasikas <[email protected]> Co-authored-by: pgayvallet <[email protected]>
michaelolo24
approved these changes
Apr 21, 2023
There was a problem hiding this comment.
Thanks for doing this work @gsoldevila, looks good for the timeline SO's! Just a question on whether or not QA teams have been looped in for additional testing? I'm not sure how prepared our teams are to test for any potential regressions while also managing any existing 8.8 feature work during the upcoming FF?
They have been, yes. If you think that a particular area is more at risk because of this PR, please feel free to inform us and/or QA. |
darnautov
approved these changes
Apr 24, 2023
## Summary **This is targeting a feature branch** Part of #154888 Following some remarks and feedback from the [feature branch PR](#154888), this PR aims at relocating some of the SO types: - Move `'url'` type back to `.kibana`. (related [comment](#154888 (comment)))
Thanks for letting us know! Whoever merges last should update the "dynamic" mappings.json (there's 8 of them), to get rid of the removed type |
yctercero
approved these changes
Apr 24, 2023
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]Module Count
Public APIs missing comments
Async chunks
Public APIs missing exports
Page load bundle
Saved Objects .kibana field count
Unknown metric groupsAPI count
ESLint disabled line counts
References to deprecated APIs
Total ESLint disabled count
History
To update your PR or re-run it, just comment with: |
This was referenced Apr 25, 2023
gsoldevila
added a commit
that referenced
this pull request
May 23, 2023
Documentation must reflect changes brought by the [dot kibana split](#154888). Our public facing documentation just mentions two indices e.g. https://www.elastic.co/guide/en/kibana/8.8/saved-object-migrations.html https://www.elastic.co/guide/en/kibana/8.8/resolve-migrations-failures.html#upgrade-migrations-old-indices this one in particular mentions that "all other indices are safe to delete".
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
May 23, 2023
Documentation must reflect changes brought by the [dot kibana split](elastic#154888). Our public facing documentation just mentions two indices e.g. https://www.elastic.co/guide/en/kibana/8.8/saved-object-migrations.html https://www.elastic.co/guide/en/kibana/8.8/resolve-migrations-failures.html#upgrade-migrations-old-indices this one in particular mentions that "all other indices are safe to delete". (cherry picked from commit 70eb9d3)
kibanamachine
referenced
this pull request
May 23, 2023
… (#158270) # Backport This will backport the following commits from `main` to `8.8`: - [Update saved objects migrations documentation for 8.8.0 (#158241)](#158241) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Gerard Soldevila","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-05-23T13:53:52Z","message":"Update saved objects migrations documentation for 8.8.0 (#158241)\n\nDocumentation must reflect changes brought by the [dot kibana\r\nsplit](https://github.com/elastic/kibana/pull/154888).\r\n\r\nOur public facing documentation just mentions two indices e.g.\r\n\r\nhttps://www.elastic.co/guide/en/kibana/8.8/saved-object-migrations.html\r\n\r\nhttps://www.elastic.co/guide/en/kibana/8.8/resolve-migrations-failures.html#upgrade-migrations-old-indices\r\nthis one in particular mentions that \"all other indices are safe to\r\ndelete\".","sha":"70eb9d3ecbb11768d091fca385bad4f20960deef","branchLabelMapping":{"^v8.9.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","release_note:skip","documentation","Feature:Migrations","backport:prev-minor","v8.8.0","v8.9.0","v8.8.1"],"number":158241,"url":"https://github.com/elastic/kibana/pull/158241","mergeCommit":{"message":"Update saved objects migrations documentation for 8.8.0 (#158241)\n\nDocumentation must reflect changes brought by the [dot kibana\r\nsplit](https://github.com/elastic/kibana/pull/154888).\r\n\r\nOur public facing documentation just mentions two indices e.g.\r\n\r\nhttps://www.elastic.co/guide/en/kibana/8.8/saved-object-migrations.html\r\n\r\nhttps://www.elastic.co/guide/en/kibana/8.8/resolve-migrations-failures.html#upgrade-migrations-old-indices\r\nthis one in particular mentions that \"all other indices are safe to\r\ndelete\".","sha":"70eb9d3ecbb11768d091fca385bad4f20960deef"}},"sourceBranch":"main","suggestedTargetBranches":["8.8"],"targetPullRequestStates":[{"branch":"8.8","label":"v8.8.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.9.0","labelRegex":"^v8.9.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/158241","number":158241,"mergeCommit":{"message":"Update saved objects migrations documentation for 8.8.0 (#158241)\n\nDocumentation must reflect changes brought by the [dot kibana\r\nsplit](https://github.com/elastic/kibana/pull/154888).\r\n\r\nOur public facing documentation just mentions two indices e.g.\r\n\r\nhttps://www.elastic.co/guide/en/kibana/8.8/saved-object-migrations.html\r\n\r\nhttps://www.elastic.co/guide/en/kibana/8.8/resolve-migrations-failures.html#upgrade-migrations-old-indices\r\nthis one in particular mentions that \"all other indices are safe to\r\ndelete\".","sha":"70eb9d3ecbb11768d091fca385bad4f20960deef"}}]}] BACKPORT--> Co-authored-by: Gerard Soldevila <[email protected]>
delanni
pushed a commit
to delanni/kibana
that referenced
this pull request
May 25, 2023
Documentation must reflect changes brought by the [dot kibana split](elastic#154888). Our public facing documentation just mentions two indices e.g. https://www.elastic.co/guide/en/kibana/8.8/saved-object-migrations.html https://www.elastic.co/guide/en/kibana/8.8/resolve-migrations-failures.html#upgrade-migrations-old-indices this one in particular mentions that "all other indices are safe to delete".
This was referenced May 26, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fix #104081
This PR move some of the SO types from the
.kibanaindex into the following ones:.kibana_alerting_cases.kibana_analytics.kibana_security_solution.kibana_ingestThis split/reallocation will occur during the
8.8.0Kibana upgrade (meaning: from any version older than8.8.0to any version greater or equal to8.8.0)This PR main changes are:
.kibanaand usage of thecore.savedObjects.getKibanaIndex()to use new APIs to target the correct index/indicesTo reviewers
Overall estimated risk of regressions: low
But, still, please take the time to review changes in your code. The parts of the production code that were the most impacted are the telemetry collectors, as most of them were performing direct requests against the
.kibanaindex, so we had to adapt them. Most other contributor-owned changes are in FTR tests and datasets.If you think a type is misplaced (either we missed some types that should be moved to a specific index, or some types were moved and shouldn't have been) please tell us, and we'll fix the reallocation either in this PR or in a follow-up.
.Kibana split
The following new indices are introduced by this PR, with the following SO types being moved to it. (any SO type not listed here will be staying in its current index)
Note: The complete type => index breakdown is available in this spreadsheet.
.kibana_alerting_cases.kibana_security_solution.kibana_analytics.kibana_ingestTasks / PRs
Sub-PRs
Implementation
indexTypesMap#154892core.savedObjects.getKibanaIndexto use the correct index #155155Individual index split
.kibana_alerting_cases#154897.kibana_security_solution#155129.kibana_analytics#155140.kibana_ingest#155130Improvements / follow-ups
getCurrentIndexTypesMapresillient to intermittent failures Allow relocating SO to different indices during migration #154151 (comment)getSavedObjectsCount] UsesoClientinstead of.kibanasearches #155035Reallocation tweaks
Tweaks to the reallocation can be done after the initial merge, as long as it's done before the public release of 8.8
urlshould get back to.kibana(see comment)Release Note
For performance purposes, Kibana is now using more system indices to store its internal data.
The following system indices will be created when upgrading to
8.8.0:.kibana_alerting_cases.kibana_analytics.kibana_security_solution.kibana_ingest