[dot-kibana-split] Misc SO type relocations #155366
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gsoldevila
added
Team:Security
|
Pinging @elastic/kibana-security (Team:Security) |
thomheymann
approved these changes
Apr 20, 2023
💚 Build Succeeded
Metrics [docs]Saved Objects .kibana field count
To update your PR or re-run it, just comment with: |
gsoldevila
added a commit
that referenced
this pull request
Apr 25, 2023
## Description Fix #104081 This PR move some of the SO types from the `.kibana` index into the following ones: - `.kibana_alerting_cases` - `.kibana_analytics` - `.kibana_security_solution` - `.kibana_ingest` This split/reallocation will occur during the `8.8.0` Kibana upgrade (*meaning: from any version older than `8.8.0` to any version greater or equal to `8.8.0`*) **This PR main changes are:** - implement the changes required in the SO migration algorithm to support this reallocation - update the FTR tools (looking at you esArchiver) to support these new indices - update hardcoded references to `.kibana` and usage of the `core.savedObjects.getKibanaIndex()` to use new APIs to target the correct index/indices - update FTR datasets, tests and utility accordingly ## To reviewers **Overall estimated risk of regressions: low** But, still, please take the time to review changes in your code. The parts of the production code that were the most impacted are the telemetry collectors, as most of them were performing direct requests against the `.kibana` index, so we had to adapt them. Most other contributor-owned changes are in FTR tests and datasets. If you think a type is misplaced (either we missed some types that should be moved to a specific index, or some types were moved and shouldn't have been) please tell us, and we'll fix the reallocation either in this PR or in a follow-up. ## .Kibana split The following new indices are introduced by this PR, with the following SO types being moved to it. (any SO type not listed here will be staying in its current index) Note: The complete **_type => index_** breakdown is available in [this spreadsheet](https://docs.google.com/spreadsheets/d/1b_MG_E_aBksZ4Vkd9cVayij1oBpdhvH4XC8NVlChiio/edit#gid=145920788). #### `.kibana_alerting_cases` - action - action_task_params - alert - api_key_pending_invalidation - cases - cases-comments - cases-configure - cases-connector-mappings - cases-telemetry - cases-user-actions - connector_token - rules-settings - maintenance-window #### `.kibana_security_solution` - csp-rule-template - endpoint:user-artifact - endpoint:user-artifact-manifest - exception-list - exception-list-agnostic - osquery-manager-usage-metric - osquery-pack - osquery-pack-asset - osquery-saved-query - security-rule - security-solution-signals-migration - siem-detection-engine-rule-actions - siem-ui-timeline - siem-ui-timeline-note - siem-ui-timeline-pinned-event #### `.kibana_analytics` - canvas-element - canvas-workpad-template - canvas-workpad - dashboard - graph-workspace - index-pattern - kql-telemetry - lens - lens-ui-telemetry - map - search - search-session - search-telemetry - visualization #### `.kibana_ingest` - epm-packages - epm-packages-assets - fleet-fleet-server-host - fleet-message-signing-keys - fleet-preconfiguration-deletion-record - fleet-proxy - ingest_manager_settings - ingest-agent-policies - ingest-download-sources - ingest-outputs - ingest-package-policies ## Tasks / PRs ### Sub-PRs **Implementation** - 🟣 #154846 - 🟣 #154892 - 🟣 #154882 - 🟣 #154884 - 🟣 #155155 **Individual index split** - 🟣 #154897 - 🟣 #155129 - 🟣 #155140 - 🟣 #155130 ### Improvements / follow-ups - 👷🏼 Extract logic into [runV2Migration](#154151 (comment)) @gsoldevila - Make `getCurrentIndexTypesMap` resillient to intermittent failures #154151 (comment) - 🚧 Build a more structured [MigratorSynchronizer](#154151 (comment)) - 🟣 #155035 - 🟣 #155116 - 🟣 #155366 ## Reallocation tweaks Tweaks to the reallocation can be done after the initial merge, as long as it's done before the public release of 8.8 - `url` should get back to `.kibana` (see [comment](#154888 (comment))) ## Release Note For performance purposes, Kibana is now using more system indices to store its internal data. The following system indices will be created when upgrading to `8.8.0`: - `.kibana_alerting_cases` - `.kibana_analytics` - `.kibana_security_solution` - `.kibana_ingest` --------- Co-authored-by: pgayvallet <[email protected]> Co-authored-by: Christos Nasikas <[email protected]> Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Georgii Gorbachev <[email protected]>
rudolf
added
the
Epic:ScaleMigrations
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This is targeting a feature branch
Part of #154888
Following some remarks and feedback from the feature branch PR, this PR aims at relocating some of the SO types:
'url'type back to.kibana. (related comment)