[dot-kibana-split] Split rules, alerts and cases saved objects into `.kibana_alerting_cases` #154897

gsoldevila · 2023-04-13T09:49:19Z

Summary

This is targeting a feature branch
Part of #154888

This PR aims at moving the following saved object types to a new .kibana_alerting_cases system index:

action
action_task_params
alert
api_key_pending_invalidation
cases
cases-comments
cases-configure
cases-connector-mappings
cases-telemetry
cases-user-actions
connector_token
rules-settings

x-pack/test/cases_api_integration/security_and_spaces/tests/common/cases/find_cases.ts

cnasikas

Cases changes LGTM!

…ana-split-cases

… into dot-kibana-split-cases

kibana-ci · 2023-04-19T13:49:55Z

💔 Build Failed

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #11 / detection engine api security and spaces enabled - Group 10 migrate_legacy_actions "before all" hook for "migrates legacy actions for rule with no actions"
[job] [logs] FTR Configs #11 / detection engine api security and spaces enabled - Group 10 migrate_legacy_actions "before all" hook for "migrates legacy actions for rule with no actions"
[job] [logs] FTR Configs #11 / detection engine api security and spaces enabled - Group 10 resolve_read_rules reading rules should create a rule and a "conflicting rule" where the SO _id matches the sourceId (see legacy-url-alias SO) of a migrated rule
[job] [logs] FTR Configs #11 / detection engine api security and spaces enabled - Group 10 resolve_read_rules reading rules should create a rule and a "conflicting rule" where the SO _id matches the sourceId (see legacy-url-alias SO) of a migrated rule
[job] [logs] FTR Configs #13 / detection engine api security and spaces enabled - Group 3 create_rules_with_exceptions creating rules with exceptions elastic admin should merge back an exceptions_list if it was removed from the immutable rule through PATCH
[job] [logs] FTR Configs #13 / detection engine api security and spaces enabled - Group 3 create_rules_with_exceptions creating rules with exceptions elastic admin should merge back an exceptions_list if it was removed from the immutable rule through PATCH
[job] [logs] FTR Configs #13 / detection engine api security and spaces enabled - Group 3 create_rules_with_exceptions creating rules with exceptions elastic admin should override any updates to pre-packaged rules if the user removes the exception list through the API but the new version of a rule has an exception list again
[job] [logs] FTR Configs #13 / detection engine api security and spaces enabled - Group 3 create_rules_with_exceptions creating rules with exceptions elastic admin should override any updates to pre-packaged rules if the user removes the exception list through the API but the new version of a rule has an exception list again

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/core-saved-objects-server`	101	102	+1

Unknown metric groups

API count

id	before	after	diff
`@kbn/core-saved-objects-server`	504	505	+1

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

elasticmachine · 2023-04-19T15:50:05Z

Pinging @elastic/kibana-core (Team:Core)

elasticmachine · 2023-04-19T15:50:07Z

Pinging @elastic/response-ops (Team:ResponseOps)

elasticmachine · 2023-04-19T15:50:10Z

Pinging @elastic/response-ops-cases (Feature:Cases)

jonathan-buttner · 2023-04-20T12:28:09Z

...k/test/cases_api_integration/security_and_spaces/tests/common/kibana_alerting_cases_index.ts

+
+  describe('Kibana index: Alerting & Cases', () => {
+    before(async () => {
+      await esArchiver.load('x-pack/test/functional/es_archives/cases/migrations/8.8.0');


Would it be possible to use kbn archiver here instead?

Talked to Christos offline, and we can't use kbn archiver because it would just go to the new index and skip the migration.

Yeah, for migrations, we're forced to use esArchiver to load the old index state unfortunately

(FWIW this PR has been merged against the feature branch - #154888)

@gsoldevila

## Description Fix #104081 This PR move some of the SO types from the `.kibana` index into the following ones: - `.kibana_alerting_cases` - `.kibana_analytics` - `.kibana_security_solution` - `.kibana_ingest` This split/reallocation will occur during the `8.8.0` Kibana upgrade (*meaning: from any version older than `8.8.0` to any version greater or equal to `8.8.0`*) **This PR main changes are:** - implement the changes required in the SO migration algorithm to support this reallocation - update the FTR tools (looking at you esArchiver) to support these new indices - update hardcoded references to `.kibana` and usage of the `core.savedObjects.getKibanaIndex()` to use new APIs to target the correct index/indices - update FTR datasets, tests and utility accordingly ## To reviewers **Overall estimated risk of regressions: low** But, still, please take the time to review changes in your code. The parts of the production code that were the most impacted are the telemetry collectors, as most of them were performing direct requests against the `.kibana` index, so we had to adapt them. Most other contributor-owned changes are in FTR tests and datasets. If you think a type is misplaced (either we missed some types that should be moved to a specific index, or some types were moved and shouldn't have been) please tell us, and we'll fix the reallocation either in this PR or in a follow-up. ## .Kibana split The following new indices are introduced by this PR, with the following SO types being moved to it. (any SO type not listed here will be staying in its current index) Note: The complete **_type => index_** breakdown is available in [this spreadsheet](https://docs.google.com/spreadsheets/d/1b_MG_E_aBksZ4Vkd9cVayij1oBpdhvH4XC8NVlChiio/edit#gid=145920788). #### `.kibana_alerting_cases` - action - action_task_params - alert - api_key_pending_invalidation - cases - cases-comments - cases-configure - cases-connector-mappings - cases-telemetry - cases-user-actions - connector_token - rules-settings - maintenance-window #### `.kibana_security_solution` - csp-rule-template - endpoint:user-artifact - endpoint:user-artifact-manifest - exception-list - exception-list-agnostic - osquery-manager-usage-metric - osquery-pack - osquery-pack-asset - osquery-saved-query - security-rule - security-solution-signals-migration - siem-detection-engine-rule-actions - siem-ui-timeline - siem-ui-timeline-note - siem-ui-timeline-pinned-event #### `.kibana_analytics` - canvas-element - canvas-workpad-template - canvas-workpad - dashboard - graph-workspace - index-pattern - kql-telemetry - lens - lens-ui-telemetry - map - search - search-session - search-telemetry - visualization #### `.kibana_ingest` - epm-packages - epm-packages-assets - fleet-fleet-server-host - fleet-message-signing-keys - fleet-preconfiguration-deletion-record - fleet-proxy - ingest_manager_settings - ingest-agent-policies - ingest-download-sources - ingest-outputs - ingest-package-policies ## Tasks / PRs ### Sub-PRs **Implementation** - 🟣 #154846 - 🟣 #154892 - 🟣 #154882 - 🟣 #154884 - 🟣 #155155 **Individual index split** - 🟣 #154897 - 🟣 #155129 - 🟣 #155140 - 🟣 #155130 ### Improvements / follow-ups - 👷🏼 Extract logic into [runV2Migration](#154151 (comment)) @gsoldevila - Make `getCurrentIndexTypesMap` resillient to intermittent failures #154151 (comment) - 🚧 Build a more structured [MigratorSynchronizer](#154151 (comment)) - 🟣 #155035 - 🟣 #155116 - 🟣 #155366 ## Reallocation tweaks Tweaks to the reallocation can be done after the initial merge, as long as it's done before the public release of 8.8 - `url` should get back to `.kibana` (see [comment](#154888 (comment))) ## Release Note For performance purposes, Kibana is now using more system indices to store its internal data. The following system indices will be created when upgrading to `8.8.0`: - `.kibana_alerting_cases` - `.kibana_analytics` - `.kibana_security_solution` - `.kibana_ingest` --------- Co-authored-by: pgayvallet <[email protected]> Co-authored-by: Christos Nasikas <[email protected]> Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Georgii Gorbachev <[email protected]>

gsoldevila added 2 commits April 13, 2023 11:29

Split cases saved objects into .kibana_cases

92620e7

Fix UT failing due to sort() discrepancies

80f4d48

gsoldevila changed the title ~~[dot-kibana-split] Split _cases_ saved objects into .kibana_cases~~ [dot-kibana-split] Split 'cases' saved objects into .kibana_cases Apr 13, 2023

gsoldevila mentioned this pull request Apr 13, 2023

Split the .kibana saved objects index into multiple indices #154888

Merged

gsoldevila changed the base branch from main to dot-kibana-split April 13, 2023 10:07

cnasikas reviewed Apr 13, 2023

View reviewed changes

x-pack/test/cases_api_integration/security_and_spaces/tests/common/cases/find_cases.ts Outdated Show resolved Hide resolved

cnasikas approved these changes Apr 13, 2023

View reviewed changes

gsoldevila changed the title ~~[dot-kibana-split] Split 'cases' saved objects into .kibana_cases~~ [dot-kibana-split] Split rules, alerts and cases saved objects into .kibana_alerting_cases Apr 13, 2023

cnasikas force-pushed the dot-kibana-split-cases branch 2 times, most recently from 9383b7f to 80f4d48 Compare April 18, 2023 07:48

cnasikas and others added 5 commits April 18, 2023 10:50

Update cases index

2049c95

Merge remote-tracking branch 'upstream/dot-kibana-split' into dot-kib…

c2c64b8

…ana-split-cases

Merge branch 'dot-kibana-split' into dot-kibana-split-cases

87ef51f

Export cases index from core package

bc57f5f

Move response ops SOs to the new index

c3b0af1

gsoldevila changed the title ~~[dot-kibana-split] Split rules, alerts and cases saved objects into .kibana_alerting_cases~~ [dot-kibana-split] Split rules, alerts and cases saved objects into .kibana_alerting_cases Apr 18, 2023

gsoldevila and others added 7 commits April 18, 2023 17:39

Update mappings.json to prevent migrations for "dynamic" archives

2852cc1

Merge branch 'dot-kibana-split' into dot-kibana-split-cases

90c6020

Add 'maintenance-window' to dynamic mappings.json

42774e2

Merge remote-tracking branch 'upstream/dot-kibana-split' into dot-kib…

573a928

…ana-split-cases

Add tests for Cases

28195a9

Merge remote-tracking branch 'upstream/dot-kibana-split' into dot-kib…

ca20aec

…ana-split-cases

Merge branch 'dot-kibana-split-cases' of github.com:gsoldevila/kibana…

4b95ae3

… into dot-kibana-split-cases

cnasikas and others added 4 commits April 19, 2023 13:45

Fix action task params tests

c44acfc

Fix alerting FTR tests

d6b951b

fix test

cbc8657

Fix data.json which doesn't have a mappings.json

e47116f

pgayvallet marked this pull request as ready for review April 19, 2023 15:50

pgayvallet requested review from a team as code owners April 19, 2023 15:50

pgayvallet merged commit 785baa0 into elastic:dot-kibana-split Apr 19, 2023

jonathan-buttner reviewed Apr 20, 2023

View reviewed changes

rudolf added the Epic:ScaleMigrations Scale upgrade migrations to millions of saved objects label Jun 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dot-kibana-split] Split rules, alerts and cases saved objects into `.kibana_alerting_cases` #154897

[dot-kibana-split] Split rules, alerts and cases saved objects into `.kibana_alerting_cases` #154897

gsoldevila commented Apr 13, 2023 •

edited

Loading

cnasikas left a comment

kibana-ci commented Apr 19, 2023 •

edited

Loading

API count

elasticmachine commented Apr 19, 2023

elasticmachine commented Apr 19, 2023

elasticmachine commented Apr 19, 2023

jonathan-buttner Apr 20, 2023

jonathan-buttner Apr 20, 2023

pgayvallet Apr 20, 2023

[dot-kibana-split] Split rules, alerts and cases saved objects into .kibana_alerting_cases #154897

[dot-kibana-split] Split rules, alerts and cases saved objects into .kibana_alerting_cases #154897

Conversation

gsoldevila commented Apr 13, 2023 • edited Loading

Summary

cnasikas left a comment

Choose a reason for hiding this comment

kibana-ci commented Apr 19, 2023 • edited Loading

💔 Build Failed

Failed CI Steps

Test Failures

Metrics [docs]

Public APIs missing comments

API count

History

elasticmachine commented Apr 19, 2023

elasticmachine commented Apr 19, 2023

elasticmachine commented Apr 19, 2023

jonathan-buttner Apr 20, 2023

Choose a reason for hiding this comment

jonathan-buttner Apr 20, 2023

Choose a reason for hiding this comment

pgayvallet Apr 20, 2023

Choose a reason for hiding this comment

[dot-kibana-split] Split rules, alerts and cases saved objects into `.kibana_alerting_cases` #154897

[dot-kibana-split] Split rules, alerts and cases saved objects into `.kibana_alerting_cases` #154897

gsoldevila commented Apr 13, 2023 •

edited

Loading

kibana-ci commented Apr 19, 2023 •

edited

Loading