Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[dot-kibana-split] Generalise FTR and integration tests to support multiple SO indices #154884

Merged
merged 2 commits into from
Apr 18, 2023
Merged

[dot-kibana-split] Generalise FTR and integration tests to support multiple SO indices #154884

merged 2 commits into from
Apr 18, 2023

Conversation

gsoldevila
Copy link
Contributor

@gsoldevila gsoldevila commented Apr 13, 2023 •
edited
Loading

Part of #154888

There are a bunch of FTR and integration tests that interact with the SO indices directly, assuming there are only two SO indices: .kibana and .kibana_task_manager.

The goal of this PR is to factorise that knowledge and make these tests use a list of SO indices instead.

@gsoldevila gsoldevila added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Team:Security Team focused on: Auth, Users, Roles, Spaces, Audit Logging, and more! enhancement New value added to drive a business result technical debt Improvement of the software architecture and operational architecture release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.8.0 labels Apr 13, 2023
@gsoldevila gsoldevila mentioned this pull request Apr 13, 2023
Merged
@gsoldevila gsoldevila changed the title Generalise FTR and integration tests to support multiple SO indices [dot-kibana-split] Generalise FTR and integration tests to support multiple SO indices Apr 13, 2023
@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/core-saved-objects-server 100 101 +1
Unknown metric groups

API count

id before after diff
@kbn/core-saved-objects-server 494 496 +2

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@pgayvallet pgayvallet marked this pull request as ready for review April 13, 2023 13:24
@pgayvallet pgayvallet requested review from a team as code owners April 13, 2023 13:24
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-security (Team:Security)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

spong
spong approved these changes Apr 13, 2023
View reviewed changes
Copy link
Member

@spong spong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security Solution changes LGTM -- thanks for the cleanup here @gsoldevila! 👍

@kc13greiner kc13greiner self-requested a review April 13, 2023 14:41
kc13greiner
kc13greiner approved these changes Apr 13, 2023
View reviewed changes
Copy link
Contributor

@kc13greiner kc13greiner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

pgayvallet
pgayvallet approved these changes Apr 18, 2023
View reviewed changes
Copy link
Contributor

@pgayvallet pgayvallet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Some minor nits can could be done later as follow ups

packages/core/saved-objects/core-saved-objects-server/src/saved_objects_index_pattern.ts
export const MAIN_SAVED_OBJECT_INDEX = '.kibana';
export const TASK_MANAGER_SAVED_OBJECT_INDEX = `${MAIN_SAVED_OBJECT_INDEX}_task_manager`;
export const SavedObjectsIndexPatterns = [MAIN_SAVED_OBJECT_INDEX, TASK_MANAGER_SAVED_OBJECT_INDEX];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 NITS:

  1. case inconsistency between the individual indices constants and the list one
  2. not a big fan of the SavedObjectsIndexPatterns naming. I think something like ALL_SAVED_OBJECT_INDICES would be more explicit

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can do this as a follow up, to avoid merge conflict nightmares with the rest of "feature" PRs.

@gsoldevila gsoldevila merged commit c7956c6 into elastic:dot-kibana-split Apr 18, 2023
gsoldevila added a commit that referenced this pull request Apr 25, 2023
@gsoldevila @pgayvallet
@cnasikas @kibanamachine @banderror
Split the .kibana saved objects index into multiple indices (#154888)
21351df 
## Description 

Fix #104081

This PR move some of the SO types from the `.kibana` index into the
following ones:
- `.kibana_alerting_cases`
- `.kibana_analytics`
- `.kibana_security_solution`
- `.kibana_ingest`

This split/reallocation will occur during the `8.8.0` Kibana upgrade
(*meaning: from any version older than `8.8.0` to any version greater or
equal to `8.8.0`*)

**This PR main changes are:**
- implement the changes required in the SO migration algorithm to
support this reallocation
- update the FTR tools (looking at you esArchiver) to support these new
indices
- update hardcoded references to `.kibana` and usage of the
`core.savedObjects.getKibanaIndex()` to use new APIs to target the
correct index/indices
- update FTR datasets, tests and utility accordingly 

## To reviewers

**Overall estimated risk of regressions: low**

But, still, please take the time to review changes in your code. The
parts of the production code that were the most impacted are the
telemetry collectors, as most of them were performing direct requests
against the `.kibana` index, so we had to adapt them. Most other
contributor-owned changes are in FTR tests and datasets.

If you think a type is misplaced (either we missed some types that
should be moved to a specific index, or some types were moved and
shouldn't have been) please tell us, and we'll fix the reallocation
either in this PR or in a follow-up.

## .Kibana split

The following new indices are introduced by this PR, with the following
SO types being moved to it. (any SO type not listed here will be staying
in its current index)

Note: The complete **_type => index_** breakdown is available in [this
spreadsheet](https://docs.google.com/spreadsheets/d/1b_MG_E_aBksZ4Vkd9cVayij1oBpdhvH4XC8NVlChiio/edit#gid=145920788).

#### `.kibana_alerting_cases`
- action
- action_task_params
- alert
- api_key_pending_invalidation
- cases
- cases-comments
- cases-configure
- cases-connector-mappings
- cases-telemetry
- cases-user-actions
- connector_token
- rules-settings
- maintenance-window

#### `.kibana_security_solution`
- csp-rule-template
- endpoint:user-artifact
- endpoint:user-artifact-manifest
- exception-list
- exception-list-agnostic
- osquery-manager-usage-metric
- osquery-pack
- osquery-pack-asset
- osquery-saved-query
- security-rule
- security-solution-signals-migration
- siem-detection-engine-rule-actions
- siem-ui-timeline
- siem-ui-timeline-note
- siem-ui-timeline-pinned-event

#### `.kibana_analytics`

- canvas-element
- canvas-workpad-template
- canvas-workpad
- dashboard
- graph-workspace
- index-pattern
- kql-telemetry
- lens
- lens-ui-telemetry
- map
- search
- search-session
- search-telemetry
- visualization

#### `.kibana_ingest`

- epm-packages
- epm-packages-assets
- fleet-fleet-server-host
- fleet-message-signing-keys
- fleet-preconfiguration-deletion-record
- fleet-proxy
- ingest_manager_settings
- ingest-agent-policies
- ingest-download-sources
- ingest-outputs
- ingest-package-policies

## Tasks / PRs

### Sub-PRs

**Implementation**
- 🟣 #154846
- 🟣 #154892
- 🟣 #154882
- 🟣 #154884
- 🟣 #155155

**Individual index split**
- 🟣 #154897
- 🟣 #155129
- 🟣 #155140
- 🟣 #155130

### Improvements / follow-ups 

- 👷🏼 Extract logic into
[runV2Migration](#154151 (comment))
@gsoldevila
- Make `getCurrentIndexTypesMap` resillient to intermittent failures
#154151 (comment)
- 🚧 Build a more structured
[MigratorSynchronizer](#154151 (comment))
- 🟣 #155035
- 🟣 #155116
- 🟣 #155366
## Reallocation tweaks

Tweaks to the reallocation can be done after the initial merge, as long
as it's done before the public release of 8.8

- `url` should get back to `.kibana` (see
[comment](#154888 (comment)))

## Release Note

For performance purposes, Kibana is now using more system indices to
store its internal data.

The following system indices will be created when upgrading to `8.8.0`:

- `.kibana_alerting_cases`
- `.kibana_analytics`
- `.kibana_security_solution`
- `.kibana_ingest`

---------

Co-authored-by: pgayvallet <[email protected]>
Co-authored-by: Christos Nasikas <[email protected]>
Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Georgii Gorbachev <[email protected]>
@rudolf rudolf added the Epic:ScaleMigrations Scale upgrade migrations to millions of saved objects label Jun 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kc13greiner kc13greiner kc13greiner approved these changes

@spong spong spong approved these changes

@pgayvallet pgayvallet pgayvallet approved these changes

@rudolf rudolf rudolf approved these changes

Assignees
No one assigned
Labels
backport:skip This commit does not require backporting enhancement New value added to drive a business result Epic:ScaleMigrations Scale upgrade migrations to millions of saved objects release_note:skip Skip the PR/issue when compiling release notes Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Team:Security Team focused on: Auth, Users, Roles, Spaces, Audit Logging, and more! Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. technical debt Improvement of the software architecture and operational architecture v8.8.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@gsoldevila @kibana-ci @elasticmachine @rudolf @pgayvallet @spong @kc13greiner