Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fleet] Fix crashes when installing a package with a lot of saved objects #148141

Merged
merged 1 commit into from
Jan 3, 2023

Conversation

xcrzx
Copy link
Contributor

@xcrzx xcrzx commented Dec 28, 2022

Resolves: #147695, #148174
Related to: #145851, #137420

Summary

This PR improves the stability of the Fleet packages installation process with many saved objects.

  1. Changed mappings of the installed_kibana and package_assets fields from nested to object with enabled: false. Values of those fields were retrieved from _source, and no queries or aggregations were performed against them. So the mappings were unused, while during the installation of packages containing more than 10,000 saved objects, an error was thrown due to the nested field limitations:

    Error installing security_detection_engine 8.4.1: The number of nested documents has exceeded the allowed limit of 
    [10000].
    This limit can be set by changing the [index.mapping.nested_objects.limit] index level setting.
    
  2. Improved the deletion of previous package assets by switching from sending multiple savedObjectsClient.delete requests in parallel to a single savedObjectsClient.bulkDelete request. Multiple parallel requests were causing the Elasticsearch cluster to stop responding for some time; see this ticket for more info.

Before
Screenshot 2022-12-28 at 11 09 35

After
Screenshot 2022-12-28 at 13 56 44

@xcrzx xcrzx force-pushed the fleet-bulk-delete branch from 36cc2bf to 5d3d24b Compare December 28, 2022 11:52
@xcrzx xcrzx self-assigned this Dec 28, 2022
@xcrzx xcrzx added v8.7.0 Team:Fleet Team label for Observability Data Collection Fleet team release_note:skip Skip the PR/issue when compiling release notes performance labels Dec 28, 2022
@xcrzx xcrzx marked this pull request as ready for review December 28, 2022 13:04
@xcrzx xcrzx requested review from a team as code owners December 28, 2022 13:04
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@xcrzx xcrzx changed the title Use SavedObjectsClient.bulkDelete to delete package assets [Fleet] Fix crashes when installing a package with a lot of saved objects Dec 28, 2022
Copy link
Member

@kpollich kpollich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code changes look small in scope, but with big impact. Really appreciate the improvements here, and the before/after traces you provided. LGTM 🚀

@banderror banderror linked an issue Dec 29, 2022 that may be closed by this pull request
@xcrzx xcrzx force-pushed the fleet-bulk-delete branch from 5d3d24b to 31c46ec Compare January 3, 2023 09:30
@kibana-ci
Copy link
Collaborator

kibana-ci commented Jan 3, 2023

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] Jest Tests #7 / AllCasesListGeneric Actions Row actions should disable row actions when bulk selecting all cases
  • [job] [logs] Jest Tests #7 / AllCasesListGeneric Actions Row actions should disable row actions when selecting a case
  • [job] [logs] Jest Tests #7 / AllCasesListGeneric Actions Row actions update the status of a case: critical

Metrics [docs]

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/saved-objects-service.html#_mappings

id before after diff
epm-packages 28 22 -6

History

  • 💚 Build #97227 succeeded 5d3d24b0644c9d2e1e158edfbe8d14e2e2ac89b7
  • 💔 Build #97207 failed 36cc2bfb528045785f17a54827a2110ebef0c3ce

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @xcrzx

@xcrzx xcrzx merged commit ae60734 into elastic:main Jan 3, 2023
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Jan 3, 2023
@xcrzx xcrzx deleted the fleet-bulk-delete branch January 3, 2023 11:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting performance release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v8.7.0
Projects
None yet
6 participants