Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fleet] Remove the mapping for the nested istalled_kibana field #148174

Closed
Tracked by #174166
xcrzx opened this issue Dec 29, 2022 · 3 comments · Fixed by #148141
Closed
Tracked by #174166

[Fleet] Remove the mapping for the nested istalled_kibana field #148174

xcrzx opened this issue Dec 29, 2022 · 3 comments · Fixed by #148141
Assignees
@xcrzx
Labels
8.7 candidate bug Fixes for quality problems that affect the customer experience Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team:Fleet Team label for Observability Data Collection Fleet team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.7.0

Comments

@xcrzx
Copy link
Contributor

xcrzx commented Dec 29, 2022

Summary

It is not possible to install Fleet packages containing more than 10000 saved objects. An error is thrown due to the nested field limitations:

Error installing security_detection_engine 8.4.1: The number of nested documents has exceeded the allowed limit of 
[10000].
This limit can be set by changing the [index.mapping.nested_objects.limit] index level setting.

We have already discussed the issue here, and it looks like it could be easily overcome by adding enabled: false to the mappings for the installed_kibana field:

kibana/x-pack/plugins/fleet/server/saved_objects/index.ts

Lines 261 to 267 in ab8dd04

installed_kibana: {
type: 'nested',
properties: {
id: { type: 'keyword' },
type: { type: 'keyword' },
},
},

@xcrzx xcrzx added Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area 8.7 candidate labels Dec 29, 2022
@xcrzx xcrzx self-assigned this Dec 29, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@xcrzx xcrzx mentioned this issue Dec 29, 2022
Merged
@xcrzx xcrzx added Team:Detections and Resp Security Detection Response Team Team:Detection Rule Management Security Detection Rule Management Team labels Dec 29, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@banderror banderror mentioned this issue Dec 29, 2022
Closed
12 tasks
@banderror banderror changed the title [Security Solution] Remove the mapping for the nested istalled_kibana field [Fleet] Remove the mapping for the nested istalled_kibana field Dec 29, 2022
@banderror banderror added Team:Fleet Team label for Observability Data Collection Fleet team v8.7.0 labels Dec 29, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@banderror banderror added bug Fixes for quality problems that affect the customer experience impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. labels Dec 29, 2022
@banderror banderror linked a pull request Dec 29, 2022 that will close this issue
[Fleet] Fix crashes when installing a package with a lot of saved objects #148141
Merged
xcrzx added a commit that referenced this issue Jan 3, 2023
@xcrzx
[Fleet] Fix crashes when installing a package with a lot of saved obj…
ae60734 
…ects (#148141)

**Resolves: #147695,
#148174
**Related to: #145851,
#137420

## Summary

This PR improves the stability of the Fleet packages installation
process with many saved objects.

1. Changed mappings of the `installed_kibana` and `package_assets`
fields from `nested` to `object` with `enabled: false`. Values of those
fields were retrieved from `_source`, and no queries or aggregations
were performed against them. So the mappings were unused, while during
the installation of packages containing more than 10,000 saved objects,
an error was thrown due to the nested field limitations:

   ```
Error installing security_detection_engine 8.4.1: The number of nested
documents has exceeded the allowed limit of
   [10000].
This limit can be set by changing the
[index.mapping.nested_objects.limit] index level setting.
   ```
2. Improved the deletion of previous package assets by switching from
sending multiple `savedObjectsClient.delete` requests in parallel to a
single `savedObjectsClient.bulkDelete` request. Multiple parallel
requests were causing the Elasticsearch cluster to stop responding for
some time; see [this
ticket](#147695) for more info.

**Before**
![Screenshot 2022-12-28 at 11 09
35](https://user-images.githubusercontent.com/1938181/209816219-ade6dd0a-0d56-4acc-929e-b88571f0fe81.png)

**After**
![Screenshot 2022-12-28 at 13 56
44](https://user-images.githubusercontent.com/1938181/209816209-16c69922-4ae2-4589-9aa4-5a28050037f4.png)
@banderror banderror mentioned this issue Jan 5, 2023
Closed
@banderror banderror mentioned this issue Jan 3, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xcrzx xcrzx

Labels
8.7 candidate bug Fixes for quality problems that affect the customer experience Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team:Fleet Team label for Observability Data Collection Fleet team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.7.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Fleet] Fix crashes when installing a package with a lot of saved objects xcrzx/kibana
3 participants
@xcrzx @banderror @elasticmachine