Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Introduce an endpoint to use for the detection rules table filters/data #137428

Closed
Tracked by #174167
xcrzx opened this issue Jul 28, 2022 · 4 comments · Fixed by #146826
Closed
Tracked by #174167
Labels
Feature:Rule Management Security Solution Detection Rule Management area performance Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. technical debt Improvement of the software architecture and operational architecture v8.7.0

Comments

@xcrzx
Copy link
Contributor

xcrzx commented Jul 28, 2022

Epic: https://github.com/elastic/security-team/issues/1974 (internal)

Summary

Currently we use the detection_engine/rules/prepackages/_status endpoint to populate the Elastic rules and Custom rules counters on the rules management page:

Screenshot 2022-07-28 at 11 57 02

We re-fetch those counters once in 5 minutes and on every rule state mutation. However, the detection_engine/rules/prepackages/_status was designed for different purposes. Using it for re-fetching the rules table counters is computationally expensive and will become even more expensive with advances in rule customization epic.

We need to extract the logic that calculates the number of custom and prebuilt rules to a separate endpoint and use it to populate the rules table values. Consider introducing an endpoint that returns all data required for the table filters: the number of custom and prebuilt rules, rule tags, etc.

@xcrzx xcrzx added technical debt Improvement of the software architecture and operational architecture Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team 8.5 candidate labels Jul 28, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@banderror banderror added Feature:Rule Management Security Solution Detection Rule Management area performance labels Jul 28, 2022
@maximpn maximpn self-assigned this Nov 18, 2022
@maximpn maximpn linked a pull request Dec 1, 2022 that will close this issue
2 tasks
maximpn added a commit that referenced this issue Dec 20, 2022
…146826)

**Addresses:** #137428

## Summary

Adds a new internal lightweight endpoint to fetch rules related information like the number of installed prebuilt rules, the number of custom rules and etc.

## Details

This PR adds a quite simple and lightweight endpoint for fetching rules related information which is
- the number of installed prebuilt rules
- the number of custom rules
- tags

UI has been updated accordingly. The result of the endpoint are mostly used in the rules table filter but not limited to.

**_The added endpoint doesn't implement full aggregation for fetching rule numbers so it's planned to be done in the following PR._**

### Comparison

The following screenshots from the browser's network tab demonstrate that the new endpoint is faster which is good since it's intended to be updated executed relatively often whenever the rules are updated.

Prebuilt rules endpoint which was used for fetching rules related information
<img width="942" alt="Screenshot 2022-12-04 at 21 50 50" src="https://user-images.githubusercontent.com/3775283/205514981-ce51dd4e-faed-4999-b770-5737e0c118d4.png">

The new endpoint
![image](https://user-images.githubusercontent.com/3775283/205887909-39b1f18d-5181-4a13-b16c-0291080011da.png)

### Checklist

- [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
@banderror
Copy link
Contributor

Once #125659 is addressed we will need to get back to the filters endpoint and optimize its performance by using aggregation queries.

@banderror
Copy link
Contributor

Re: #137428 (comment)

Aggregations are already used for fetching aggregated rule tags. What we can do is to use aggs for fetching the rest of the data needed for filters. We can do it as part of a separate ticket: #166364

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Rule Management Security Solution Detection Rule Management area performance Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. technical debt Improvement of the software architecture and operational architecture v8.7.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants