[Security Solution][Detection Engine] improves ES|QL investigation fields for detection rules #177746
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vitaliidm
changed the title
vitaliidm
added
release_note:enhancement
Team:Detections and Resp
|
/ci |
…thub.com/vitaliidm/kibana into detection-engine/esql-highlighted-fields
|
/ci |
|
/ci |
|
/ci |
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
|
Pinging @elastic/security-detection-engine (Team:Detection Engine) |
vitaliidm
changed the title
e40pud
approved these changes
Mar 6, 2024
| }; | ||
|
|
||
| export const useInvestigationFields: UseInvestigationFields = ({ | ||
| isEsqlRule, |
There was a problem hiding this comment.
Do we need isEsqlRule parameter? I guess if rule is not ESQL then esqlQuery will be undefined and that should cover it
There was a problem hiding this comment.
agree, can be simplified, removed it
Looks like it affect only preview flyout. Thanks for catching this, going to investigate |
💛 Build succeeded, but was flaky
Failed CI StepsTest Failures
Metrics [docs]Module Count
Async chunks
History
To update your PR or re-run it, just comment with: cc @vitaliidm |
kibanamachine
added
v8.14.0
backport:skip
7 tasks
6 tasks
vitaliidm
added a commit
that referenced
this pull request
May 20, 2024
… rule type (#180927) ## Summary - addresses elastic/security-team#9203 - adds alert suppression for new terms rule type - similarly to [custom investigation fields](#177746) list of available suppression fields: - shows only ES|QL fields returned in query for aggregating queries - shows ES|QL fields returned in query + index fields for non-aggregating queries. Since resulted alerts for this type of query, are enriched with source documents. ### Demo 1. run esql rule w/o suppression 2. run esql rule w/ suppression per rule execution. Since ES|QL query is aggregating, no alerts suppressed on already agrregated field `host.ip` 3. run suppression on interval 20m 4. run suppression for custom ES|QL field which is the same as `host.ip`, hence same results 5. run suppression on interval 100m https://github.com/elastic/kibana/assets/92328789/4bd8cf13-6e23-4842-b775-605c74ae0127 ### Limitations Since suppressed alerts deduplication relies on alert timestamps, sorting of results other than `@timestamp asc` in ES|QL query may impact on number of suppressed alerts, when number of possible alerts more than max_signals. This affects only non-aggregating queries, since suppression boundaries for these alerts set as rule execution time ### Checklist - [x] Functional changes are hidden behind a feature flag Feature flag `alertSuppressionForEsqlRuleEnabled` - [x] Functional changes are covered with a test plan and automated tests. - elastic/security-team#9389 - [x] Stability of new and changed tests is verified using the [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner). - FTR(x100): https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5907 - Cypress(x100): https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6011 - [x] Comprehensive manual testing is done by two engineers: the PR author and one of the PR reviewers. Changes are tested in both ESS and Serverless. - [x] Mapping changes are accompanied by a technical design document. It can be a GitHub issue or an RFC explaining the changes. The design document is shared with and approved by the appropriate teams and individual stakeholders. Existing AlertSuppression schema field is used for ES|QL rule, the one that already used for Query, New terms and IM rules. ```yml alert_suppression: $ref: './common_attributes.schema.yaml#/components/schemas/AlertSuppression' ``` where ```yml AlertSuppression: type: object properties: group_by: $ref: '#/components/schemas/AlertSuppressionGroupBy' duration: $ref: '#/components/schemas/AlertSuppressionDuration' missing_fields_strategy: $ref: '#/components/schemas/AlertSuppressionMissingFieldsStrategy' required: - group_by ``` - [x] Functional changes are communicated to the Docs team. A ticket or PR is opened in https://github.com/elastic/security-docs. The following information is included: any feature flags used, affected environments (Serverless, ESS, or both). - elastic/security-docs#5156 --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Nikita Indik <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Screen.Recording.2024-03-05.at.17.41.14.mov
Checklist
Unit or functional tests were updated or added to match the most common scenarios
Flaky Test Runner was used on any tests changed
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5365