[Security Solution][Detection Engine] adds alert suppression to ES|QL rule type

[vitaliidm]

Summary

• addresses https://github.com/elastic/security-team/issues/9203
• adds alert suppression for new terms rule type
• similarly to custom investigation fields list of available suppression fields:
  • shows only ES|QL fields returned in query for aggregating queries
  • shows ES|QL fields returned in query + index fields for non-aggregating queries. Since resulted alerts for this type of query, are enriched with source documents.

Demo

1. run esql rule w/o suppression
2. run esql rule w/ suppression per rule execution. Since ES|QL query is aggregating, no alerts suppressed on already agrregated field host.ip
3. run suppression on interval 20m
4. run suppression for custom ES|QL field which is the same as host.ip, hence same results
5. run suppression on interval 100m

_esql.suppression.mov

Limitations

Since suppressed alerts deduplication relies on alert timestamps, sorting of results other than @timestamp asc in ES|QL query may impact on number of suppressed alerts, when number of possible alerts more than max_signals.
This affects only non-aggregating queries, since suppression boundaries for these alerts set as rule execution time

Checklist

• Functional changes are hidden behind a feature flag

  Feature flag alertSuppressionForEsqlRuleEnabled

• Functional changes are covered with a test plan and automated tests.

  • https://github.com/elastic/security-team/pull/9389

• Stability of new and changed tests is verified using the Flaky Test Runner.

  • FTR(x100): https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5907
  • Cypress(x100): https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6011

• Comprehensive manual testing is done by two engineers: the PR author and one of the PR reviewers. Changes are tested in both ESS and Serverless.

• Mapping changes are accompanied by a technical design document. It can be a GitHub issue or an RFC explaining the changes. The design document is shared with and approved by the appropriate teams and individual stakeholders.

  Existing AlertSuppression schema field is used for ES|QL rule, the one that already used for Query, New terms and IM rules.

      alert_suppression:
        $ref: './common_attributes.schema.yaml#/components/schemas/AlertSuppression'

  where

      AlertSuppression:
        type: object
        properties:
          group_by:
            $ref: '#/components/schemas/AlertSuppressionGroupBy'
          duration:
            $ref: '#/components/schemas/AlertSuppressionDuration'
          missing_fields_strategy:
            $ref: '#/components/schemas/AlertSuppressionMissingFieldsStrategy'
        required:
          - group_by

• Functional changes are communicated to the Docs team. A ticket or PR is opened in https://github.com/elastic/security-docs. The following information is included: any feature flags used, affected environments (Serverless, ESS, or both).

  • [Request] add ES|QL rule alert suppression docs  security-docs#5156

[kibanamachine]

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#6011

[✅] Security Solution Detection Engine - Cypress: 100/100 tests passed.
[✅] [Serverless] Security Solution Detection Engine - Cypress: 100/100 tests passed.

see run history

[kibanamachine]

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#6029

[❌] Security Solution Detection Engine - Cypress: 74/100 tests passed.
[✅] [Serverless] Security Solution Detection Engine - Cypress: 100/100 tests passed.

see run history

[PhilippeOberti]

desk tested and code LGTM for the Threat Hunting Investigations team!

[nikitaindik]

Why not pass a mock alert_suppression object here and then check that it's rendered like we do for custom query rules?

Also, I think we need to add alert suppression properties in the test plan under "ESQL" here.

[vitaliidm]

added in 8702a48

[nikitaindik]

Encountered a weird rule preview behaviour while doing local testing. Sometimes rule preview displayed different alerts for me after clicking "Refresh". I made a screen recording once it started happening, but I couldn't reproduce it. Not sure if it's a bug and if it's related to rule suppression or not. Perhaps the screen recording might give @vitaliidm a hint on what might be going on.

rule_preview.mov

[nikitaindik]

Reviewed the code with focus on the Rule Management owned files. The changes LGTM. Left a couple comments.

[vitaliidm]

Encountered a weird rule preview behaviour while doing local testing. Sometimes rule preview displayed different alerts for me after clicking "Refresh". I made a screen recording once it started happening, but I couldn't reproduce it. Not sure if it's a bug and if it's related to rule suppression or not. Perhaps the screen recording might give @vitaliidm a hint on what might be going on.

Not enough information to figure out what's going on here. I wasn't able to reproduce
I suspect it might be related to how preview runs multiple rule executions.
Would need additional info this one:

• What's the rule configured schedule? Exported rule would be helpful.
• HAR file would be helpful, would allow to see what response in preview request + generated alerts
• Very suspicious, alerts genereated w/o original timestamp. I don't think I see that before.
• Anything in Kibana logs?
• What's the data you have in that index? Are there events w/o timestamp? Data coming from agent? Or just indexed manually and can be reproduced easily

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 83bf8dd

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	15.1MB	15.1MB	+555.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
securitySolution	83.5KB	83.5KB	+45.0B

History

• 💚 Build #210821 succeeded 65b6957
• 💔 Build #210788 failed 9125e5a
• 💔 Build #210737 failed def83f5
• 💛 Build #210507 was flaky eb7db73
• 💚 Build #210456 succeeded 02290ac
• 💔 Build #210449 failed 07188d9

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @vitaliidm