[Security Solution][Detections] Implementation of RuleExecutionLogClient

[xcrzx]

Addresses: #106461, #106466

Summary

This PR adds rule execution log implementation build on top of rule_registry.

How to test this implementation

1. Enable experimental FF echo "xpack.securitySolution.enableExperimental: ['ruleRegistryEnabled']" >> ./config/kibana.dev.yml
2. Enable rule registry write echo "xpack.ruleRegistry.write.enabled: true" >> ./config/kibana.dev.yml
3. Create a reference rule by running ./x-pack/plugins/security_solution/server/lib/detection_engine/reference_rules/scripts/create_reference_rule_query.sh

It creates a rule that generates 10 alerts every minute or so. The created rule is not visible in our UI and is not accessible through most of our API endpoints. But, we can fetch rule execution logs using the setection_engine/rules/_find_statuses endpoint.

cd ./x-pack/plugins/security_solution/server/lib/detection_engine/scripts/
./find_rules_statuses_by_ids.sh [\"rule-id-here\"]

The output could look like this:

{
  "93ef7e00-d996-11eb-8993-0f3373136c03": {
    "current_status": {
      "alert_id": "93ef7e00-d996-11eb-8993-0f3373136c03",
      "status_date": "2021-07-01T15:39:58.020Z",
      "last_failure_at": "2021-07-01T15:39:58.020Z",
      "last_failure_message": "Response Error",
      "status": "failed",
      "gap": null,
      "bulk_create_time_durations": null,
      "search_after_time_durations": [
        "1385.66"
      ]
    },
    "failures": [
      {
        "alert_id": "93ef7e00-d996-11eb-8993-0f3373136c03",
        "status_date": "2021-07-01T15:39:58.020Z",
        "last_failure_at": "2021-07-01T15:39:58.020Z",
        "last_failure_message": "Response Error",
        "status": "failed",
        "gap": null,
        "bulk_create_time_durations": null,
        "search_after_time_durations": null
      }
    ]
  }
}

These are logs written by the RuleExecutionLogClient during the rule execution.

Design

• RuleExecutionLogClient - contains feature switch logic and "decides" which concrete client to use to read/write logs. It is supposed to be the only publicly available class, and all API users should interact with logs exclusively through it.
• Adapters - glue code needed for backward compatibility with the existing usage patterns. They are supposed to be sole integration points for the new log clients.
• RuleExecutionLogClient - turned-down implementation of the execution log, left here only for reference.

Integration Phase 1

Integration Phase 2

Integration Phase 3

TODO

• Test if the implementation could work on top of Kibana system indices
• Add a feature flag to disable write to the rule execution log
• Measure rule execution log performance in comparison to sidecar SOs

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: d319d63
• Storybooks Preview

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
lists	229	230	+1
securitySolution	2337	2338	+1
total			+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
lists	276.1KB	276.6KB	+508.0B
securitySolution	6.4MB	6.4MB	-3.0B
total			+505.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
securitySolution	208.2KB	209.2KB	+1.1KB

Unknown metric groups

API count

id	before	after	diff
ruleRegistry	60	82	+22
securitySolution	1298	1300	+2
total			+24

API count missing comments

id	before	after	diff
ruleRegistry	60	82	+22
securitySolution	1247	1249	+2
total			+24

Non-exported public API item count

id	before	after	diff
ruleRegistry	9	11	+2
securitySolution	27	28	+1
total			+3

History

• 💔 Build #142275 failed 2e5ea8c84ba24549c23327474017f083f33339ba
• 💚 Build #142153 succeeded cb401fa185b0ae30c13af4c7ce140d7bd3048de0
• 💛 Build #142138 was flaky 9d097cb3c7933a8c1e381eb8d8d92afe521c33bf
• 💔 Build #141940 failed 645cad08db51a41fd366473a33eabb557c16aa19
• 💔 Build #141931 failed e3eb4e3612ee55312fc2534b9d433eae8ff09558

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @xcrzx

[banderror]

LGTM to be merged!

I left some comments, but none of them are anything significant to block this PR. Still, let's quickly go through them and decide, which of them could be addressed in a follow-up PR.

🚀 🎉

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.