[Security Solution] Extend event_log plugin with functionality required for Rule Execution Log

[peluja1012]

Parent ticket: #118324

Summary

The Security Solution needs to store rule execution events and metrics to power various rule monitoring workflows. Based on all the previous discussions, we decided that Kibana event_log plugin is a good fit for this purpose. We will need to extend it to fully support Security Solution use cases.

We anticipate that we will need to be able to execute the following types of queries to event log:

1. Write execution logs from rule type executors.
2. Fetch N last log entries for 1 rule.
   • Context: Rule Details Page.
   • Rule id is known beforehand.
3. Fetch N aggregations (of rule execution events or metrics) for M rules.
   • Context: Rule Monitoring table, fetching N*M values for the whole table in a single request to Elasticsearch.
   • M rule ids are known beforehand.
4. Fetch N rules sorted by K metrics (indexing time, query time, etc). Some kind of aggregations with sorting.
   • Context: Rule Management/Monitoring table, sorting by columns.
   • Rule ids are not known beforehand.
5. Fetch K current metrics across all rules. E.g., average indexing time during the last execution across all rules, etc.
   • Context: Rule Management/Monitoring table.
   • Rule ids are not known beforehand.
6. Fetch the last event (status change) across all rules (similar to num 5).
   • Context: Rule Management/Monitoring table.
   • Rule ids are not known beforehand.
7. Fetch Nth percentile of a current metric across all rules (similar to num 5).
   • Context: Rule Management/Monitoring table.
   • Rule ids are not known beforehand.
8. Fetch the number of rules matching criteria (for example, rules with status=warning etc).
   • Context: Rule Management/Monitoring table.
   • Rule ids are not known beforehand.

This list is not full or final, we might not need some of these queries, and we might need others. It depends on the final requirements and UX wireframes. What we can see from these types of queries, though, is:

• Most of the queries will require the ability to define free-formed Elasticsearch DSL queries (with filters, aggregations, limiting _source, sorting by multiple fields, etc) on the Security Solution side.
• Some of the queries are rule-agnostic (calculate something across all rules). Current RBAC model of event_log require developers to provide references to at least 1 saved object (in our case it would be at least 1 rule) when fetching from the log, which makes it impossible to execute such queries.

We think that event_log plugin should be a good fit for these rule monitoring needs but it will need to be extended to provide additional functionality.

Requirements

TBD. We will define technical requirements and a proposal for software design in a dedicated RFC for extending event_log plugin with RBAC and flexible search API. This section will be updated.

Draft:

• [DONE] It should be possible to extend event_log schema with standard ECS and custom fields needed for implementing Rule Execution Log.
• We should be able to filter events by solution (consumer), kibana space id, rule instance id, etc.
• When querying the log via event_log, we should be able to execute free-formed Elasticsearch DSL queries. The plugin should provide a flexible .search() method ideally with the same signature that ElasticsearchClient exposes.
• event_log should support RBAC based on Alerting feature privileges. We should be able to grant those privileges via our existing "Security" feature privileges:
  

Sub-tasks

• Extend Event Log schema
  • Add standard ECS fields and custom fields required for Rule Execution Log: status, metrics, etc (see the RFC) @xcrzx
  • Add technical fields needed for RBAC and such (space_ids, consumers, ruleTypeId, etc)
• Add flexible .search() method for executing arbitrary Elasticsearch queries
  • Existing PR that adds support for aggregations [Event Log] Added a new event log API, which allows to fetch events summary aggregations for saved objects by Ids. #91731
• Add support for RBAC based on Alerting feature privileges
  • Review event_log RBAC RFC
  • Implement RBAC support

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[gmmorris]

Adding Feature:RAC to disambiguate this issue from the backlog of Feature:EventLog issues owned by the Alerting team.

[banderror]

Outdated