[RAC] Alerts as Data Bootstrapping Index Creation

[spong]

This issue is for finalizing the architecture/implementation for bootstrapping the Alerts as Data index creation process.

Existing implementations include the bootstrapping of the .siem-signals index, and the bootstrapping of the .kibana-event-log index.

.siem-signals

Most all logic lives in:
kibana/x-pack/plugins/security_solution/server/lib/detection_engine/index/

With routes and index/ilm definitions residing in:
kibana/x-pack/plugins/security_solution/server/lib/detection_engine/routes/index/

.kibana-event-log

Exists as a dedicated plugin named eventLog and can be found in: kibana/x-pack/plugins/event_log/
(Fantastic docs @pmuellr :)

Reference docs:

• Schema Considerations (internal)
• Unified Architecture (internal)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[pmuellr]

FWIW, the bootstrapping of the current event log takes place here - https://github.com/elastic/kibana/blob/master/x-pack/plugins/event_log/server/es/init.ts

There's one thing we haven't done, keep forgetting to do this - because we're always writing to an alias, we should use the require_alias option when indexing the documents (going to open an issue for this now for event log). This will ensure that we don't accidentally create an index with the alias name, when we index the documents. That shouldn't happen, since we are creating the alias before doing any indexing, but in the past this happened when the bootstrapping code was not robust enough.

[mikecote]

There's been a few ideas bounced around for using the event log for alerts as data (now or later). I wanted to lay out the issues that the event log has today for future reference.

1. The event log uses a single index. This won't work with the requirements of creating an index per space, rule type.
2. The event log is designed to be secured behind saved objects. Users who want direct access to the data indices to build visualizations would get access to all the data captured by the event log, which they shouldn't see.
3. The event log is append-only. I've seen requirements for updating alert documents to support workflows like acknowledging, change status, assign, etc. Moving away from append-only may prevent leveraging data streams.
4. The event log deletes data older than 90 days by default. This configuration is global for all events captured. I'm assuming alert data will want its own customizable data retention policy?

cc @dgieselaar @sqren @pmuellr

[sorenlouv]

This has probably been discussed elsewhere already but: Would it make sense to write alert events to multiple destinations: the event log, and another append-only index (eg alerting-log-*)? This way the alerting framework is still in control of persisting alerts, which ensures consistency across consumers.

1. The event log uses a single index

We can decide to write to an index per rule type, eg .alerting-log-observability-apm-transaction-latency

2. [...] users [...] would get access to all the data captured by the event log, which they shouldn't see.

If we write to an index per alert type it will be possible to give granular permissions:
.alerting-log-* vs .alerting-log-observability-* vs .alerting-log-observability-apm-*

3. The event log is append-only. I've seen requirements for updating alert documents to support workflows like acknowledging, change status, assign, etc.

All those actions could be events instead of mutations on existing events

[mikecote]

This has probably been discussed elsewhere already but: Would it make sense to write alert events to multiple destinations: the event log, and another append-only index (eg alerting-log-*)? This way the alerting framework is still in control of persisting alerts, which ensures consistency across consumers.

IT, depends. There have been talks about mutating this data as well, would each index need to update in this scenario?

All those actions could be events instead of mutations on existing events

Would this work for filtering and sorting? Like list alerts that have recovered, are assigned, are open, etc.

[mikecote]

We can decide to write to an index per rule type, eg .alerting-log-observability-apm-transaction-latency

If we write to an index per alert type it will be possible to give granular permissions:
.alerting-log-* vs .alerting-log-observability-* vs .alerting-log-observability-apm-*

I agree that both of those points are needed and confirms that the event log can't be used to solve that problem today.

[dgieselaar]

Would this work for filtering and sorting? Like list alerts that have recovered, are assigned, are open, etc.

Yes, if the whole state is captured on the last event, which in my head it would be. But it might be a little too premature to dive into all the technical details of this specific approach?

[dgieselaar]

I agree that both of those points are needed and confirms that the event log can't be used to solve that problem today.

agree 👍 (although the event log can be namespaced too, but I don't see a compelling reason right now to store everything there).

[mikecote]

Yes, if the whole state is captured on the last event, which in my head it would be.

Ah, I didn't think of that, nice!

[peluja1012]

Closing in favor of #98912