[FEATURE] Enhancing Monitor Configs to support custom data sources.

[eirsep]

Is your feature request related to a problem?

Today alerting implements multiple flavors of Monitor Runners to offer different alerting capabilities, such as Query Level Monitor, Bucket Level Monitors and Document Level Monitors. The execution workflow for these monitors are very coupled with their direct functional needs. There is also a tight coupling between the configuration and execution logic. The data sources for both reads and writes are hard coded, with assumption that they would always remain same from the Alerting purview. There is currently no way to store alerts or findings for different users/stakeholders of alerting plugin in different indices.

What solution would you like?
Enhancing Monitor Config

Enhancing monitor configs provides more users with more control over various configurations in each monitor and options to decide input and output sources. These are optional params which users of the plugins (such as Security Analytics) can use to injects their own configurations such as source data, query index, findings index etc. helping with separation of concerns in data layer for Alerting Plugin.

The new addition in the config will be stored as a part of the monitor document in the index itself.

Monitor configs allow users of alerting to define optional configurable params to override the following defaults:

Data Sources

We envision Monitors to have multiple data sources, in addition to the current source index. It can read from either indices with user data. Alerting could also read data from findings index which other monitors are writing into. Users can pass source type to be findings instead of the default indices. Alerting would then parse findings index documents to pick up the list of docs to be queried on and treat that as source of the monitor.

Monitors would also be able to consume results of another monitor by consuming findings index documents as source data (TODO)
Source Type Values: FINDINGS, INDICES
Default Source type:

Query Index

Allows users to pass an index where queries will be stored. This override is essential in enabling users of Alerting plugin to build and store rules which can be loaded on demand from rule index.
The current behavior today can be modeled into a default config which can be overridden in the create monitor request for each monitor.

Findings Index
Allows users to create custom set of indices of findings and its history. Users could harness this option to store their alerts

Alerts Index

Allows users to create custom set of indices for alerts, alerts history. Users could harness this option to store their alerts in a separate place.