[R&D] Data flow stopped rule

[miltonhultgren]

As a follow up to the investigation done here, we've identified a need to be able to define a rule that checks if data stops coming in from an expected source.

In the Stack Monitoring case we have regular SDHs where data stops coming in for some reason (config changes, cluster upgrades, outages or cluster overload), it would be great if we had a flexible way to create rules around such situations.

I'm not certain about what kind of granularity we should look at (pure document rate into an index/data stream with awareness of sending frequency, splitting by metricset, etc.).

Ideally, this rule would be made in a way that it can be used no matter what the source of data is, as long as we expect it to be coming into Elasticsearch we should alert if it stops (with some config for how long an accepted delay might be).

[elasticmachine]

Pinging @elastic/actionable-observability (Team: Actionable Observability)

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[pmuellr]

We've been talking about NO_DATA indications since when alerting started, but never came up with how we would actually do this. I think the rule executors would need to return an indication of this, since the framework can't figure it out. Should it be a new action group, like Recovered, which is provided by the framework? I think that was the direction I was thinking of going on that. Some discussion here: #51099

I'm not sure how long it would take us to do this; also not sure how it would interact with the current "NO DATA" action group already used by at least metric threshold. I'd think we'd want to make sure we could migrate from that existing one, to the new one, or just "make it work" with the existing one (even better).

[katrin-freihofner]

https://github.com/elastic/actionable-observability/issues/80