[alerting] investigate ability to handle some "NO DATA" conditions via a normal query

[pmuellr]

A customer described a scenario they wanted to alert on with the index threshold. They wanted to do a grouped query, and then identify instances that have NOT reported any data.

I don't think that's possible without knowing all the possible instances ahead of time - eg, via an alert parameter (an array of instance ids to look for). My thinking is that we're doing aggregations on the group term, and "missing instances" won't show up in any bucket, since they aren't there.

In conversation, it was mentioned that using min_doc_count: 0 could perhaps help with this, but I don't think it will given the previous paragraph. However, a non-grouped query using the count aggregation could use this to return 0 instead of not returning anything.

It's even possible it's doing this today, not real sure without doing a test on it. But seems like something worth investigating. Also not clear if we'd enable this as some kind of option, or just always arrange to return 0 for those. One of the problems with always returning 0 in that case is that it's different than the other aggregations which do not operate this way (and can't - they need actual values from the documents - count is only counting documents returned and doesn't need a value out of the index documents).

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[pmuellr]

@dgieselaar noted the following in a side conversation:

Just to clarify (maybe you got this and I’m misunderstanding your issue), if you add min_doc_count: 0 to a terms aggregation, it will return buckets with a doc_count of 0 for all terms recorded in the index, including for non-matching documents. Contrived example: if I have two documents in an index, one with service.name:opbeans-java, and one with service.name:opbeans-node, if I search for documents matching service.name:opbeans-node, and add a terms aggregation on service.name with min_doc_count: 0, I get two buckets, one for opbeans-node and one for opbeans-java . Screenshot to illustrate:

[ymao1]

@pmuellr Is this issue to investigate No Data conditions for the stack rules (es query, index threshold) or to investigate a framework level No Data detection?

[pmuellr]

From the context, I think this was just for the stack rules. I believe the most general issue we have open for "NO DATA" right now is #67296 - it's not clear if NO DATA should be an action group, a status, a ???. And there's some history, as some metrics rules can detect no data conditions and provide that indication in the alerts (somehow, can't remember the exact mechanism, I think it's a "no data" sort of action group though).