[Alerting] Allow rule types to specify custom timeout values

[ymao1]

The task manager has a default timeout value of 5m for all tasks. This can be overridden in the task definition, but alerting currently does not take advantage of this ability. This issue is to allow rule type producers to specify a custom timeout value when registering a rule type and for the alerting framework to use these custom timeouts when registering the task.

We can also think about whether we want to set a framework level default timeout value for all rule types that is longer than the task manager default of 5m. Perhaps this should be configurable as well.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[gmmorris]

@jasonrhodes / @oatkiller / @chrisronline / @devin / @madirey - as prolific rule type implementers I'm curious to hear your thoughts on this idea.

If you had the option of setting a custom timeout for your rule types (or even individual rules) would you use this ability?
What would you base this custom timeout on?

[MikePaquette]

Thanks @ymao1 for pointing out this ticket!
@gmmorris

If you had the option of setting a custom timeout for your rule types (or even individual rules) would you use this ability?

Yes. As users adopt frozen tier data, it is reasonable to expect that they'll have a few rules that query that data tier, for example looking for matches of new threat indicators against archived events, and they might be OK if that rule takes longer to run, say 15-minutes, so we would not want their rule to be cancelled/fail after 5 minutes.

Today, we have one security rule_type (Indicator Match) which is more resource-intensive, actually issuing multiple queries as part of its execution, and thus routinely takes much longer to execute than other rule_types, so a custom timeout for the rule_type would be useful. In addition, allowing per-individual rule timeout values would also be useful, so "regular" custom query rules or EQL rules could be pointed at searchable snapshot data and run successfully even if their query takes longer than 5-min.

What would you base this custom timeout on?

From the user perspective, we don't want to burden the rule_author with details about the underlying alerting framework or the task manager. We wonder if the timeout could take into consideration the configured rule interval, for example, maybe it could be 5-minutes OR the configured rule interval, whichever is greater, or something like that?

cc: @oatkiller @jethr0null

[ymao1]

@MikePaquette Opened an issue to investigate allowing individual rule instances to specify their own timeout values: #113823