Cloud context in kibana alerting

[Kushmaro]

When kibana is in cloud, it would be highly beneficial to have some cloud context available for the alert actions (like the ones available for the alert itself)

Using "production deployment" (for deployment being monitored)
Using "monitoring deployment" (for deployment doing the monitoring)

Most notably:

• Production Cloud Deployment ID
• Production Cloud Deployment name (already available as "context.clusterName")
• Production Cloud Region + Provider
• Production Organization/Account Id
• Cloud UI URL (can vary in ESS/ESSP/ECE)

Additional nice-to-haves:

• Full cloud deployment aliased URL

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[mikecote]

Relates to #67660 for providing the cloud plugin such capability.

[gmmorris]

This should be available in any cloud Kibana, as it's in the kibana.yml, but we don't have any "cloud specific" variables at the moment, so we need to verify this works correctly.

Also worth noting - this will not work if a customer has a mixed environment (cloud + on-prem), which even though we discourage, isn't officially prohibited. In such a situation actions would result in different output depending on the Kibana running them.

[gmmorris]

Relates to #67660 for providing the cloud plugin such capability.

Are these related or actually identical? 🤔

[mikecote]

In theory, #67660 would be for the alerting team, and this issue would be for the cloud team if we decide to go such an approach.

[Kushmaro]

There's an important distinction to be made on the "deployment Id" and "region" field -
If a customer is using a central monitoring deployment, the context for the alert should be for the deployment firing the alert (and not for the monitoring deployment doing the monitoring itself), that's why I'm not sure the "config_settings" hold here, as they are for the "monitoring deployment" but not for the "production deployment" being monitored.

just wanted to explicitly call that out @gmmorris
I've edited the issue to better reflect this.

[gmmorris]

Thanks @Kushmaro , I don't know much about how these things are configured.... would this be specific to the monitoring deployment or the rule?
Could a monitoring deployment monitor multiple other deployments? Or would all the rules in a single monitoring deployment all monitor the same deployment?
If it's the latter, I assume we're still talking about something that's passed in via kibana.yml?

[Kushmaro]

Could a monitoring deployment monitor multiple other deployments?

ding ding ding :) yep, that's exactly how it works.
So just like context.clusterName passes the name of the cluster being monitored (called "Production cluster" in stack monitoring terms) the deployment ID's showing in the alerts, should be those of the deployment being monitored (sent over somehow to the monitoring cluster)

Hope this makes sense, if not, I'd be happy to jump on a zoom.
/cc @ravikesarwani who can also help drive this forward.

[ravikesarwani]

We recommend users to send their observability (logs and metrics) data to a separate monitoring cluster. In “production environments” most users will consolidate many production clusters observability data to a single monitoring cluster. Stack monitoring recommends and supports this scenario.

Having said that there are times when users still enable self monitoring (because of cost, simplicity or other reasons) wherein production and monitoring cluster is the same.

I think it would be ideal to have the cloud context available for both (production and monitoring) deployments so users can add in the Action messages or be able to use it in other ways (reporting etc). The reason I say we need both is because depending upon the type of alert the first inclination for the users (or internal teams) maybe to verify the monitoring data of the related entities before jumping on the actual production cluster/cloud console to start making any changes.

[gmmorris]

Could a monitoring deployment monitor multiple other deployments?

ding ding ding :) yep, that's exactly how it works.
So just like context.clusterName passes the name of the cluster being monitored (called "Production cluster" in stack monitoring terms) the deployment ID's showing in the alerts, should be those of the deployment being monitored (sent over somehow to the monitoring cluster)

I'm sure this is something missing in my knowledge, but I don't understand how the Alerting framework could possibly do this automatically, given the framework isn't querying anything.

I suggest @arisonl, @ravikesarwani and @Kushmaro have a chat and figure out how these pieces are meant to fit together from a product perspective, because I get the feeling there's a piece missing in the middle between the framework part and the Cloud part of this ER. 🤔

[mikecote]

I'm going to close this issue. We think it will be best to start adding these variables in the rule type (as context variables) first. Rule types will be aware of the "production deployment" as the framework isn't aware.