[APM] Improvements to transaction duration anomaly alert flow

[formgeist]

Summary

Related issue #72636 and PR #75719

I would like to suggest the following enhancements to the create flow for the new transaction duration anomaly alert.

Proposal

Allow user to create an anomaly alert from anywhere in the APM app

Since ML jobs are tied to the environment, I suggest that we should allow the user to create anomaly alerts from anywhere in the APM app. It's hard to manage to have to set up these alerts on a per service basis. We can still allow the user to define a specific service, and default to the selected service in the alert flyout. But there should be another option to create an alert.

Enable alert creation within the Anomaly detection settings view

Another way to enable the user to create an anomaly alert is to provide a quick option in the Anomaly detection jobs table that will invoke the flyout with the service environment pre-selected. That way the user can easily create the alert right after the job creation.

@elastic/apm-ui @nehaduggal @alex-fedotyev Thoughts on this UX?

[sorenlouv]

Another way to enable the user to create an anomaly alert is to provide a quick option in the Anomaly detection jobs table that will invoke the flyout with the service environment pre-selected

I love the quick option in the Anomaly Detection Settings view 👍 Will make it a much more seamless experience to setup an anomaly job and then create alerts for it.

Question: would it perhaps be better to make Service: All and Environment: <selected environment> readonly for this flow? Fewer options leads to a simpler UI and leads to fewer errors (both on the user side and developer side).
If the user wants to create alerts for another environment they can simply find the right environment in the list.
If they want to limit the alert to a single service they should go to that specific service and create the alert.

Having more constrained flows with fewer options will lead to less confusion I think.
(This is also why I really would like to see "Name", "Tags", "Check every" and "Notify every" hidden far away behind an "Advanced settings" toggle)

Overview of existing APM alerts
Since it'll be possible to create alerts from different places in APM, it can quickly become confusing whether an alert was already created for a particular service and/or environment. Should we show a call out at the top of the create dialog:

3 alerts already exists for this environment. Click here to see them.

That would make it much easier to verify which services/environments already have alerts and help the user not create duplicate alerts (sometimes they want that which is fine but at least we can call it out).

[ogupte]

This is related to #65981

[formgeist]

I converted my proposal to two separate issues after a conversation with @sqren. We have several additions and improvements in mind for the alerts, but I think the most pressing is allowing users to create anomaly alerts that are globally scoped as well.

#76361
#76365

[formgeist]

Question: would it perhaps be better to make Service: All and Environment: readonly for this flow? Fewer options leads to a simpler UI and leads to fewer errors (both on the user side and developer side).
If the user wants to create alerts for another environment they can simply find the right environment in the list.
If they want to limit the alert to a single service they should go to that specific service and create the alert.

I think it might feel a little counterintuitive that the user has to go out of the existing flyout and move to another page (global) to specify an "all"-scoped alert. I think if we limit the option in the service name condition to "All" or selected service, it makes the interaction less about specifying multiple other services or similar. We won't support that user flow.

Having more constrained flows with fewer options will lead to less confusion I think.
(This is also why I really would like to see "Name", "Tags", "Check every" and "Notify every" hidden far away behind an "Advanced settings" toggle)

Those options have already been proposed to the Alerting team, I will follow up with them to see what improvements they have considered for the alert flyouts.

Overview of existing APM alerts
Since it'll be possible to create alerts from different places in APM, it can quickly become confusing whether an alert was already created for a particular service and/or environment. Should we show a call out at the top of the create dialog:

3 alerts already exists for this environment. Click here to see them.

That would make it much easier to verify which services/environments already have alerts and help the user not create duplicate alerts (sometimes they want that which is fine but at least we can call it out).

I think that's a great idea, but I would like to expand to the be more about what Observability alerts do I have set up because I imagine in most common cases we're thinking of it as the user is defining alerts for logs, metrics and APM for the same "environment" or application (collection of services and infrastructure) and they want a unified view of what's configured. I suggest that we consider this some more when we think about having a list of alerts related to APM/Observability.

[sorenlouv]

I think it might feel a little counterintuitive that the user has to go out of the existing flyout and move to another page (global) to specify an "all"-scoped alert.

We are talking about the flyout that pops up on the anomaly detection settings page, right?
Afaict the service must be read-only and fixed to "All" (there's no service in context and there could be 1000s of services making the drop down extremely long if we were to show all of them). Wrt making environment readonly its just a matter of opening the flyout on the line below (not moving to another page).

[formgeist]

We are talking about the flyout that pops up on the anomaly detection settings page, right?

Ah, I must've missed this context from your response. Yes, with the anomaly detection settings view it makes sense to make the service name and environment read-only.