[APM] HTTP Status Code in Throughput Chart

[LucaWintergerst]

Describe the feature:

7.14 used to show the http code in the throughput charts. In 7.15 those are displayed nowhere.

Imagine a user getting a lot of 403 or 404 responses. That's definitely something important to be investigated, so we should display this somewhere.
Especially combined with the fact that 4xx codes are not considered an Error by Elastic APM, users will loose the ability to explore response codes completely when upgrading to 7.15

At the very least a user should have the option of changing the chart from the default to the split version (possibly losing the comparison while doing so)

Describe a specific use case for the feature:
7.15:

7.14 and earlier:

[jozefkoscak]

Similarly in my case.
I use apm for the application to log the processing of envets that may end in success or failure.
Seeing this data in a graph is very useful for me.
You do a very good job, and you add great features in each update, but I lost one useful feature in this update.

[Twiddlle]

I miss that functionality too.

[MartinKolarik]

I'm also missing this breakdown after the update. Furthermore, it was previously possible to filter for the status codes using transaction.result: "HTTP 4xx" which is set even on transactions with sampled: false but in the latest version the UI seems to exclude unsampled transactions from the charts, so even this manual filtering doesn't work anymore to get the throughput by result type.

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[sorenlouv]

Thanks @LucaWintergerst for opening this.
Breakdown by http status code was removed with the introduction of the comparison feature to avoid too much ui clutter. I think however that having the ability to see breakdown by status code (and possibly other dimensions) could be very useful, so I'll bring this back to the team for discussion.

Btw. When I say "Comparison feature", this is what I mean:

[alex-fedotyev]

Let me play "devil's advocate" for a moment.

Today we show throughput and failure rates (which maps to 5xx rates in general).
Those are displayed on different charts though, but because of that it is possible to compare each of those numbers to historical individually.

I wonder if that failed transactions trend is not useful, readable, discoverable or actionable?

(I recognize that 4xx are not displayed today, which is something to consider for sure).

[MartinKolarik]

In our case, I'm typically more interested in current 2xx/3xx/4xx/5xx breakdown (yes, even 3xx) than any historical trend. I understand different uses may have different priorities but find removing the breakdown completely rather unfortunate.

[formgeist]

@alex-fedotyev let me know how you want to proceed on this issue?

[sorenlouv]

I wonder if that failed transactions trend is not useful, readable, discoverable or actionable?

Problem with failed transactions is that it only includes 5xx (by default at least). Even if we changed it to include 4xx (which probably won't happen) it would not be possible to distinguish 4xx and 5xx from each other.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[MartinKolarik]

This is still very much missing, while I find myself turning off the historical data quite often.

[sorenlouv]

@boriskirov Breakdown by http status was a useful feature that we dropped when time comparisons was introduced. For 8.3 we are adding "Expected bounds" as an option to comparisons.

WDYT about adding "HTTP status" to the same dropdown? Selecting that would breakdown the latency, throughput and failed transaction rate charts by http status.

The comparison dropdown starts to feel a little overloaded because breakdown is not the same as comparison. But the semantics might not be too important. It's probably more important that we re-introduce a useful feature.

[sorenlouv]

@boriskirov @chrisdistasio This has come up several times. Before 7.14 we showed throughput broken down by transaction.result. We stopped doing that when we introduced comparisons. I suggest we bring it back in one shape or another. WDYT?

[LucaWintergerst]

This might be too much to ask for, but I'm wondering if it's worth it to consider a similar approach to Discover+Lens which allows for arbitrary breakdowns in 8.7.
Or perhaps something similar to what we're doing in the "explore data" UI as part of User experience, which allows for breakdowns of a predefined set of dimensions/fields

That way we could satisfy many user requests at once. Comparing response codes is obviously one if them, but then we'd for example be able to draw a chart comparing all different version of an app, or compare the performance for each host, pod or whatever it might be

[sorenlouv]

This might be too much to ask for, but I'm wondering if it's worth it to consider a similar approach to Discover+Lens which allows for arbitrary breakdowns in 8.7.

I've actually been toying with the idea of something generic. Aka the ability to choose any dimension to break down by.
I haven't seen the Discover+Lens approach. Do you have a link to some mockups?

Fr 8.7 we are working on service version comparisons with is a related concept but not exactly the same. Not in the first version but eventually these comparisons will make it possible to choose a custom dimension, and then compare different values of said dimension. So the user could pick transaction.result and then compare 2xx, 3xx, 4xx and 5xx. The downside is that the user would have to pick the values manually which is a very tedious workflow compared to what we used to have.

[LucaWintergerst]

@sqren you can see a GIF of it here: #143117

[sorenlouv]

@sqren you can see a GIF of it here: #143117

That looks very much like what I had in mind. Afaict it breaks down by the top 4 values. Do you think this should be configurable directly in the UI so the user can choose to see more values? For transaction.result there could be more than 4 different http codes.

@boriskirov @chrisdistasio WDYT?

[LucaWintergerst]

ideally yes, but the alternative could also be a "Edit in Lens" button if the Charts are Lens-compatible