[Lens] Custom rank agg for top values / pick hidden metrics

[flash1293]

It's not possible to rank top values by a calculation like differences / cumulative sum / moving average / formula due to mathematical and technical limitations. Right now, in these situations Lens automatically falls back to rank alphabetically.

However, in some cases (elastic/integrations#3115 (comment)) this problematic because it leaves no way to rank in a way to show the "most important" series (by some definition). It's possible for table visualizations because they allow do define a metric, then hide it for the chart, but not for other chart types (e.g. xy).

There are a few options I can think of to solve this use case:

• Like agg based: Allow the user to define a ranking metric inline in the top values dimension - we would need to come up with an additional UI for that (e.g. rendering the list of metrics in a combobox instead of a list group and so on). Pros: Full flexibility, Cons: Complex to implement (new UI, bad fit for architecture)
• Like TSVB: Allow the user to pick metrics which are not visible dimensions but used by them (e.g. the components of a formula or the undelrying metric of a "differences" calculation). Often, the metrics used in there are the ones defining "most important". Pros: Easy to implement (basically just adjusting filtering logic for pickable metrics to rank by), Cons: Not 100% flexible (what if the relevant metric isn't used in a visible dimension? Hacky workaround for the limitation: Use formula and add and remove the metric in question so it shows up - is this enough?)
• Like Lens table: Allow to hide metrics from the chart - define them as regular dimensions so they become available for ranking top values, but then prevent them from being displayed. Pros: Full flexibility, low effort, Cons: Doesn't work for pie/gauge/metrics as they only allow to specify a single metric (introducing another "hidden" dimension group might solve the issue but might be confusing for users and much higher effort)

[elasticmachine]

Pinging @elastic/kibana-vis-editors @elastic/kibana-vis-editors-external (Team:VisEditors)

[drewdaemon]

You've nicely stated the problem and some potential routes 👍 . I just came across the agg-based approach in my 8.3 testing and thought it's flexibility and directness was pretty nice.

[ghudgins]

Summary of conversation on 15 june 2022 -

We like first / pasted below option the best. While there are similar patterns of selecting a custom metric in "moving average" we'll give this a design pass

Like agg based: Allow the user to define a ranking metric inline in the top values dimension - we would need to come up with an additional UI for that (e.g. rendering the list of metrics in a combobox instead of a list group and so on). Pros: Full flexibility, Cons: Complex to implement (new UI, bad fit for architecture)

[shahzad31]

we also want this in the user experience and synthetics data

essentially the browser with most traffic when breakdown is done. Is it possible to prioritise this for 8.4?

[flash1293]

@shahzad31 as discussed offline, it's possible to rank top values by a metric if it's used in a y axis dimension and it's outside of formula. Does this cover your use case?

[justinkambic]

As an FYI this PR is blocked on this. We're using breakdowns for a chart we want to migrate to a Lens embeddable, but we need to show the top 10 items in the legend, not based on alphabetized ranking but on count.

[mbondyra]

Started working on this one - so far more as a POC as the designs are not ready yet. I am not sure though if we want to allow users to sort only by basic metrics or also by differences / cumulative sum / moving average / formula?

[ghudgins]

I think we need to stick to basic metrics that we can actually fetch with vs. those calculated in browser