[Lens] optimize duplicate metric operations

[drewdaemon]

Summary

Part of #135265

Testing

Create a visualization and add a dimension with this formula

median(bytes) + median(bytes) +

sum(bytes) + sum(bytes) +

max(hour_of_day) + max(hour_of_day) +

average(bytes) + average(bytes) +

standard_deviation(bytes) + standard_deviation(bytes) +

min(machine.ram, shift='1h') + min(machine.ram, shift='1h') +

min(machine.ram, shift='2h') + min(machine.ram, shift='2h')

Check the Elasticsearch request in the inspector. There should only be 7 aggregations, not 14.

Then try this formula

sum(bytes) + sum(bytes) +

sum(bytes, kql='geo.dest: "GA" ') + sum(bytes, kql='geo.dest: "GA" ') +

sum(bytes, kql='geo.dest: "AL" ') + sum(bytes, kql='geo.dest: "AL" ') 

+ sum(bytes, lucene='geo.dest: "AL" ') + 
sum(bytes, lucene='geo.dest: "AL" ') + 

sum(bytes, kql='geo.dest: "AL" ', reducedTimeRange='1m') + sum(bytes, kql='geo.dest: "AL" ', reducedTimeRange='1m')

Check the Elasticsearch request in the inspector. There should only be 5 aggregations, not 10.

Checklist

Delete any items that are not applicable to this PR.

• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/kibana-vis-editors @elastic/kibana-vis-editors-external (Team:VisEditors)

[dej611]

There's something strange happening somewhere here as I've noticed this weird behaviour:

• using your formula I have an error about the standard_deviation operation missing:

• if I remove the standard_deviation then the formula works:

• but also using standard_deviation only it works:

Cannot replicate the same behaviour in main, so there's probably some string manipulation here which breaks it.

[drewdaemon]

@dej611 good find! 😬 I'll look into this.

[drewdaemon]

@dej611 I'm having trouble reproducing that issue. I copied your formula, but it appears to work. Could you share any more details?

The error must originate here, but I haven't figured out how to get it to throw

kibana/src/plugins/data/common/search/aggs/metrics/std_deviation.ts

Line 120 in 0a3e40b

return bucket[agg.id].std_deviation;

[flash1293]

Works pretty well, left a comment about a further opportunity to optimize but that can be split out. I also don't know how Marco caused this error, but it would be nice to reproduce. @dej611 if you can still cause it can be share your config? Maybe it's about a certain combination with bucket aggs?

[drewdaemon]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 26485c3

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
data	2429	2431	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
lens	1.2MB	1.2MB	+1.5KB

Unknown metric groups

API count

id	before	after	diff
data	3132	3134	+2

History

• 💚 Build #72447 succeeded 0a3e40b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[flash1293]

This has the same problem the percentile optimization had - if a column is referenced for sorting by top values, it's failing because the column won't exist. In this case Filter 1 and Filter 2 are both a sum of bytes filtered the same way.

Ranking by Filter 2 works fine byt ranking by Filter 1 breaks because that column got optimized away.

This is a constructed example, but the same thing can happen if there is a regular metric dimension used for ranking plus a formula dimension which is internally using the same metric on the same chart.

[drewdaemon]

Closing in favor of a more holistic PR: #140859

[dej611]

Works pretty well, left a comment about a further opportunity to optimize but that can be split out. I also don't know how Marco caused this error, but it would be nice to reproduce. @dej611 if you can still cause it can be share your config? Maybe it's about a certain combination with bucket aggs?

I managed to reproduce it also in main the same issue, so it's a general bug. Here's my configuration:

If I remove the date_histogram then it works ok.

Edit: just found out that the time range picked is important to reproduce the bug: set the time picker to come custom date range with no data and the error will appear.