[Lens] Use Lens embeddable only for data fetching

[stratoula]

Describe the feature:

Lens has the ability to do various calculations on the provided data (e.g. with formula) and offers some caching capabilities that can be reused by other consumers. There are cases that I don't want to render a chart (or I want to use a different renderer) but I want Lens to be used for the data fetching.

One way to do it is to expose an api that can be used (instead of using the Lens embedable). Another approach is to use the Lens embeddable but without rendering anything. We should explore the capabilities here and decide how we would like to offer this feature.

With ES|QL this might not be very important as ES|QL is more powerful than Lens formulas and provides a lot of other calculations.

[elasticmachine]

Pinging @elastic/kibana-visualizations @elastic/kibana-visualizations-external (Team:Visualizations)

[smith]

Adding this to Infra UI's dependencies board. We have a lot of lens charts and are interacting with multiple APIs that sometimes pose challenges with Lens.

This isn't blocking anything specific right now but could help increase the number of use cases where we can consider using Lens.

cc @roshan-elastic

[roshan-elastic]

Cool @smith - do you want me to push this with Ninoslav?

If so, just need a bit more context as to the value this brings...maybe there are a couple of sample use cases which could bring it to life for Infra?

[smith]

@roshan I think most of the current use cases are in the APM app, so as long as somebody from Observability is pushing this (and what's on #159295) and Nino knows about it we're good.

[roshan-elastic]

Cool @smith - do you have a link to that wider 'asks' board you mentioned the other week? I'll start ensuring they go on there as part of my workflow.

[gbamparop]

@roshan-elastic you might be referring to this board https://github.com/orgs/elastic/projects/1196/.

[roshan-elastic]

Thanks @gbamparop !

@ninoslavmiskovic - Do you think we should put this issue in the O11y platform dependencies board?

Or, should we keep it only in our Infra UI dependencies board that we manage?

[smith]

It should be on both boards.

[smith]

It was already on the infra board. I added it to the observability board as well under Lens with APM UI as the requester. (cc @gbamparop @sqren)

If APM needs this we'll let them take it since we don't have a specific request for them. If APM decides they don't need it either, our team can revisit it if and when we need something specific.

[sorenlouv]

I don't think we have any use cases for this.
@kpatticha You've recently looked into Lens and the gaps that need to be bridge for us to migrate. Is this one of them?

[kpatticha]

We don't have a specific use case, so it's not blocking us. However, I listed it in the ticket

[thomasneirynck]

Just to keep track of this: a concrete downstream effect of this is that ES|QL mode in Discover will submit queries twice

[stratoula]

On the opposite side there are cases that the users want to use their own data and use Lens only for rendering

I think we should prioritize this as it would make the ESQL mode more performant in Discover.

[stratoula]

I am going to create another issue for Lens only for rendering as I think it is more important than using Lens for data fetching.

The reasons is that with ES|QL you can easily do all these calculations you could do with Lens formulas so I think that is better to use ESQL rather Lens for these use cases.

[stratoula]

Here is the issue for Lens only for rendering #167631

[sorenlouv]

Hey @stratoula

I just recalled that in APM we actually have a use case. I'm not sure it will be solved with #167631 so putting my use case here instead.

--

The use case: Most visualisations in APM app can be rendered using several different types of documents:

• Sampled transactions: These docs are ingested in the traces-apm* index. They have every dimension captured but this also makes them very big, they have low storage efficiency and expensive to query. So these are heavily sampled which affects their accuracy and we only use these as a last resort.
• Metric transactions: these docs are ingested in the metrics-apm* index and are pre-aggregated transactions. This means that a single metric doc may represent 100s or 1000s of transaction docs. They have high storage efficiency, fast to query and accurate (not sampled). Transaction metrics are available in 3 granularities: 1 minute, 10 minutes and 60 minutes. A 60 minute metric doc contains transactions captured over a 60 minute period. We select the resolution based on the time range, eg for a 1 hour time range we use 1m metrics. For 24 hours we use 10m, for 7 days we use 60m metrics.
• Other metric docs: there are also other types of metric documents that I won't dive into right now (service metrics, instance metrics etc).

So the question is: would it be possible for us to create an embeddable where we can embed this logic and automatically choose the right documents to render with, but still allow the user to modify the visualisation in Lens?

[stratoula]

would it be possible for us to create an embeddable where we can embed this logic and automatically choose the right documents to render with, but still allow the user to modify the visualisation in Lens?

@sqren I don't think that Lens should have this logic. In my mind apm should fetch the data as they want, send them to Lens, Lens should not run any query but receive this data as input, apply the configuration you want, render the embeddable. This is related to #167631.

Opening these charts in Lens editor is something we need to explore on the 167631 issue.

[sorenlouv]

@stratoula What I'm trying to understand, is how users will be able to modify the visualisation once inside Lens if we do all the data fetching outside Lens. Are those two requirements reconcilable?
I'm not suggesting Lens should contain the knowledge about transactions and metrics - this is very APM specific.

[stratoula]

@sqren in my mind Lens should work as it works with ES|QL now.

It fetches the datatable so I have a table with columns/rows. The user can change colors/chart configuration and select columns but not run aggregations.

Something like that:

So in that mode Lens only works as a rendering tool offering chart configuration but not data manipulation (meaning the generation of another datatable)

If we want the users to be able to change the datatable (run aggregations etc), then this will be tricky and we need to think about it.

[stratoula]

This is possible now for ES|QL #171081