[Epic] Release Istio integration to Beta

[ChrsMark]

Context
Istio is one of the most popular service meshes, its position in the CNCF landscape has helped its adoption and growing ecosystem thus becoming one of the layers that users need to observe to aim for full-stack observability.

User outcome
The goal of this work is to increase the breadth of technologies we support the integration outcomes (ingestion and curated data views with dashboards) by enabling users to collect and ingest Istio data

• Elastic cloud operators who are using elastic agent should be able to add the Istio integration
• The data collected should be metrics, logs and any other metadata available.
• Elastic cloud users should be able to toggle on/off which data they want to collect
• Elastic cloud users who install the Istio integration should have an out of the box dashboard to provide them with correlation relevant to the Istio data collected.

Tasks to be completed:

• Support collection of logs

  • Implement log collection based on default Istio logging provider (envoy)
  • Implement log collection based on Json output format

• Implement Metrics collection (relevant info)

• Istio installation guide in here

• Define some default dashboards as assets to be installed with integration. Minimal samples and this can be enhanced even later

Relevant backlog stories for reference:
Create a integrations to collect Istio logs.
Related to elastic/beats#15505

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ChrsMark]

One thing to consider here is to add testing notes to https://github.com/elastic/observability-dev/tree/main/docs/dc/integrations/testing about Istio. Unfortunately we don't have so far but it would be useful to document the process of bringing up an Istio env locally on top of k8s etc.

[gsantoro]

Some questions about migrating the Istio dashboards from Metricbeat to Integrations:

1. Should we implement most of the dashboard in TSDB (like already done before) or should we move them to Lens? I am not sure what's the difference. So far I have only noticed that Lens doesn't have a Data formatter for Duration (that convert a number to time).
2. There are two Dashboard called respectively "Overview" and "Traffic". I would like to rename them to Istiod and Proxy instead since those are the data source names that we use in the metrics. Thought?
3. the Metricbeat version has a Tag cloud for "Destination app", I would like to add one for the "Source app" as well.

4. I would like to add 2 controllers for source_app and destination_app to filter the traffic based on a sublist of source/destination apps.
   

[gsantoro]

5. Should we add a graph visualization (similarly to what we had in metricbeat)?
   This is the one from metricbeat (taken from https://www.elastic.co/blog/istio-monitoring-with-elastic-observability) where we connect source_app to destination_app and response_code
   

Instead of the previous one, I would suggest to graph source/destination workload instead. Each source/destination app can have multiple versions (like api versions). I believe that this other metric label gives more useful details for the user.

[gsantoro]

6. I have generated some more graphs. What do you guys think about them? Can we embed them with the integration?

There is this concept of drilldown that would enable us to click on one of those graph nodes to navigate to another kibana page (eg. discovery or dashboards). Should we investigate it for this iteration?

[gsantoro]

7. Istiod dashboards
   

[ChrsMark]

Some questions about migrating the Istio dashboards from Metricbeat to Integrations:

1. Should we implement most of the dashboard in TSDB (like already done before) or should we move them to Lens? I am not sure what's the difference. So far I have only noticed that Lens doesn't have a Data formatter for Duration (that convert a number to time).

In general, it was suggested to use Lens. Not sure if there is any limitation here. You can either use Lens directly or leave it for another iteration and ship it with the TSDB ones.

2. There are two Dashboard called respectively "Overview" and "Traffic". I would like to rename them to Istiod and Proxy instead since those are the data source names that we use in the metrics. Thought?

The idea of "Overview" and "Traffic" comes from the fact the Istiod metrics provide an overview of the istio service mesh while for Proxy you can see the Traffic on the cluster/mesh.
Personally I like it as is but you can of course change it if @mlunadia from product's perspective agrees on this.

3. the Metricbeat version has a Tag cloud for "Destination app", I would like to add one for the "Source app" as well.

+1 Feel free to do it.

4. I would like to add 2 controllers for source_app and destination_app to filter the traffic based on a sublist of source/destination apps.

That would be helpful.

In general, I would suggest to feel free to add or remove anything you find valid. Then we can review the differences maybe. Adding stuff can only be for good I would say. This one applies for the "graphs" questions too, if it's doable to embed them into the dashboards then you can try it.

[gsantoro]

1. Ok. I'll try to migrate them all to Lens. I'll take notes of limitations and write them down. We are going to have a mix of lens and TSDB
2. Ok. I don't have a strong opinion on this. I'll revert the naming back to Overview and Traffic
3. Great I'll keep the new tag cloud as well
4. Cool I'll keep the controllers
5. About the graphs, I can't embed them into the dashboard. Do we want to still use them? can we put them into the integrations? Mostly I am asking since in that blog post you created some.

[gsantoro]

Limitations of using Lens for Istio dashboard:

• Gauge visualisation is not available in Lens (as it is). There is a technical preview of a gauge but it doesn't provide the same functionalities
• Line chart with percentile in TSDB allow to have multiple percentiles in the same chart (eg. 25, 50, 75, 95, 99 percentiles). We would need 5 different charts to represent 1 in the TSDB version.
• Some metrics are in milliseconds and are converted to seconds by the TSDB visualisation. Lens dashboard doesn't support "Duration" format
• Mixing and matching Lens and TSDB visualisations will give a non homogeneous look to the entire dashboard, so for this iteration I'll stick to TSDB for the entire dashboard.

[mlunadia]

1. Let's use what gives the best result for the user.
2. If you think it makes more sense for the user to have the name as the source, I support this based on simplicity
3. +1 add it
4. Given that you have most context with Istio I think things like this are a useful change and see no problem on exploring them.

[mlunadia]

Given the comments above let's for TSDB over lens for these dashboards

[gizas]

1. In general indeed Lens is the suggested way to go.
   I attach two more docs for references:
   https://docs.google.com/document/d/1uyyFGx6xA5Kvl8c-ZdvXdvBGrHTylxU9F69TGqfzdmw/edit
   https://docs.google.com/document/d/1_7ZYOO5KsSG4J3ovdlB4obBF5i0FQfsiXrRtI3wWFpg/edit#heading=h.gi2lpayry05p

For Gauges I would say to give an extra effort with Lens if possible. For chart the changes are big so ok leave them as is.

(Although in second doc it says: Gauges (Lens gauges are still experimental and they might not be a good replacement for commonly used TSVB gauges yet))

2. You can enhance the markdown with sample text to explain what is what. I think some text can be helpful whatever title you put

3. +1 add it

4. Mind to add the latest controllers (not the old ones which are deprecated)

5. For graph I would say remove it. Although beautiful dont see the actual value to users

Ingress, Egress Traffic will be valuable to this dashbaords?
In many dashboards we have Title and same subtitle in the bottom. Keep only one
And also (I know is painfull) keep a consistency in colours. For eg all lines to be green or blue

[gsantoro]

Mostly for @gizas but left here for posterity

1. about the gauges. This is the comparison for the same metric

The top one is from TSDB, the second one is from Lens. No need for comments 😄

2. What would you suggest as text for the dashboards? Do you have an example?
3. ok
4. Yes, I am using the latest controllers.
5. There is no way to embed the graph in the integration anyway. So I won't include them
6. I am not sure if there are metrics about Ingress and Egress. I'll have a look
7. I assume you mean about the Gauge having both titles on top and at the bottom. I managed to find a way just now to remove the bottom title. It looks a bit hacky though. You need to provide a Custom Label = " " (that's a blank space). I'll update them all
8. About the consistency of the lines. For all the charts with a single line they are blue. All the other ones are using the default palette since they need to use many colours for the many different labels (eg. source or destination labels).

Thanks for the feedback

[gizas]

Something like below with title "Istiod" and second "Proxy" and short description what we see, or any importan information for how you should read the dashboards

[ChrsMark]

Hey @gsantoro , #4253 looks quite good at the moment. We could merge it when approved and Dashboards can always be tuned/improved in follow-up PRs. This would help reduce the scope of the PRs and iterate on smaller chunks of work. In any case, I have left my review at #4253 (review) already, it's up to you to decide when it's considered ready :)