[System] Replace TSVB gauges in system metrics and host overview dashboards #4975

drewdaemon · 2023-01-11T23:24:35Z

What does this PR do?

System and host metric overview dashboards now use Lens metrics instead of TSVB gauges.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Related issues

Screenshots

Host overview

Before

After

System overview

Before

After

elasticmachine · 2023-01-12T00:10:50Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-06-27T19:34:48.862+0000
Duration: 14 min 27 sec

Steps errors

Expand to view the steps failures

`Test integration: system`

Took 1 min 42 sec . View more details here
Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

`Google Storage Download`

Took 0 min 0 sec . View more details here

`Google Storage Download`

Took 0 min 0 sec . View more details here

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-01-12T00:10:59Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`3/3`)	💚
Files	100.0% (`4/4`)	💚
Classes	100.0% (`4/4`)	💚
Methods	60.759% (`48/79`)	👍
Lines	99.527% (`2736/2749`)	👍
Conditionals	100.0% (`0/0`)	💚

drewdaemon · 2023-01-18T18:17:24Z

failure may relate to #4950

…pdate-metric-overview-dashboards

leehinman

Overall looks good, just some questions on the queries, especially "NOT system.network.name : l*"

leehinman · 2023-01-31T15:24:14Z

packages/system/kibana/dashboard/system-79ffd6e0-faa0-11e6-947f-177f697178b8.json

@@ -8,7 +8,7 @@
                "highlightAll": true,
                "query": {
                    "language": "kuery",
-                    "query": ""
+                    "query": "host.name:\"docker-fleet-agent\""


Is this supposed to be here? I don't think this dashboard is limited to docker-fleet-agent

@gvnmagni : do you have any comment on this feedback? Could this be something that snuck in while we were trying things out?

honestly, I don't know. When we updated these visualizations we tried to replicate what was happening with the previous version of them and given my lack of competences on how data is structured behind these chart I had to intensively rely for help on Joe Reuter, who I believe originally made those. So sometimes I just followed instructions and I can't actually remember if these queries came from him or this is just an error (which would be weird, I have been very careful in not creating anything that could be unexpected).

Same happens for the next comments, the ones related to the "NOT system.network.name" query. Is it possible that is was made to avoid having dirty data of some sort? What does happen to the values if we remove these queries?

Makes sense, @gvnmagni ! Don't worry, I will evaluate the queries here and address this feedback. Thanks for your help. 🙏

leehinman · 2023-01-31T15:26:09Z

packages/system/kibana/dashboard/system-79ffd6e0-faa0-11e6-947f-177f697178b8.json

+                            "internalReferences": [],
+                            "query": {
+                                "language": "kuery",
+                                "query": "NOT system.network.name : l*"


what is this query for?

leehinman · 2023-01-31T15:26:41Z

packages/system/kibana/dashboard/system-79ffd6e0-faa0-11e6-947f-177f697178b8.json

                                "language": "kuery",
-                                "query": "data_stream.dataset : \"system.network\""
+                                "query": "NOT system.network.name : l*"


what is this query for?

leehinman · 2023-01-31T15:27:04Z

packages/system/kibana/dashboard/system-79ffd6e0-faa0-11e6-947f-177f697178b8.json

+                            "internalReferences": [],
+                            "query": {
+                                "language": "kuery",
+                                "query": "NOT system.network.name : l*"


and this query?

leehinman · 2023-01-31T15:27:20Z

packages/system/kibana/dashboard/system-79ffd6e0-faa0-11e6-947f-177f697178b8.json

+                            "internalReferences": [],
+                            "query": {
+                                "language": "kuery",
+                                "query": "NOT system.network.name : l*"


leehinman · 2023-01-31T15:30:06Z

packages/system/kibana/dashboard/system-Metrics-system-overview.json

                                "language": "kuery",
-                                "query": "data_stream.dataset : \"system.memory\""


do we still want this query?

leehinman · 2023-01-31T15:30:33Z

packages/system/kibana/dashboard/system-Metrics-system-overview.json

                                "language": "kuery",
-                                "query": "data_stream.dataset : \"system.cpu\""
+                                "query": ""


do we still want this query?

It is key that we prefilter on dataset. Same above.

drewdaemon · 2023-02-09T14:01:55Z

@joshdover this PR currently changes nothing about how these metrics are computed. They will still suffer from #1437.

I would like to use this as an opportunity to choose one of the more consistent strategies I mentioned here. That will reduce user confusion in the long run.

How do I get an answer as to which strategy makes sense?

botelastic · 2023-03-11T15:01:17Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

botelastic · 2023-04-10T15:58:33Z

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!

…pdate-metric-overview-dashboards

botelastic · 2023-07-27T19:50:16Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

botelastic · 2023-08-26T20:14:18Z

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!

ruflin · 2023-08-28T08:55:03Z

Reopening as I'm not sure if these changes should still make it in. @joshdover

botelastic · 2023-09-27T09:14:20Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

drewdaemon · 2023-09-27T20:51:13Z

Closing in favor of #6743

use lens metric instead of TSVB gauge

0e1b914

drewdaemon added the enhancement New feature or request label Jan 11, 2023

drewdaemon changed the title ~~use lens metric instead of TSVB gauge~~ Enhancements for system metrics and host overview dashboards Jan 11, 2023

drewdaemon changed the title ~~Enhancements for system metrics and host overview dashboards~~ Replace TSVB gauges in system metrics and host overview dashboards Jan 11, 2023

drewdaemon added 2 commits January 11, 2023 17:54

update changelog and manifest

ee5d4a9

update directions on host overview

2f21649

drewdaemon mentioned this pull request Jan 11, 2023

Make system integration the gold standard for Kibana best practices #4868

Open

11 tasks

elastic deleted a comment from elasticmachine Jan 12, 2023

andrewkroh added the Integration:system System label Jan 18, 2023

andrewkroh changed the title ~~Replace TSVB gauges in system metrics and host overview dashboards~~ [System] Replace TSVB gauges in system metrics and host overview dashboards Jan 18, 2023

Merge branch 'main' of https://github.com/elastic/integrations into u…

a63c693

…pdate-metric-overview-dashboards

drewdaemon marked this pull request as ready for review January 23, 2023 14:26

drewdaemon requested review from a team as code owners January 23, 2023 14:26

drewdaemon requested review from faec and leehinman January 23, 2023 14:26

drewdaemon mentioned this pull request Jan 30, 2023

[System] Issue with short time frame (5min) #1437

Closed

leehinman requested changes Jan 31, 2023

View reviewed changes

botelastic bot added the Stalled label Mar 11, 2023

botelastic bot closed this Apr 10, 2023

drewdaemon reopened this Jun 27, 2023

drewdaemon requested a review from a team as a code owner June 27, 2023 17:17

botelastic bot removed the Stalled label Jun 27, 2023

drewdaemon marked this pull request as draft June 27, 2023 17:17

Merge branch 'main' of https://github.com/elastic/integrations into u…

c88f60b

…pdate-metric-overview-dashboards

botelastic bot added the Stalled label Jul 27, 2023

botelastic bot closed this Aug 26, 2023

ruflin reopened this Aug 28, 2023

botelastic bot removed the Stalled label Aug 28, 2023

botelastic bot added the Stalled label Sep 27, 2023

botelastic bot removed the Stalled label Sep 27, 2023

drewdaemon closed this Sep 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[System] Replace TSVB gauges in system metrics and host overview dashboards #4975

[System] Replace TSVB gauges in system metrics and host overview dashboards #4975

drewdaemon commented Jan 11, 2023 •

edited

Loading

elasticmachine commented Jan 12, 2023 •

edited

Loading

Build stats

`Test integration: system`

`Google Storage Download`

`Google Storage Download`

elasticmachine commented Jan 12, 2023 •

edited

Loading

drewdaemon commented Jan 18, 2023

leehinman left a comment

leehinman Jan 31, 2023

drewdaemon Jan 31, 2023

gvnmagni Feb 1, 2023

drewdaemon Feb 1, 2023

leehinman Jan 31, 2023

leehinman Jan 31, 2023

leehinman Jan 31, 2023

leehinman Jan 31, 2023

leehinman Jan 31, 2023

leehinman Jan 31, 2023

ruflin Feb 1, 2023

drewdaemon commented Feb 9, 2023

botelastic bot commented Mar 11, 2023

botelastic bot commented Apr 10, 2023

botelastic bot commented Jul 27, 2023

botelastic bot commented Aug 26, 2023

ruflin commented Aug 28, 2023

botelastic bot commented Sep 27, 2023

drewdaemon commented Sep 27, 2023

		"language": "kuery",
		"query": "data_stream.dataset : \"system.memory\""

[System] Replace TSVB gauges in system metrics and host overview dashboards #4975

[System] Replace TSVB gauges in system metrics and host overview dashboards #4975

Conversation

drewdaemon commented Jan 11, 2023 • edited Loading

What does this PR do?

Checklist

Related issues

Screenshots

Host overview

Before

After

System overview

Before

After

elasticmachine commented Jan 12, 2023 • edited Loading

💔 Build Failed

Build stats

Steps errors

Test integration: system

Google Storage Download

Google Storage Download

🤖 GitHub comments

elasticmachine commented Jan 12, 2023 • edited Loading

🌐 Coverage report

drewdaemon commented Jan 18, 2023

leehinman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drewdaemon commented Feb 9, 2023

botelastic bot commented Mar 11, 2023

botelastic bot commented Apr 10, 2023

botelastic bot commented Jul 27, 2023

botelastic bot commented Aug 26, 2023

ruflin commented Aug 28, 2023

botelastic bot commented Sep 27, 2023

drewdaemon commented Sep 27, 2023

drewdaemon commented Jan 11, 2023 •

edited

Loading

elasticmachine commented Jan 12, 2023 •

edited

Loading

`Test integration: system`

`Google Storage Download`

`Google Storage Download`

elasticmachine commented Jan 12, 2023 •

edited

Loading