[Fleet] Help users find integration health dashboards #153848

joshdover · 2023-03-28T09:50:45Z

Goals

Help users:

Identify data volume by integration and agent to being able to spot noisy integrations or specific agents producing more or less data than expected
Help find ingestion and processing errors preventing collected data from being fully processed and indexed

High level proposal

In elastic/integrations#5335 new dashboards were added to the "Elastic Agent" package that is installed on every cluster using Fleet. These are not easy to find and we should provide appropriate affordances within the Fleet and Integration UIs to these dashboards.

Below are my recommendations on where to add links to each dashboard, please add any suggestions on improvements we can make to these ideas. I'm not an expert designer 😄

Ingest overview dashboard

This is a very high-level dashboard that currently shows data volume by integration and agent and acts an entry point to drill down into the other dashboards. We could add a link from the main Fleet landing page by adding an "Ingest Overview" link next to the "Agent activity" button.

Agent info dashboard

This should probably have a similar link as Ingest Overview on the Fleet UI to "Agent Overview". If it's too noisy to have so many buttons, maybe we just leave the Ingest Overview one and let users find the agent by the in-dashboard nav.

Integrations overview dashboard

This dashboard is useful for looking at a specific integration. I think we should link to a filtered version of this view from the "data streams" tab. We can add a "View integration metrics" link to the "actions" submenu for row.

This could also be linked from the Integrations Details page for each integration.

Implementation

A couple of functional requirements:

There should not be a full-page refresh with the Elastic logo waiting bar when navigating to the dashboard
We should use stable APIs for linking to dashboards - specifically the "app locator" feature. See as an example: https://github.com/elastic/kibana/pull/149721/files#diff-17e1fcea36e0edf158dfce2100a18d24929ce72f105922ca38b0854fa8ff4f35R35

elasticmachine · 2023-03-28T09:50:49Z

Pinging @elastic/fleet (Team:Fleet)

joshdover · 2023-03-28T09:50:57Z

cc @P1llus

joshdover · 2023-03-28T09:52:52Z

cc @kpollich - would be good to get your thought on this one since you've been focused on exposing more agent metrics in the UI

jen-huang · 2023-03-28T19:14:28Z

There might be an opportunity here to add an API/service to easily search for an integration's dashboard based on name, cc @kpollich.

criamico · 2023-04-12T14:37:12Z

I'm working on this and I have a basic version working (a button using the locator API) however I have some questions:

Do you know if it's possible to check in advance if a dashboard exists? I'll try to see if it's possible to hide the links when the dashboards don't exist.
What do you think of adding a single link on top of the agent list page that on click opens a popover containing the links? This would avoid cluttering the page with too many buttons. I made a quick "mockup" here:

@kpollich @joshdover

hop-dev · 2023-04-12T14:40:13Z

@criamico do we have an opportunity to fix this known issue while in the area?

#146781

criamico · 2023-04-12T15:04:23Z

@hop-dev mmm I'll take a look and see if it's a quick one. Otherwise we can pull it to this sprint or the next.

joshdover · 2023-04-12T16:09:19Z

Do you know if it's possible to check in advance if a dashboard exists? I'll try to see if it's possible to hide the links when the dashboards don't exist.

We can do a check similar to the one in this PR: https://github.com/elastic/kibana/pull/149721/files#diff-17e1fcea36e0edf158dfce2100a18d24929ce72f105922ca38b0854fa8ff4f35R35

What do you think of adding a single link on top of the agent list page that on click opens a popover containing the links? This would avoid cluttering the page with too many buttons. I made a quick "mockup" here:

This could work, but I also wonder if we can leverage the whitespace on the left side of the screen so these metric dashboards are a bit separated from agent actions.

criamico · 2023-04-12T16:25:52Z

We can do a check similar to the one in this PR

Thanks, I'll take a look, I'm a bit new to how the dashboards work :)

I also wonder if we can leverage the whitespace on the left side of the screen so these metric dashboards are a bit separated from agent actions.

I agree on moving the button to the left, it makes it more visible to the user.

jen-huang · 2023-04-12T20:58:33Z

I also wonder if we can leverage the whitespace on the left side of the screen so these metric dashboards are a bit separated from agent actions.

I agree on moving the button to the left, it makes it more visible to the user.

++ to moving it to the left, but I would also leave the links separated to avoid an extra menu click since there's plenty of room after moving it.

criamico · 2023-04-13T09:36:45Z

That's how it looks with two buttons on the left:

criamico · 2023-04-13T09:53:17Z

We can do a check similar to the one in this PR: https://github.com/elastic/kibana/pull/149721/files#diff-17e1fcea36e0edf158dfce2100a18d24929ce72f105922ca38b0854fa8ff4f35R35

@joshdover
It seems that the Kibana savedObjects client is now deprecated (see also #149098). if I understand correctly every plugin should have its own server side API, not sure if we already have one @kpollich

kpollich · 2023-04-13T11:57:57Z

@criamico - Do we need client-side SO here to resolve the dashboard ID's? If so maybe we can add the dashboards as a meta property on the /agents list API response body? We'd be able to resolve the ID's server side then and return them in that response that drives most of this page.

If the ID's are predictable, though, maybe we can hard code similar to Josh's example above? Happy to take a look at a draft PR as well if that helps.

criamico · 2023-04-13T12:23:18Z

@kpollich Hardcoding them just works and in fact I think it's what we've done so far, an example is here. I'll have a draft PR with the basic functionalities down and we can discuss there.

Closes #153848 ## Summary - Adds two link buttons on top of agent list page to access "Ingest overview" and "Agent Info" dashboards <img width="1444" alt="Screenshot 2023-04-13 at 15 22 53" src="https://user-images.githubusercontent.com/16084106/231772174-00c00a8e-62f1-43ea-a935-bc12f56f3e50.png"> The links are built using the new URL service [locator](https://github.com/elastic/kibana/blob/e80abe810837eeeff7fdcd594c6f8950590b49cd/x-pack/plugins/fleet/public/hooks/use_locator.ts#L14) and the [getRedirectLink](https://github.com/elastic/kibana/blob/main/src/plugins/share/README.mdx#using-locator-of-another-app) method; - Refactoring existing instances of `useKibanaLink` to use the url locator instead; These new dashboards were already accessible from the ` elastic_agent.*` datastreams table actions, however I replaced the `useKibanaLink` hook there as well: <img width="1412" alt="Screenshot 2023-04-13 at 16 03 47" src="https://user-images.githubusercontent.com/16084106/231784273-693c7f36-4545-4c06-a05e-4f09e53bd903.png"> TODO: I don't know where to add the "Integrations" dashboard yet, I'm not sure it should go on the Integrations details page. ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) --------- Co-authored-by: Kibana Machine <[email protected]>

harshitgupta-qasource · 2023-05-02T05:55:54Z

Hi Team,

We have created 04 test cases for this feature under the Fleet test suite at the links:

Validate that Ingest Overview Metrics button and Agent Info Metrics button is available under the Agents tab.
Validate that Ingest overview dashboard is visible on clicking on the Ingest overview button.
Verify that the Agent info dashboard is visible on clicking on the Agent Info button.
[Self-Managed]: Validate that Ingest Overview Metrics button and Agent Info Metrics buttons are not visible when there are no agents in agent list table.

Please let us know if we are missing any scenarios to be covered here.

Thanks!

harshitgupta-qasource · 2023-05-12T07:46:42Z

Hi Team,

We have executed 04 testcases under the Feature test run for the 8.8.0 release at the link:

Find integration health dashboards

Status:

PASS: 04

Build details:
VERSION: 8.8 BC3
BUILD: 62994
COMMIT: 85b22d3

As the testing is completed on this feature, we are marking this as QA:Validated.

Please let us know if anything else is required from our end.
Thanks

joshdover added enhancement New value added to drive a business result Team:Fleet Team label for Observability Data Collection Fleet team labels Mar 28, 2023

kpollich assigned criamico Apr 4, 2023

kpollich added QA:Needs Validation Issue needs to be validated by QA v8.8.0 labels Apr 5, 2023

criamico mentioned this issue Apr 13, 2023

[Fleet] Surface new overview dashboards in fleet #154914

Merged

2 tasks

criamico closed this as completed in #154914 Apr 18, 2023

harshitgupta-qasource added QA:Validated Issue has been validated by QA and removed QA:Needs Validation Issue needs to be validated by QA labels May 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Help users find integration health dashboards #153848

[Fleet] Help users find integration health dashboards #153848

joshdover commented Mar 28, 2023

elasticmachine commented Mar 28, 2023

joshdover commented Mar 28, 2023

joshdover commented Mar 28, 2023

jen-huang commented Mar 28, 2023

criamico commented Apr 12, 2023 •

edited

Loading

hop-dev commented Apr 12, 2023

criamico commented Apr 12, 2023

joshdover commented Apr 12, 2023

criamico commented Apr 12, 2023 •

edited

Loading

jen-huang commented Apr 12, 2023

criamico commented Apr 13, 2023

criamico commented Apr 13, 2023 •

edited

Loading

kpollich commented Apr 13, 2023

criamico commented Apr 13, 2023 •

edited

Loading

harshitgupta-qasource commented May 2, 2023

harshitgupta-qasource commented May 12, 2023 •

edited

Loading

[Fleet] Help users find integration health dashboards #153848

[Fleet] Help users find integration health dashboards #153848

Comments

joshdover commented Mar 28, 2023

Goals

High level proposal

Ingest overview dashboard

Agent info dashboard

Integrations overview dashboard

Implementation

elasticmachine commented Mar 28, 2023

joshdover commented Mar 28, 2023

joshdover commented Mar 28, 2023

jen-huang commented Mar 28, 2023

criamico commented Apr 12, 2023 • edited Loading

hop-dev commented Apr 12, 2023

criamico commented Apr 12, 2023

joshdover commented Apr 12, 2023

criamico commented Apr 12, 2023 • edited Loading

jen-huang commented Apr 12, 2023

criamico commented Apr 13, 2023

criamico commented Apr 13, 2023 • edited Loading

kpollich commented Apr 13, 2023

criamico commented Apr 13, 2023 • edited Loading

harshitgupta-qasource commented May 2, 2023

harshitgupta-qasource commented May 12, 2023 • edited Loading

criamico commented Apr 12, 2023 •

edited

Loading

criamico commented Apr 12, 2023 •

edited

Loading

criamico commented Apr 13, 2023 •

edited

Loading

criamico commented Apr 13, 2023 •

edited

Loading

harshitgupta-qasource commented May 12, 2023 •

edited

Loading