Metrics system overview dashboard redirects to a 404 Dashboard not found page in Kibana

[chamakaleprattik]

Kibana version:
8.11.2
Elasticsearch version:
8.11.2
Server OS version:
Linux
Browser version:
Chrome
Browser OS version:
Version 120.0.6099.216 (Official Build) (arm64)
Original install method (e.g. download page, yum, from source, etc.):
Using the Elastic integration
Describe the bug:
We have elastic agent v8.10.4 running on a windows and a linux device. They are able to send the metrics information to the elastic node. If I open the Metrics system overview dashboard it shows me the overview of the system health but if I click on the system name it redirects me to the 404 dashboard not found page.
I have tried the same issue with kibana version v8.10.4 and v8.11.2. My agent is still running on v8.10.4
Steps to reproduce:

1. Install Elastic agent on windows/linux device
2. Search Metrics system overview dashboard and open it.
3. Click on the hostname which redirectes you to a new page and gives the 404 dashboard not found error message.

Expected behavior:
After clicking the hostname it should redirect to Metrics system host overview dashboard
Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:
I just did a reverse test, which is to open the [Metrics System] Host overview dashboard and click on the System overview link in it. It takes me to the 404 dashboard not found page again.

[mbudge]

We're on 8.11.4 and this is affecting all of the links where they exist in Fleet managed dashboard.

I've tried reinstalling integration assets but it doesn't fix it.

Every link I've tested so far in every Fleet managed dashboard has returned a 404 Not Found error.

[elasticmachine]

Pinging @elastic/kibana-presentation (Team:Presentation)

[davismcphee]

@elastic/kibana-presentation Apologies if this is totally unrelated to Dashboard itself, it wasn't clear to me who to assign this one too. I tried adding Team:Integrations as well, but it looks like that label doesn't actually ping a team...

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

Every link I've tested so far in every Fleet managed dashboard has returned a 404 Not Found error.

This sounds pretty catastrophic. Will attempt to reproduce the issue.

[mbudge]

Tracked in support ticket #01547752

elastic v8.11.4

[kpollich]

I tried to come up with a minimal reproduction case here in a fresh cluster with a few agents enrolled and was unsuccessful

Screen.Recording.2024-01-26.at.9.03.39.AM.mov

Summary of steps taken

1. Enroll a Fleet Server + a few agents, ensure the "enable system monitoring" and "enable agent monitoring" options are checked (they are enabled by default) when enrolling
2. Navigate to the [Metrics System] Overview dashboard installed by the system integration
3. Click into the host names and observe no 404's
4. Verify that the elastic_agent and system integrations are shown as installed in the default space and their assets appear in the "Assets" tab and that these links also don't 404

Some points of clarification that would be helpful here:

1. Are there multiple spaces in use here? If so, was the first agent enrolled in another space, or were any integrations ever installed in another space?
2. Do the elastic_agent and system integration appear as "installed" in the integrations UI as above?
3. Do the dashboard, saved search, etc assets under the "Assets" tab in the integration settings appear?

My initial suspicion is that there are multiple spaces in use for these scenarios, so I will attempt to reproduce the specific behavior using spaces. If that's not the case, then please disregard the below.

Please note, that multi-space usage of integrations is not fully supported at this time as mentioned in https://www.elastic.co/guide/en/fleet/current/integrations.html. Some integrations may work across spaces, others may not.

We also call out that integration assets can only be installed in a single space at a time here: https://www.elastic.co/guide/en/fleet/current/install-uninstall-integration-assets.html. If you want to clone assets between spaces today, you'll need to use Kibana's built in "copy" function for dashboards, etc.

We are actively working on full multi-space support for Fleet/Integrations in this calendar year, and there is a public issue for some of the more immediate annoyances (as opposed to the large scope of work we're defining now) here: #172964. Please also see elastic/integrations#3434 for a lengthy discussion with other users that outlines some of Fleet's current limitations and pain points when used in multiple spaces.

[kpollich]

I took a pass at reproducing this with multiple spaces, and I see some similar behavior to the issue, but the Integrations UI callout makes it somewhat obvious what's going on here by mentioning there are no assets for the integration in this space.

Screen.Recording.2024-01-26.at.9.28.03.AM.mov

[ITSEC-Hescalona]

This has always happened, there's still no elegant solution, I've been dealing with this problem for years. It occurs when you have multiple spaces, it doesn't happen with the default. Before, I used to fix it by editing the default dashboards and correcting the links (which was unfortunately manual and tedious), but now they don't let you edit the system dashboards. I've always had the same issue with the system dashboards and some others offered as integrations.

Truth is, at least for me, I'm bored with these minor issues. There are many of them, and in the long run, how can you trust something that fails so often?

[kpollich]

@ITSEC-Hescalona - Integrations are not space aware, so the links present in these dashboard sources are hardcoded, e.g.

https://github.com/elastic/integrations/blob/a2c55c4cbf752e0490f9fe2d3e68698517c7b74d/packages/system/kibana/dashboard/system-Metrics-system-overview.json#L368

These URL's won't include a space ID in them e.g. /s/my-second-space so when the integration is installed in one space and then copied to another, the URL's aren't updated to reflect that space difference. It makes sense that editing the assets manually was a tedious workaround before, though we've recently made these assets read-only by default to avoid cases where users make customizations that are blown away when an integration is updated.

cc @drewdaemon - Maybe you are a good person to ask about what can be done today to make these URL's space-aware during the clone operation? If a dashboard includes templated links like we generate for agent hostnames today, is there anything we can do in the dashboard JSON to indicate that the end result URL should include the space indicator where the dashboard is installed?

I know this relates closely to #167901 as well.

[ITSEC-Hescalona]

"I get you, but it's not what we sees. In the example I'm giving you, the dashboard that I directly opened is this one: s/SPACE/app/dashboards#/view/4d1b0886-0d82-4451-9855-d2481706749f, and a link points to /s/SPACE/app/dashboards#/view/system-bae11b00-9bfc-11ea-87e4-49f31ec44891. Opening it from the space results in a 404 error, but the same URL from the default does show up. I don't know if you understand what I mean."

[mbudge]

Fleet support for multiple spaces should have been sorted 12 to 18 months ago, as Fleet has been GA since mid-2021. You can't expect us to have 1 operations team per deployment/Kibana instance to get this stuff working.

[drewdaemon]

@ITSEC-Hescalona I understand what you mean. The key is that the IDs change when you copy integration assets to another space, but the hyperlinks don't. As @kpollich mentioned, the official answer is that integrations aren't (yet) space aware. Still, I totally understand the frustration.

To improve the situation today

• if a user uses the HTTP import API directly (not copy-to-spaces option in the UI) to copy integration assets to other spaces, they can include a compatibilityMode parameter in the import request. This will automatically generate aliases for any saved objects which change IDs, thereby preserving the links. One caveat: this feature introduced an aliasing bug to dashboards in 8.11. The fix is slated to be released in 8.13 and 8.12.1
• integration authors can use the new dashboard links panel instead of hard-coding navigation into markdown panels. Hyperlinks created with the links panel will survive the ID changes that occur when copying integration assets to other spaces. This covers the majority of the cases (though unfortunately not the one pictured in this issue, because those links are generated from a data view field formatter)

And for the future @kpollich mentioned #167901. This effort will fill in a key platform feature which will allow dashboards and other integration assets to be shared between spaces instead of copied. That opens the door for Fleet to install assets across spaces with none of the saved object ID changes associated with copying.

[ITSEC-Hescalona]

@drewdaemon Thanks !!! great explanation, you changed me from very frustrated to frustrated but hopeful

[kpollich]

I think #172963 would be a good path to remediating this, by allowing Fleet to duplicate assets across many spaces. This would be a stopgap solution until first-class shareable analytics ship in #167901.

Please also see #175831, which I've created as a centralized place for further discussion around multi-space usage of Fleet. It includes substantial historical and technical context, and will provide continual updates as we implement first-class multi-space RBAC support in Fleet.

[lalit-satapathy]

Hi @drewdaemon , @kpollich

Thanks for the analysis and work around being explored.

I am trying to understand the severity of impact of this issue for existing integration users. Do we have a clear summary of the impact of this issue, which integrations dashboards will be impacted and since which release? I am wondering how do we ensure not other SDH gets filed and any user doc we can create to explain the same.

[drewdaemon]

@lalit-satapathy I can say that before 8.0 you could have multiple saved objects in multiple spaces with identical IDs. Since 8.0, copying saved objects to other spaces changes their ID, no matter what (ref).

Any integration that hard-codes a saved object ID in a URL (e.g. using a markdown panel for navigation) will have a broken link when the assets are copied into a new space. My guess is that this is most integrations.

I think #175831 will be the center of the discussion moving forward and will include documentation of the issue.