[Fleet] [Kibana] Integrations fail to upgrade or (un)/(re)-install when using multiple Kibana Spaces #161804
Comments
matthiasledergerber
added
the
bug
|
Further testing and information
No Dashboards of Cisco ASA or Palo Alto Networks visible. Only the Sample Dashboard Checking the Agent Policies, the Integration is still there: This message starts to appear The integration seems to be stuck in upgrading The kibana logs While navigating in Fleet the following message appears
Trying to reinstall: These actions somehow influence the other Dashboards from other integrations. One integration that was added per default with the Upgrade to Elastic-Stack 8.2.2 was "System". Here we see the effect that we also encounter in our on-premise Elastic-Stack 8.8.2: "Could not find the search: system-8030c1b0-fa77-11e6-ae9b-81e5311e8cab" (id varries) Logfile of Step 8
|
juliaElastic
added
the
Team:Fleet
|
Pinging @elastic/fleet (Team:Fleet) |
|
@jlind23 I think we should add this bug to the next sprint, it is coming up in many support issues. |
|
This is severely impacting usability |
## Summary Fix #162639 Found a bug when paths were pushed twice in package archive, this results in an error when trying to reinstall endpoint package, fails with a conflict error when creating transforms. See more details here: #162639 (comment) It was introduced with this change, perhaps unintentionally: https://github.com/elastic/kibana/pull/151655/files I think this bug is also the reason of many [issues](#161804) with package reinstallation. I could reproduce another bug when trying to reinstall `system` integration, that is also fixed by removing the duplicate paths. ``` [2023-07-28T17:41:27.116+02:00][ERROR][plugins.fleet] Error: Non-unique import objects detected: [dashboard:system-0d3f2380-fa78-11e6-ae9b-81e5311e8cab,dashboard:system-277876d0-fa2c-11e6-bbd3-29c986c96e5a,dashboard:system-5517a150-f9ce-11e6-8115-a7c18106d86a,dashboard:system-71f720f0-ff18-11e9-8405-516218e3d268,dashboard:system-79ffd6e0-faa0-11e6-947f-177f697178b8,dashboard:system-Logs-syslog-dashboard,dashboard:system-Metrics-system-overview,dashboard:system-Windows-Dashboard,dashboard:system-bae11b00-9bfc-11ea-87e4-49f31ec44891,dashboard:system-bb858830-f412-11e9-8405-516218e3d268,dashboard:system-d401ef40-a7d5-11e9-a422-d144027429da,search:system-06b6b060-7a80-11ea-bc9a-0baf2ca323a3,search:system-324686c0-fefb-11e9-8405-516218e3d268,search:system-62439dc0-f9c9-11e6-a747-6121780e0414,search:system-6f4071a0-7a78-11ea-bc9a-0baf2ca323a3,search:system-757510b0-a87f-11e9-a422-d144027429da,search:system-7e178c80-fee1-11e9-8405-516218e3d268,search:system-8030c1b0-fa77-11e6-ae9b-81e5311e8cab,search:system-9066d5b0-fef2-11e9-8405-516218e3d268,search:system-Syslog-system-logs,search:system-b6f321e0-fa25-11e6-bbd3-29c986c96e5a,search:system-ce71c9a0-a25e-11e9-a422-d144027429da,search:system-eb0039f0-fa7f-11e6-a1df-a78bd7504d38] at Function.nonUniqueImportObjects (errors.ts:35:12) ``` ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Fix elastic#162639 Found a bug when paths were pushed twice in package archive, this results in an error when trying to reinstall endpoint package, fails with a conflict error when creating transforms. See more details here: elastic#162639 (comment) It was introduced with this change, perhaps unintentionally: https://github.com/elastic/kibana/pull/151655/files I think this bug is also the reason of many [issues](elastic#161804) with package reinstallation. I could reproduce another bug when trying to reinstall `system` integration, that is also fixed by removing the duplicate paths. ``` [2023-07-28T17:41:27.116+02:00][ERROR][plugins.fleet] Error: Non-unique import objects detected: [dashboard:system-0d3f2380-fa78-11e6-ae9b-81e5311e8cab,dashboard:system-277876d0-fa2c-11e6-bbd3-29c986c96e5a,dashboard:system-5517a150-f9ce-11e6-8115-a7c18106d86a,dashboard:system-71f720f0-ff18-11e9-8405-516218e3d268,dashboard:system-79ffd6e0-faa0-11e6-947f-177f697178b8,dashboard:system-Logs-syslog-dashboard,dashboard:system-Metrics-system-overview,dashboard:system-Windows-Dashboard,dashboard:system-bae11b00-9bfc-11ea-87e4-49f31ec44891,dashboard:system-bb858830-f412-11e9-8405-516218e3d268,dashboard:system-d401ef40-a7d5-11e9-a422-d144027429da,search:system-06b6b060-7a80-11ea-bc9a-0baf2ca323a3,search:system-324686c0-fefb-11e9-8405-516218e3d268,search:system-62439dc0-f9c9-11e6-a747-6121780e0414,search:system-6f4071a0-7a78-11ea-bc9a-0baf2ca323a3,search:system-757510b0-a87f-11e9-a422-d144027429da,search:system-7e178c80-fee1-11e9-8405-516218e3d268,search:system-8030c1b0-fa77-11e6-ae9b-81e5311e8cab,search:system-9066d5b0-fef2-11e9-8405-516218e3d268,search:system-Syslog-system-logs,search:system-b6f321e0-fa25-11e6-bbd3-29c986c96e5a,search:system-ce71c9a0-a25e-11e9-a422-d144027429da,search:system-eb0039f0-fa7f-11e6-a1df-a78bd7504d38] at Function.nonUniqueImportObjects (errors.ts:35:12) ``` - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
|
Good news that we found and fixed the root cause of Will check if the other error |
## Summary Fix elastic#162639 Found a bug when paths were pushed twice in package archive, this results in an error when trying to reinstall endpoint package, fails with a conflict error when creating transforms. See more details here: elastic#162639 (comment) It was introduced with this change, perhaps unintentionally: https://github.com/elastic/kibana/pull/151655/files I think this bug is also the reason of many [issues](elastic#161804) with package reinstallation. I could reproduce another bug when trying to reinstall `system` integration, that is also fixed by removing the duplicate paths. ``` [2023-07-28T17:41:27.116+02:00][ERROR][plugins.fleet] Error: Non-unique import objects detected: [dashboard:system-0d3f2380-fa78-11e6-ae9b-81e5311e8cab,dashboard:system-277876d0-fa2c-11e6-bbd3-29c986c96e5a,dashboard:system-5517a150-f9ce-11e6-8115-a7c18106d86a,dashboard:system-71f720f0-ff18-11e9-8405-516218e3d268,dashboard:system-79ffd6e0-faa0-11e6-947f-177f697178b8,dashboard:system-Logs-syslog-dashboard,dashboard:system-Metrics-system-overview,dashboard:system-Windows-Dashboard,dashboard:system-bae11b00-9bfc-11ea-87e4-49f31ec44891,dashboard:system-bb858830-f412-11e9-8405-516218e3d268,dashboard:system-d401ef40-a7d5-11e9-a422-d144027429da,search:system-06b6b060-7a80-11ea-bc9a-0baf2ca323a3,search:system-324686c0-fefb-11e9-8405-516218e3d268,search:system-62439dc0-f9c9-11e6-a747-6121780e0414,search:system-6f4071a0-7a78-11ea-bc9a-0baf2ca323a3,search:system-757510b0-a87f-11e9-a422-d144027429da,search:system-7e178c80-fee1-11e9-8405-516218e3d268,search:system-8030c1b0-fa77-11e6-ae9b-81e5311e8cab,search:system-9066d5b0-fef2-11e9-8405-516218e3d268,search:system-Syslog-system-logs,search:system-b6f321e0-fa25-11e6-bbd3-29c986c96e5a,search:system-ce71c9a0-a25e-11e9-a422-d144027429da,search:system-eb0039f0-fa7f-11e6-a1df-a78bd7504d38] at Function.nonUniqueImportObjects (errors.ts:35:12) ``` ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
|
Not fixed in 8.9.1 |
|
We've Upgraded our Cluster form 8.9.0 to 8.9.1. However, this issue still persists and our Dashboards in Multiple Spaces have broken references. All Integrations are affected:
I can provide more information if needed. |
Kibana version: 8.8.2
Elasticsearch version: 8.8.2
Server OS version: Debian 12
Browse: Edge
Original install method: apt
Describe the bug:
It seems as the bug started to occur around Elastic-Stack 8.8.0. Currently installed Elastic Integrations with Visualisations and Data Views cannot be upgraded due to existing objects with duplicated ids (?). A uninstall routine that starts automatically fails too and the integration stops to be usable at all. As soon as Kibana is restarted, all visualizations generate an Error that the data view cannot be found. This leads to unusable visualisations that are delivered with the integration. I've tried to reinstall the integrations, delete all objects like templates, data views, visualisations, pipelines, etc. by hand but as soon as Kibana restarts everything that comes from the integrations start to break again. The integrations were also completely removed before reinstalling but this doesn't work either. We initially had three Kibana instances but this report was done with one single instance. We have several Kibana Spaces.
Sometimes after cleaning up everything and trying to reinstall the integration again, the kibana.log still reports of ID issues. However, when tracing these IDs of the dashboards or visualizations mentioned, they cannot be found in any space of Kibana
After running the following command the integration was able to be installed despite the report of these gost-IDs
Im summary
Integrations cannot be upgraded
Integrations cannot be reinstalled
Integrations cannot be installed
One guess is that this issue must have occurred from our upgrade path. We upgrade every 1-2 Versions since 7.x. Therefore it could be possible that this issue wasn't catched by the CI. However, it seems as also the latest build of 8.8.2 has this issue - in a new elastic stack.
Steps to reproduce:
I was partially able to reproduce this issue with elastic-stack on docker.
I've tested this with 8.7.1 too. There is the same issue, However after the restart of Kibana i still had the Dashboards. In my production Cluster, such a try of a reinstall renders ALL other Dashboards in ALL Spaces that are out-of-the-box integration vom Elastic unusable (data view not found see screenshot below).
The important part is, this could be forced with a clean elastic stack installation. And one integration having this issue crashes all other predefined dashboards
Expected behavior:
The Integration should upgrade or reinstall without throwing errors of duplicated ID. It should automatically resolve the dependency error. Out-of-the-Box integrations should be usable.
Screenshots (if relevant):
Result for user in a production cluster when trying to reinstall the missing assets, it renders the other out-of-the-box dashboards useless. This also happens after just restarting Kibana:
Errors in browser console (if relevant):
Provide logs and/or server output (if relevant):
As soon as i try to reinstall an integration it reports of the following errors
Any additional context:
Reported in the forum: https://discuss.elastic.co/t/elastic-integrations-fail-to-install-and-lead-to-broken-dashboards-when-used-with-multiple-kibana-spaces/337246/2
Related GitHub Discussion:
#108959 Handle ID changes to Saved Objects installed by Fleet
#59598 Saved object id migration issues
The text was updated successfully, but these errors were encountered: