Not fixed in 8.9.1 / Integrations in multiple Kibana Spaces

[cemetery84]

          Not fixed in 8.9.1 

If you update or reinstall some Integrations and restart Kibana after this you get logs-* and metrics-* Data Views in the default Space. And logs-* and metrics-* is deleted in other Spaces.

Originally posted by @cemetery84 in #161804 (comment)

[matthiasledergerber]

Yes i can confirm the issue still persists. We've cleaned up all the Integrations with duplicate data views from above and it worked fine but after Upgrading to 8.9.1 it broke again.

For us, it is currently an bigger issue

Windows

Bitwarden

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[juliaElastic]

@matthiasledergerber Could you please give more details on how the issue can be reproduced?

[cemetery84]

If you update or reinstall some Integrations and restart Kibana after this you get logs-* and metrics-* Data Views in the default Space. And logs-* and metrics-* is deleted in other Spaces. --> Important after reinstall some integrations you must restart Kibana....and go to the Data View only in the default Space its now logs-* and metrics-* if you restart kibana again you get a second logs-* and metrics-* and so on...Only can fix if you delete logs-* and metrics-* in the default Space an add the dataview manual in the right space. Now Kibana can restart and nothing happend. But if you update/reinstall some integration it starts again....i think some startup-checks create the logs-* and metrics-* in the default space and if the index exist other spaces can´t use them. Because the use the same custom data view ID.

[matthiasledergerber]

If you update or reinstall some Integrations and restart Kibana after this you get logs-* and metrics-* Data Views in the default Space. And logs-* and metrics-* is deleted in other Spaces. --> Important after reinstall some integrations you must restart Kibana....and go to the Data View only in the default Space its now logs-* and metrics-* if you restart kibana again you get a second logs-* and metrics-* and so on...Only can fix if you delete logs-* and metrics-* in the default Space an add the dataview manual in the right space. Now Kibana can restart and nothing happend. But if you update/reinstall some integration it starts again....i think some startup-checks create the logs-* and metrics-* in the default space and if the index exist other spaces can´t use them. Because the use the same custom data view ID.

@juliaElastic I can confirm this. After Upgrading this issue happens. Seems to be related with multiple kibana spaces and integrations.

[juliaElastic]

I could reproduce this on a 8.9.1 cluster, when installing the system integration in a non-default space, after kibana restart, the logs-* and metrics-* data views are deleted from the non-default space, and recreated in default space.

I did some investigation and noticed that the data views are being recreated in default space during the kibana saved object migration, that runs on every kibana restart. When I disabled this with migrations.skip: true in kibana.yml, the data views didn't disappear from the space.

Hey @elastic/kibana-core, could you confirm that this is an issue with SO migrations, that index patterns are not being migrated with the right namespace? Can it be fixed?

Seeing this in kibana logs during the SO migration:

[2023-08-22T13:33:49.851+02:00][DEBUG][savedobjects-service] [.kibana_analytics] OUTDATED_DOCUMENTS_SEARCH_READ RESPONSE
[2023-08-22T13:33:49.851+02:00][INFO ][savedobjects-service] [.kibana_analytics] Starting to process 2 documents.
[2023-08-22T13:33:49.851+02:00][DEBUG][savedobjects-service] [.kibana_analytics] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_TRANSFORM. took: 16ms.

[2023-08-22T13:33:49.861+02:00][DEBUG][savedobjects-service] [.kibana_analytics] OUTDATED_DOCUMENTS_TRANSFORM RESPONSE
[2023-08-22T13:33:49.862+02:00][DEBUG][savedobjects-service] [.kibana_analytics] OUTDATED_DOCUMENTS_TRANSFORM -> TRANSFORMED_DOCUMENTS_BULK_INDEX. took: 11ms.

[2023-08-22T13:33:49.907+02:00][DEBUG][elasticsearch.query.data] 200 - 2.0B
POST /.kibana_analytics_8.11.0_001/_bulk?require_alias=false&wait_for_active_shards=all&refresh=false&filter_path=items.*.error
{"index":{"_id":"index-pattern:logs-*"}}
{"index-pattern":{"title":"logs-*","timeFieldName":"@timestamp","allowNoIndex":true},"type":"index-pattern","references":[],"managed":true,"namespaces":["default"],"coreMigrationVersion":"8.8.0","typeMigrationVersion":"8.0.0","updated_at":"2023-08-22T11:33:01.658Z","created_at":"2023-08-22T11:33:01.658Z"}
{"index":{"_id":"index-pattern:metrics-*"}}
{"index-pattern":{"title":"metrics-*","timeFieldName":"@timestamp","allowNoIndex":true},"type":"index-pattern","references":[],"managed":true,"namespaces":["default"],"coreMigrationVersion":"8.8.0","typeMigrationVersion":"8.0.0","updated_at":"2023-08-22T11:33:01.658Z","created_at":"2023-08-22T11:33:01.658Z"}

[2023-08-22T13:33:49.907+02:00][DEBUG][savedobjects-service] [.kibana_analytics] TRANSFORMED_DOCUMENTS_BULK_INDEX RESPONSE
[2023-08-22T13:33:49.908+02:00][DEBUG][savedobjects-service] [.kibana_analytics] TRANSFORMED_DOCUMENTS_BULK_INDEX -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 46ms.

When I install the system package in test space, the index pattern is created correctly (see "namespaces":["test"]):

POST /.kibana_ingest_8.11.0/_update/epm-packages%3Asystem?refresh=false&retry_on_conflict=3&_source_includes=namespace%2Cnamespaces%2CoriginId&require_alias=true
{"doc":{"epm-packages":{"installed_kibana":[{"id":"system-0d3f2380-fa78-11e6-ae9b-81e5311e8cab","type":"dashboard"},{"id":"system-277876d0-fa2c-11e6-bbd3-29c986c96e5a","type":"dashboard"},{"id":"system-5517a150-f9ce-11e6-8115-a7c18106d86a","type":"dashboard"},{"id":"system-71f720f0-ff18-11e9-8405-516218e3d268","type":"dashboard"},{"id":"system-79ffd6e0-faa0-11e6-947f-177f697178b8","type":"dashboard"},{"id":"system-Logs-syslog-dashboard","type":"dashboard"},{"id":"system-Metrics-system-overview","type":"dashboard"},{"id":"system-Windows-Dashboard","type":"dashboard"},{"id":"system-bae11b00-9bfc-11ea-87e4-49f31ec44891","type":"dashboard"},{"id":"system-bb858830-f412-11e9-8405-516218e3d268","type":"dashboard"},{"id":"system-d401ef40-a7d5-11e9-a422-d144027429da","type":"dashboard"},{"id":"system-06b6b060-7a80-11ea-bc9a-0baf2ca323a3","type":"search"},{"id":"system-324686c0-fefb-11e9-8405-516218e3d268","type":"search"},{"id":"system-62439dc0-f9c9-11e6-a747-6121780e0414","type":"search"},{"id":"system-6f4071a0-7a78-11ea-bc9a-0baf2ca323a3","type":"search"},{"id":"system-757510b0-a87f-11e9-a422-d144027429da","type":"search"},{"id":"system-7e178c80-fee1-11e9-8405-516218e3d268","type":"search"},{"id":"system-8030c1b0-fa77-11e6-ae9b-81e5311e8cab","type":"search"},{"id":"system-9066d5b0-fef2-11e9-8405-516218e3d268","type":"search"},{"id":"system-Syslog-system-logs","type":"search"},{"id":"system-b6f321e0-fa25-11e6-bbd3-29c986c96e5a","type":"search"},{"id":"system-ce71c9a0-a25e-11e9-a422-d144027429da","type":"search"},{"id":"system-eb0039f0-fa7f-11e6-a1df-a78bd7504d38","type":"search"}]},"updated_at":"2023-08-22T11:45:34.433Z"}}
[2023-08-22T13:45:34.710+02:00][DEBUG][elasticsearch.query.data] 200 - 5.8KB
POST /_bulk?refresh=false&require_alias=true
{"index":{"_id":"index-pattern:4a400166-1e44-4538-8d88-d16defe56229","_index":".kibana_analytics_8.11.0"}}
{"index-pattern":{"title":"logs-*","timeFieldName":"@timestamp"},"type":"index-pattern","references":[],"managed":true,"namespaces":["test"],"originId":"logs-*","coreMigrationVersion":"8.8.0","typeMigrationVersion":"7.11.0","updated_at":"2023-08-22T11:45:34.469Z","created_at":"2023-08-22T11:45:34.469Z"}
{"index":{"_id":"index-pattern:0bca408b-a828-48c2-83a6-3df946eee1d6","_index":".kibana_analytics_8.11.0"}}
{"index-pattern":{"title":"metrics-*","timeFieldName":"@timestamp"},"type":"index-pattern","references":[],"managed":true,"namespaces":["test"],"originId":"metrics-*","coreMigrationVersion":"8.8.0","typeMigrationVersion":"7.11.0","updated_at":"2023-08-22T11:45:34.469Z","created_at":"2023-08-22T11:45:34.469Z"}

[pgayvallet]

I did some investigation and noticed that the data views are being recreated in default space during the kibana saved object migration

When I install the system package in test space, the index pattern is created correctly (see "namespaces":["test"]):

Sorry, what's the difference between the two, here?

[juliaElastic]

When the package is installed in a non-default space, the namespaces field includes the correct non-default space name, but after kibana restarts, index patterns are being changed to namespaces: default

[pgayvallet]

I created #164454 to track the issue on Core's side

[juliaElastic]

After discussing internally, it seems we found the issue. The index patterns are created by Fleet without specifying a typeMigrationVersion, which is set incorrectly to 7.11.0 by default in kibana core. This causes a migration logic to run, which overrides the namespaces field to default.
The issue is raised here in kibana core side.

[juliaElastic]

The fix is merged and will be included in 8.9.2

[cemetery84]

The fix is merged and will be included in 8.9.2

Now Logs and metrics Data View are fixed but winlogbeat (Data View) has now the problem

[matthiasledergerber]

I can confirm this fix worked, however there seems to be another issue with the integrations in one of the latest upgrades. I'm not sure where the problem is exactly, it seems as the install/uninstall routine can get stuck becaused of saved object conflicts that lead to a reinstall loop. This fills Kibana spaces randomly with saved objects from the integration even if the integration wasn't installed into the Kibana space and furthermore it makes the integration stuck in the Fleet page even if there are no installed integrations anymore
#166798