System integration references non existent tags in dashboard and searches

[maxcold]

Kibana version:
8.11

Elasticsearch version:
8.11

Server OS version:
Mac OS

Browser version:
Chrome

Browser OS version:

Original install method (e.g. download page, yum, from source, etc.):

Describe the bug:
When installing a system integration, eg. by choosing Collect system logs and metrics in the New Hosts tab on the Add Integration page, and then searching for dashboard in the search bar, there is a warning
SearchBar: Tag with id "security-solution-default" not found. Tag "security-solution-default" is referenced by the search result "dashboard:system-Logs-syslog-dashboard". Skipping displaying the missing tag. in the browser console.

It's not clear where these tags are coming from as they are not referenced explicitly on the system-Logs-syslog-dashboard dashboard which is an asset of system integration.

While it's only a console warning, I'm not sure what else referencing non existent tags can affect. It used to crash Kibana's search but was fixed in

• Fix global search crash on missing tag #159196

Steps to reproduce:

1. Open browser Dev Tools console
2. add integration
3. select New Host and make sure Collect system logs and metrics option is selected
4. wait for integration installation
5. search dashboard in the Kibana search bar
6. there are warnings in the browser console

Expected behavior:
no warnings

Screenshots (if relevant):

Screen.Recording.2023-12-20.at.15.41.04.mov

Errors in browser console (if relevant):

SearchBar: Tag with id "fleet-pkg-system-default" not found. Tag "fleet-pkg-system-default" is referenced by the search result "dashboard:system-Logs-syslog-dashboard". Skipping displaying the missing tag.
SearchBar: Tag with id "security-solution-default" not found. Tag "security-solution-default" is referenced by the search result "dashboard:system-Logs-syslog-dashboard". Skipping displaying the missing tag.

Provide logs and/or server output (if relevant):

Any additional context:

[elasticmachine]

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security)

[maxcold]

Removing the Cloud Security team tag as the issue can be reproduced with any integration as far as I can tell. I guess that it should be tagged with the Fleet team tag, but I will leave tagging to those who are better aware of which team it should belong to

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[maxcold]

Adding Team: Fleet as the problem is concerning the System integration. Hope that makes sense

[jlind23]

Hi @maxcold I believe this should rather be created in the https://github.com/elastic/integrations repository and looks like this could have been added by @marc-gr - https://github.com/elastic/integrations/blame/da6480c09e86a288c5c811f139f3b9f8a4f69b61/packages/system/kibana/tags.yml#L12

[maxcold]

Hi @jlind23 , thanks for bringing this up! As the issue was coming from globalSearchBar in Kibana, I wasn't sure where to open the issue. Kibana repo seemed like a good starting point.
As for the fix you mentioned, I still see the warning in the logs in version 8.13.0 of the stack (system integration v1.54.0).
One thing that I noticed is that I see this issue in logs only in us-west2 (LA) which is our cloud-first testing env, and not in other envs (I looked into Iowa as well). I guess due to the differences in the logging levels between these envs. I'm not sure how big of a problem the issue is taking into account that it's only a warning in our testing env, but who knows what it can lead to

[marc-gr]

The tag reference added in the integration should be automatically created by the Security Solution (more context #164582). IIRC for this to be the case user should have entered the Security Solution UI at least once, but I might be wrong on this.