Migrate system dashboards to by value

[ruflin]

See #3448 for more details

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[cmacknz]

I believe the VisEditors team is handling this

[ruflin]

The visEditor team handled this for the metrics dashboard but it seems the logs dashboards still use by reference.

[joshdover]

Moving to by-value for all system dashboards would improve the performance in the onboarding experience since we install this package by default when creating the first agent policy. Creating the Kibana SOs takes a significant amount of time in some cases (5-10s) and using by-value viz would reduce the number of SOs to create.

[mrodm]

Checking the visualizations that are set as reference in the system package, there are some dashboards that share some visualizations (same references).

For instance, the following list show the visualization by reference that appear in dashboard kibana/dashboard/system-035846a0-a249-11e9-a422-d144027429da.json along with the number of dashboards where that visualization is used:

 - system-804dd400-a248-11e9-a422-d144027429da (visualization): 2
 - system-5bb93ed0-a249-11e9-a422-d144027429da (visualization): 1
 - system-e2516c10-a249-11e9-a422-d144027429da (visualization): 2
 - system-18348f30-a24d-11e9-a422-d144027429da (visualization): 2
 - system-d770b040-9b35-11ea-87e4-49f31ec44891 (visualization): 4
 - system-2c71e0f0-9c0d-11ea-87e4-49f31ec44891 (visualization): 1
 - system-abd44840-9c0f-11ea-87e4-49f31ec44891 (visualization): 1
 - system-006d75f0-9c03-11ea-87e4-49f31ec44891 (visualization): 2
 - system-21aadac0-9c0b-11ea-87e4-49f31ec44891 (visualization): 2
 - system-25f31ee0-9c23-11ea-87e4-49f31ec44891 (visualization): 2

What should we do in those cases? Keep the visualization as a reference in those dashboards? Or duplicate them in each dashboard so they are by value? @ruflin @joshdover @elastic/elastic-agent-data-plane

[ruflin]

@mrodm Interesting find. Can you add some details (maybe screenshot) on what kind of visualizations these are and what they contain?

[mrodm]

In most of the dashboards in the system package happens this issue with visualizations shared between dashboards.

I'm running the Elastic stack using elastic-package, and I don't have data in visualizations (I hope it could be helpful), but I'll try to show some examples:

• A visualization to keep a link to several dashboards and the title:

  • Dashboard: [System Windows Security] User Logons
    
  • Dashboard: [Logs System] SSH login attempts
    

• There are some dashboards that are defined once and they are also created again with the suffix "- Simple metrics" that share some visualizations too. For instance:

  • [System Windows Security] User Logons
    
  • [System Windows Security] User Logons - Simple metrics
    

[jlind23]

@mrodm @ruflin I would try to avoid duplicating dashboard as it is going to add some toil work when updating them.
My path forward would be:

• For shared viz, leave them as is.
• Migrate other viz to by value.

[ruflin]

@andrewkroh Could someone from your team comment on why there are 2 dashboards with the same visualisations for windows security? Ideally your team would take care of updating these.

I understand why the navigation is reused. Lets keep it that way for now. I hope we get to a point where we find a better way to build these navigations.

+1 on the suggested path forward form @jlind23

[andrewkroh]

I'm not exactly sure why. These dashboards have been transformed and copied multiple times. They originated in Beats. We'll get someone to take a closer look and perform a cleanup (and probably make sure everything uses Lens). The most recent work was performed in elastic/beats#18775 which gives an idea of what the dashboards should look like.

Issue: #5022