[System] Migrate dashboards to "by value" #5023

mrodm · 2023-01-17T15:55:11Z

What does this PR do?

Migrate the visualizations used in the dashboards that are by reference to be by value.

Just the visualizations that are used in one dashboard are migrated. Those visualizations that are shared between two or more dashboard are left as they are.

There were 13 dashboards using visualization by reference and now there are 12.
There is also a decrease in the number of visualizations since they are now in the dashboards by value. From 103 to 45 visualizations.

Process followed to migrate the dashboards:

bootstrap the minimum required kibana version of the Elastic stack using elastic-package. According to the manifest: 8.1.0 (link)
```
elastic-package stack up -v -d --version 8.1.0
```
look for the required visualizations to migrate, and press the "Unlink from library" button:
export the dashboards using elastic-package command:
```
elastic-package export dashboards
```

NOTE: not able to test the dashboards with data. This migration has been carried out using a local Elastic stack by means of elastic-package

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Related issues

Closes Migrate system dashboards to by value #3449

Screenshots

…trics dashboard

…ric dashboard

…e Metrics dashboard

elasticmachine · 2023-01-17T16:09:44Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-01-19T13:27:22.596+0000
Duration: 16 min 50 sec

Test stats 🧪

Test	Results
Failed	0
Passed	188
Skipped	0
Total	188

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-01-17T16:09:57Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`3/3`)	💚
Files	100.0% (`4/4`)	💚
Classes	100.0% (`4/4`)	💚
Methods	60.759% (`48/79`)	👎 -34.112
Lines	98.793% (`2702/2735`)	👎 -1.207
Conditionals	100.0% (`0/0`)	💚

mrodm · 2023-01-17T17:46:47Z

packages/system/data_stream/auth/_dev/test/pipeline/test-multiline.log-config.yml

+dynamic_fields:
+  "@timestamp": "^[0-9]{4}(-[0-9]{2}){2}T[0-9]{2}(:[0-9]{2}){2}\\.[0-9]{3}"


As suggested in #4950, added this dynamic field to fix tests due to timestamps:

--- want +++ got @@ -1,7 +1,7 @@ { "expected": [ { - "@timestamp": "2022-05-21T21:54:44.000Z", + "@timestamp": "2023-05-21T21:54:44.000Z", "ecs": { "version": "8.0.0" },

mrodm · 2023-01-17T18:12:43Z

packages/system/kibana/visualization/system-26877510-9b72-11ea-87e4-49f31ec44891.json

    "migrationVersion": {
-        "visualization": "7.10.0"
+        "visualization": "8.0.0"
    },
-    "namespaces": [
-        "default"
-    ],


Don't know why this namespaces field is removed.
These dashboards have been exported using elastic-package export dashboards command.

mrodm · 2023-01-17T18:55:53Z

packages/system/changelog.yml

+- version: "1.21.0-next"
+  changes:
+    - description: Migrate dashboards to be by value
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/5023


Added just an entry in changelog with -next suffix so it is not going to be published.

In this way, @elastic/elastic-agent-data-plane can decide when it needs to be released by updating the manifest and removing this suffix in another PR.

elasticmachine · 2023-01-18T15:30:53Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

elasticmachine · 2023-01-18T15:31:00Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

cmacknz · 2023-01-18T15:31:29Z

I think some of the visualizations here were created by the SEI team, we will want them to review as well.

andrewkroh

I tested that they all still load in 8.1.0. 👍 (I didn't have data populated for them though)

packages/system/changelog.yml

Apparently I wasn't testing 1.21.0-next. I used elastic-package install successfully, but I still had 1.20.4 installed. So I can't say that I tested it.

andrewkroh

This time I can confirm that they all still load (😄 and that I was testing this code).

fearful-symmetry · 2023-01-18T17:45:15Z

packages/system/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.21.0-next"


Do we normally tag versions here with -next? Don't know if I've seen that on an integration before?

If -next is added to the version, it is not published. This can be used to allow other PRs to be added without releasing.

Here is a doc about it where it shows a possible use case: https://github.com/elastic/integrations/blob/main/docs/developer_workflow_design_build_test_integration.md#remember-to-bump-up-the-version

I set the version as this (without updating manifest.yml), so the package owner team can release this when they decide.

As talked to @jlind23 and @pierrehilbert about this versioning, I'll update the changelog entry and the manifest to be 1.21.0 version. So once it is merged, it will be published.

drewdaemon

This is great.

But I'm wondering: why was this done manually instead of using our automated tooling?

mrodm · 2023-01-19T09:02:21Z

But I'm wondering: why was this done manually instead of using our automated tooling?

I didn't know about that tool, and as I needed to just migrate specific visualizations I did it manually following the process in the description of the PR. Taking advantage of the "Unlink from library" button and using elastic-package export dashboards.

elasticmachine · 2023-01-19T14:27:45Z

Package system - 1.21.0 containing this change is available at https://epr.elastic.co/search?package=system

mrodm added 13 commits January 16, 2023 17:52

Migrate system windows overview dashboard

80c73d0

Migrate logs system - syslog dashboard

6a9ea2b

Migrate [System Windows Security] Group Management Events - Simple Me…

fd546a4

…trics dashboard

Migrate [System Windows Security] User Logons - Simple Metrics dashboard

cda0ace

Migrate [Logs System] New users and groups dashboard

4280b3d

Migrate [Logs System] Sudo commands dashboard

5508cd2

Migrate [Logs System] SSH login attempts dashboard

32cc3fa

Migrate [System Windows Security] User Management Events dashboard

62551ef

Migrate [System Windows Security] User Management Events - Simple Met…

053633c

…ric dashboard

Migrate [System Windows Security] User Logons dashboard

6e7644f

Migrate [System Windows Security] Group Management Events dashboard

6a81552

Migrate [System Windows Security] Failed and Blocked Accounts dashboard

29f26b7

Migrate [System Windows Security] Failed and Blocked Accounts - Simpl…

d408a6a

…e Metrics dashboard

mrodm added the enhancement New feature or request label Jan 17, 2023

mrodm self-assigned this Jan 17, 2023

mrodm added 2 commits January 17, 2023 16:59

Add next entry changelog

5c814e6

Remove typo

138bc74

mrodm commented Jan 17, 2023

View reviewed changes

Added dynamic field for timestamp to avoid issues with years

936d41c

mrodm force-pushed the migrate_system_dashboards_by_value branch from b147231 to 936d41c Compare January 17, 2023 18:00

mrodm commented Jan 17, 2023

View reviewed changes

mrodm requested a review from a team January 17, 2023 18:18

mrodm marked this pull request as ready for review January 17, 2023 18:18

mrodm requested review from a team as code owners January 17, 2023 18:18

mrodm requested review from cmacknz and fearful-symmetry January 17, 2023 18:18

mrodm commented Jan 17, 2023

View reviewed changes

cmacknz added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team [elastic/elastic-agent-data-plane] label Jan 18, 2023

cmacknz added the Team:Security-External Integrations label Jan 18, 2023

cmacknz requested a review from a team January 18, 2023 15:31

andrewkroh previously approved these changes Jan 18, 2023

View reviewed changes

packages/system/changelog.yml Outdated Show resolved Hide resolved

andrewkroh approved these changes Jan 18, 2023

View reviewed changes

andrewkroh added the Integration:system System label Jan 18, 2023

andrewkroh changed the title ~~Migrate system dashboards by value~~ [System] Migrate dashboards to "by value" Jan 18, 2023

Update description in changelog entry

dbb5b41

fearful-symmetry reviewed Jan 18, 2023

View reviewed changes

drewdaemon approved these changes Jan 18, 2023

View reviewed changes

Update manifest version and changelog

89f83b3

mrodm merged commit 33adbbe into elastic:main Jan 19, 2023

mrodm deleted the migrate_system_dashboards_by_value branch January 19, 2023 14:08

drewdaemon mentioned this pull request Oct 15, 2024

[System] Inline all saved searches #8220

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[System] Migrate dashboards to "by value" #5023

[System] Migrate dashboards to "by value" #5023

mrodm commented Jan 17, 2023 •

edited

Loading

elasticmachine commented Jan 17, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Jan 17, 2023 •

edited

Loading

mrodm Jan 17, 2023

mrodm Jan 17, 2023

mrodm Jan 17, 2023

elasticmachine commented Jan 18, 2023

elasticmachine commented Jan 18, 2023

cmacknz commented Jan 18, 2023

andrewkroh left a comment

andrewkroh left a comment

fearful-symmetry Jan 18, 2023

mrodm Jan 18, 2023 •

edited

Loading

mrodm Jan 19, 2023

drewdaemon left a comment

mrodm commented Jan 19, 2023

elasticmachine commented Jan 19, 2023

		dynamic_fields:
		"@timestamp": "^[0-9]{4}(-[0-9]{2}){2}T[0-9]{2}(:[0-9]{2}){2}\\.[0-9]{3}"

		@@ -1,4 +1,9 @@
		# newer versions go on top
		- version: "1.21.0-next"

[System] Migrate dashboards to "by value" #5023

[System] Migrate dashboards to "by value" #5023

Conversation

mrodm commented Jan 17, 2023 • edited Loading

What does this PR do?

Checklist

Related issues

Screenshots

elasticmachine commented Jan 17, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Jan 17, 2023 • edited Loading

🌐 Coverage report

mrodm Jan 17, 2023

Choose a reason for hiding this comment

mrodm Jan 17, 2023

Choose a reason for hiding this comment

mrodm Jan 17, 2023

Choose a reason for hiding this comment

elasticmachine commented Jan 18, 2023

elasticmachine commented Jan 18, 2023

cmacknz commented Jan 18, 2023

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

fearful-symmetry Jan 18, 2023

Choose a reason for hiding this comment

mrodm Jan 18, 2023 • edited Loading

Choose a reason for hiding this comment

mrodm Jan 19, 2023

Choose a reason for hiding this comment

drewdaemon left a comment

Choose a reason for hiding this comment

mrodm commented Jan 19, 2023

elasticmachine commented Jan 19, 2023

mrodm commented Jan 17, 2023 •

edited

Loading

elasticmachine commented Jan 17, 2023 •

edited

Loading

elasticmachine commented Jan 17, 2023 •

edited

Loading

mrodm Jan 18, 2023 •

edited

Loading