[DOCS] Document the new Detection rule monitoring dashboard
#3478
Assignees
Labels
Effort: Medium
Issues that take moderate but not substantial time to complete
Feature: Rules
Priority: Medium
Issues that have relevance, but aren't urgent
Team: Detections/Response
Detections and Response
Team: Docs
v8.9.0
Comments
banderror
added
Team: Docs
Team: Detections/Response
4 tasks
banderror
added a commit
to elastic/kibana
that referenced
this issue
Jun 20, 2023
**Addresses:** elastic/security-team#6032 ## Summary This PR adds a new `[Elastic Security] Detection rule monitoring` Kibana dashboard and a new `POST /internal/detection_engine/health/_setup` API endpoint. ## Dashboard The dashboard can be helpful for monitoring the health and performance of Security detection rules. Users of the dashboard must have read access to the `.kibana-event-log-*` index. The dashboard is automatically installed into the current Kibana space when a user visits a page in Security Solution - similar to how we install the Fleet package with prebuilt detection rules. <img width="1791" alt="Kibana dashboards page" src="https://github.com/elastic/kibana/assets/7359339/92cb3c75-39ea-4069-b70f-8f531869edf7"> <img width="1775" alt="Security dashboards page" src="https://github.com/elastic/kibana/assets/7359339/3b27aeb6-2222-40fd-a453-c204fcee4f31">  ## API endpoint The PR also adds a new endpoint for setting up anything related to monitoring rules and the health of the Detection Engine. If you call the endpoint, it will install the new dashboard to the Default Kibana space: ``` POST /internal/detection_engine/health/_setup ``` In order to install the dashboard to a different Kibana space, you will need to call it like that: ``` POST /s/<space-id>/internal/detection_engine/health/_setup ``` The user calling the endpoint must have access to Security Solution. No additional privileges are required, because the endpoint installs the dashboard on behalf of the internal user (`kibana_system`). ### Checklist - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] elastic/security-docs#3478 - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
|
Add note in docs about required role/privilege to query and view the indices on this dashboard. @banderror: Could you confirm what specific role or privilege is needed? |
joepeeples
added
Priority: Medium
2 tasks
|
@joepeeples Sorry for not being 100% clear in the PR. To be very specific, here are the RBAC requirements (I added this section to the original PR): RBAC requirementsFor installing the dashboardThe user calling the 
For viewing the dashboardIn order to be able to view the dashboard, users will need to have at least the following 2 privileges:
|
|
@joepeeples I just opened the 2nd PR with some fixes, and would appreciate any comments on the text I added to the dashboard: elastic/kibana#160316 Thank you 🙏 |
banderror
added a commit
to elastic/kibana
that referenced
this issue
Jun 27, 2023
**Epic:** elastic/security-team#6032 (internal) **Related to:** #159875 ## Summary In this PR we: - add a text block to the dashboard itself with helpful info about it - fix the 4 tables at the bottom of the dashboard - add unit tests for the dashboard's source `.json` files ## Text block <img width="1792" alt="Screenshot 2023-06-22 at 20 15 30" src="https://github.com/elastic/kibana/assets/7359339/55d267b7-7c39-4cdf-a917-fd17e9231a59"> ## Tables There were two issues with the tables: 1. When having the same prebuilt rules installed in two or more Kibana spaces, sorting in a table could break if the table rendered two different rules with the same name. It has been fixed by making the rule ID the first field in the table and making a few minor tweaks. Thanks to @maximpn for noticing the bug. 2. Widths of the columns were off in some cases. The fixed tables look like that: <img width="1775" alt="Screenshot 2023-06-22 at 19 52 13" src="https://github.com/elastic/kibana/assets/7359339/53475848-3238-4866-af70-080b8acd1f9e"> ## Full comparison | **BEFORE** | **AFTER** | |:------------:|:-----------:| |  |  | ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - elastic/security-docs#3478
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Jun 27, 2023
**Epic:** elastic/security-team#6032 (internal) **Related to:** elastic#159875 ## Summary In this PR we: - add a text block to the dashboard itself with helpful info about it - fix the 4 tables at the bottom of the dashboard - add unit tests for the dashboard's source `.json` files ## Text block <img width="1792" alt="Screenshot 2023-06-22 at 20 15 30" src="https://github.com/elastic/kibana/assets/7359339/55d267b7-7c39-4cdf-a917-fd17e9231a59"> ## Tables There were two issues with the tables: 1. When having the same prebuilt rules installed in two or more Kibana spaces, sorting in a table could break if the table rendered two different rules with the same name. It has been fixed by making the rule ID the first field in the table and making a few minor tweaks. Thanks to @maximpn for noticing the bug. 2. Widths of the columns were off in some cases. The fixed tables look like that: <img width="1775" alt="Screenshot 2023-06-22 at 19 52 13" src="https://github.com/elastic/kibana/assets/7359339/53475848-3238-4866-af70-080b8acd1f9e"> ## Full comparison | **BEFORE** | **AFTER** | |:------------:|:-----------:| |  |  | ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - elastic/security-docs#3478 (cherry picked from commit f622809)
kibanamachine
referenced
this issue
in elastic/kibana
Jun 27, 2023
…60617) # Backport This will backport the following commits from `main` to `8.9`: - [[Security Solution] Fix rule monitoring dashboard (#160316)](#160316) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Georgii Gorbachev","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-06-27T12:45:12Z","message":"[Security Solution] Fix rule monitoring dashboard (#160316)\n\n**Epic:** https://github.com/elastic/security-team/issues/6032\r\n(internal)\r\n**Related to:** https://github.com/elastic/kibana/pull/159875\r\n\r\n## Summary\r\n\r\nIn this PR we:\r\n\r\n- add a text block to the dashboard itself with helpful info about it\r\n- fix the 4 tables at the bottom of the dashboard\r\n- add unit tests for the dashboard's source `.json` files\r\n\r\n## Text block\r\n\r\n<img width=\"1792\" alt=\"Screenshot 2023-06-22 at 20 15 30\"\r\nsrc=\"https://github.com/elastic/kibana/assets/7359339/55d267b7-7c39-4cdf-a917-fd17e9231a59\">\r\n\r\n## Tables\r\n\r\nThere were two issues with the tables:\r\n\r\n1. When having the same prebuilt rules installed in two or more Kibana\r\nspaces, sorting in a table could break if the table rendered two\r\ndifferent rules with the same name. It has been fixed by making the rule\r\nID the first field in the table and making a few minor tweaks. Thanks to\r\n@maximpn for noticing the bug.\r\n2. Widths of the columns were off in some cases.\r\n\r\nThe fixed tables look like that:\r\n\r\n<img width=\"1775\" alt=\"Screenshot 2023-06-22 at 19 52 13\"\r\nsrc=\"https://github.com/elastic/kibana/assets/7359339/53475848-3238-4866-af70-080b8acd1f9e\">\r\n\r\n## Full comparison\r\n\r\n| **BEFORE** | **AFTER** |\r\n|:------------:|:-----------:|\r\n| \r\n| \r\n|\r\n\r\n\r\n### Checklist\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [x]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n - https://github.com/elastic/security-docs/issues/3478","sha":"f622809ee824a3341d767e7174177fbe7ac9578b","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:skip","impact:high","Team:Detections and Resp","Team: SecuritySolution","Feature:Rule Monitoring","Team:Detection Rule Management","v8.9.0","v8.10.0"],"number":160316,"url":"https://github.com/elastic/kibana/pull/160316","mergeCommit":{"message":"[Security Solution] Fix rule monitoring dashboard (#160316)\n\n**Epic:** https://github.com/elastic/security-team/issues/6032\r\n(internal)\r\n**Related to:** https://github.com/elastic/kibana/pull/159875\r\n\r\n## Summary\r\n\r\nIn this PR we:\r\n\r\n- add a text block to the dashboard itself with helpful info about it\r\n- fix the 4 tables at the bottom of the dashboard\r\n- add unit tests for the dashboard's source `.json` files\r\n\r\n## Text block\r\n\r\n<img width=\"1792\" alt=\"Screenshot 2023-06-22 at 20 15 30\"\r\nsrc=\"https://github.com/elastic/kibana/assets/7359339/55d267b7-7c39-4cdf-a917-fd17e9231a59\">\r\n\r\n## Tables\r\n\r\nThere were two issues with the tables:\r\n\r\n1. When having the same prebuilt rules installed in two or more Kibana\r\nspaces, sorting in a table could break if the table rendered two\r\ndifferent rules with the same name. It has been fixed by making the rule\r\nID the first field in the table and making a few minor tweaks. Thanks to\r\n@maximpn for noticing the bug.\r\n2. Widths of the columns were off in some cases.\r\n\r\nThe fixed tables look like that:\r\n\r\n<img width=\"1775\" alt=\"Screenshot 2023-06-22 at 19 52 13\"\r\nsrc=\"https://github.com/elastic/kibana/assets/7359339/53475848-3238-4866-af70-080b8acd1f9e\">\r\n\r\n## Full comparison\r\n\r\n| **BEFORE** | **AFTER** |\r\n|:------------:|:-----------:|\r\n| \r\n| \r\n|\r\n\r\n\r\n### Checklist\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [x]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n - https://github.com/elastic/security-docs/issues/3478","sha":"f622809ee824a3341d767e7174177fbe7ac9578b"}},"sourceBranch":"main","suggestedTargetBranches":["8.9"],"targetPullRequestStates":[{"branch":"8.9","label":"v8.9.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/160316","number":160316,"mergeCommit":{"message":"[Security Solution] Fix rule monitoring dashboard (#160316)\n\n**Epic:** https://github.com/elastic/security-team/issues/6032\r\n(internal)\r\n**Related to:** https://github.com/elastic/kibana/pull/159875\r\n\r\n## Summary\r\n\r\nIn this PR we:\r\n\r\n- add a text block to the dashboard itself with helpful info about it\r\n- fix the 4 tables at the bottom of the dashboard\r\n- add unit tests for the dashboard's source `.json` files\r\n\r\n## Text block\r\n\r\n<img width=\"1792\" alt=\"Screenshot 2023-06-22 at 20 15 30\"\r\nsrc=\"https://github.com/elastic/kibana/assets/7359339/55d267b7-7c39-4cdf-a917-fd17e9231a59\">\r\n\r\n## Tables\r\n\r\nThere were two issues with the tables:\r\n\r\n1. When having the same prebuilt rules installed in two or more Kibana\r\nspaces, sorting in a table could break if the table rendered two\r\ndifferent rules with the same name. It has been fixed by making the rule\r\nID the first field in the table and making a few minor tweaks. Thanks to\r\n@maximpn for noticing the bug.\r\n2. Widths of the columns were off in some cases.\r\n\r\nThe fixed tables look like that:\r\n\r\n<img width=\"1775\" alt=\"Screenshot 2023-06-22 at 19 52 13\"\r\nsrc=\"https://github.com/elastic/kibana/assets/7359339/53475848-3238-4866-af70-080b8acd1f9e\">\r\n\r\n## Full comparison\r\n\r\n| **BEFORE** | **AFTER** |\r\n|:------------:|:-----------:|\r\n| \r\n| \r\n|\r\n\r\n\r\n### Checklist\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [x]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n - https://github.com/elastic/security-docs/issues/3478","sha":"f622809ee824a3341d767e7174177fbe7ac9578b"}}]}] BACKPORT--> Co-authored-by: Georgii Gorbachev <[email protected]>
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Ticket: https://github.com/elastic/security-team/issues/6032 (internal)
PRs: elastic/kibana#159875, elastic/kibana#160316
Summary
In elastic/kibana#159875 we added a new Kibana dashboard called
[Elastic Security] Detection rule monitoring. All the details about it should be in the PR description, but please ping me if you have any questions.We'd like the dashboard to be mentioned in the docs so that our users become aware of it and start using it for monitoring rules and troubleshooting issues with the Detection Engine.
The API endpoint added in the above-mentioned PR doesn't need to be documented.
Known issue
A "known issue" should be added to the release notes: the number of
Rule executionson the dashboard will not always be equal to the sum of theSucceeded+Warning+Failednumbers. This is normal and the reason for that is the following: any rule can write multiple statuses per each execution. One of the typical cases is gap detection: if a rule detects a gap in rule execution, it will first write an intermediateFailedstatus and continue to run, and then it will write a final status (e.g. could beWarning) before finishing its execution.The text was updated successfully, but these errors were encountered: