[Security Solution] Add hyper link to migration documentation for users for the Security Solution Legacy system. #113055
Assignees
Labels
Team:Detections and Resp
Security Detection Response Team
Team:Security Solution Platform
Security Solution Platform Team
v7.16.0
Comments
FrankHassanabad
added
Team:Detections and Resp
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
yctercero
added
the
Team:Security Solution Platform
FrankHassanabad
added a commit
that referenced
this issue
Jan 12, 2022
…otifications by migrating them before the deletion (#122610) ## Summary Bug: #122456, #113055 Previously when we brought back the legacy notifications we did not migrate them on delete so the legacy notifications end up becoming dangling alerts that run forever with warns in your log files like so: ```sh [2022-01-06T10:09:16.593-07:00][ERROR][plugins.alerting] Executing Rule frank-test:siem.notifications:549ac8c0-6f11-11ec-b6be-193e9dcb2837 has resulted in Error: Saved object [alert/36fe8520-6e7d-11ec-83a2-150f18947658] not found ``` This fixes the bug by migrating the actions first before deleting them in the routes of: * `delete_rules_route.ts` * `delete_rules_bulk_route.ts` That allows the action to be returned in the response body on successful delete and also deletes the legacy notification. I could not find a good way to turn off the legacy notification from within its self so for now I updated the logging message to include instructions on how to remove legacy notifications that are dangling or to upgrade/update from legacy to non-legacy in the log messages. Also, this changes the line of code for deleting legacy actions from: ```ts rulesClient.find({ options: { hasReference: { <-- This currently will delete the first found rule that happens to have a reference of the `rule.id` That should be only legacy notifications today but tomorrow or later could be a different rule by accident. type: 'alert', id: rule.id, }, }, }), ``` to ```ts rulesClient.find({ options: { filter: 'alert.attributes.alertTypeId:(siem.notifications)', <--- This filter should ensure we only delete `siem.notifications` as extra safety measure. hasReference: { type: 'alert', id: rule.id, }, }, }), ``` **Manual testing** Either you can use the `internal` rule route endpoints or do a true upgrade test scenario. For these manual testing steps I am going to outline the developer fast way of adding a legacy rule and then deleting it from there. Create a rule and activate it normally within security_solution: <img width="1197" alt="create_rule" src="https://user-images.githubusercontent.com/1151048/148859970-e7587c2f-de7a-4f4c-9280-4acdaeac03dd.png"> Do not add actions to the rule at this point as we will first exercise the older legacy system. However, you want at least one action configured such as a slack notification: <img width="575" alt="define_connector" src="https://user-images.githubusercontent.com/1151048/148859992-ad83aa8b-7c86-49ad-bae9-fcc291f6824a.png"> Within dev tools do a query for all your actions and grab one of the _id of them without their prefix: ```sh GET .kibana/_search { "query": { "term": { "type": "action" } } } ``` Go to the file detection_engine/scripts/legacy_notifications/one_action.json and add this id to the file. Something like this: Mine was `_id : action:879e8ff0-1be1-11ec-a722-83da1c22a481`, so I will be copying the ID of `879e8ff0-1be1-11ec-a722-83da1c22a481`: ```json { "name": "Legacy notification with one action", "interval": "1m", <--- You can use whatever you want. Real values are "1h", "1d", "1w". I use "1m" for testing purposes. "actions": [ { "id": "879e8ff0-1be1-11ec-a722-83da1c22a481", <--- My action id "group": "default", "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "actionTypeId": ".slack" <--- I am a slack action id type. } ] } ``` Query for an alert you want to add manually add back a legacy notification to it. Such as: ```json GET .kibana/_search { "query": { "term": { "alert.alertTypeId": "siem.queryRule" } } } ``` Grab the `_id` without the alert prefix. For mine this was `933ca720-1be1-11ec-a722-83da1c22a481`: Within the directory of detection_engine/scripts execute the script: ```sh ./post_legacy_notification.sh 933ca720-1be1-11ec-a722-83da1c22a481 { "ok": "acknowledged" } ``` which is going to do a few things. See the file detection_engine/routes/rules/legacy_create_legacy_notification.ts for the definition of the route and what it does in full, but we should notice that we have now have actions on the rule: <img width="1050" alt="edit_rule" src="https://user-images.githubusercontent.com/1151048/148860221-4ee3b2b4-64a4-4e4e-b762-611b0e5db1d1.png"> If you query for legacy notifications you should see the one you have: ```sh GET .kibana/_search { "query": { "term": { "alert.alertTypeId": "siem.notifications" } } } ``` If you query for saved objects associated such as `siem-detection-engine-rule-actions`: ```sh GET .kibana/_search { "query": { "term": { "type": { "value": "siem-detection-engine-rule-actions" } } } } ``` You should also see 1. After you delete the rule, both should be gone if you repeat the query above now. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
FrankHassanabad
added a commit
to FrankHassanabad/kibana
that referenced
this issue
Jan 13, 2022
…otifications by migrating them before the deletion (elastic#122610) ## Summary Bug: elastic#122456, elastic#113055 Previously when we brought back the legacy notifications we did not migrate them on delete so the legacy notifications end up becoming dangling alerts that run forever with warns in your log files like so: ```sh [2022-01-06T10:09:16.593-07:00][ERROR][plugins.alerting] Executing Rule frank-test:siem.notifications:549ac8c0-6f11-11ec-b6be-193e9dcb2837 has resulted in Error: Saved object [alert/36fe8520-6e7d-11ec-83a2-150f18947658] not found ``` This fixes the bug by migrating the actions first before deleting them in the routes of: * `delete_rules_route.ts` * `delete_rules_bulk_route.ts` That allows the action to be returned in the response body on successful delete and also deletes the legacy notification. I could not find a good way to turn off the legacy notification from within its self so for now I updated the logging message to include instructions on how to remove legacy notifications that are dangling or to upgrade/update from legacy to non-legacy in the log messages. Also, this changes the line of code for deleting legacy actions from: ```ts rulesClient.find({ options: { hasReference: { <-- This currently will delete the first found rule that happens to have a reference of the `rule.id` That should be only legacy notifications today but tomorrow or later could be a different rule by accident. type: 'alert', id: rule.id, }, }, }), ``` to ```ts rulesClient.find({ options: { filter: 'alert.attributes.alertTypeId:(siem.notifications)', <--- This filter should ensure we only delete `siem.notifications` as extra safety measure. hasReference: { type: 'alert', id: rule.id, }, }, }), ``` **Manual testing** Either you can use the `internal` rule route endpoints or do a true upgrade test scenario. For these manual testing steps I am going to outline the developer fast way of adding a legacy rule and then deleting it from there. Create a rule and activate it normally within security_solution: <img width="1197" alt="create_rule" src="https://user-images.githubusercontent.com/1151048/148859970-e7587c2f-de7a-4f4c-9280-4acdaeac03dd.png"> Do not add actions to the rule at this point as we will first exercise the older legacy system. However, you want at least one action configured such as a slack notification: <img width="575" alt="define_connector" src="https://user-images.githubusercontent.com/1151048/148859992-ad83aa8b-7c86-49ad-bae9-fcc291f6824a.png"> Within dev tools do a query for all your actions and grab one of the _id of them without their prefix: ```sh GET .kibana/_search { "query": { "term": { "type": "action" } } } ``` Go to the file detection_engine/scripts/legacy_notifications/one_action.json and add this id to the file. Something like this: Mine was `_id : action:879e8ff0-1be1-11ec-a722-83da1c22a481`, so I will be copying the ID of `879e8ff0-1be1-11ec-a722-83da1c22a481`: ```json { "name": "Legacy notification with one action", "interval": "1m", <--- You can use whatever you want. Real values are "1h", "1d", "1w". I use "1m" for testing purposes. "actions": [ { "id": "879e8ff0-1be1-11ec-a722-83da1c22a481", <--- My action id "group": "default", "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "actionTypeId": ".slack" <--- I am a slack action id type. } ] } ``` Query for an alert you want to add manually add back a legacy notification to it. Such as: ```json GET .kibana/_search { "query": { "term": { "alert.alertTypeId": "siem.queryRule" } } } ``` Grab the `_id` without the alert prefix. For mine this was `933ca720-1be1-11ec-a722-83da1c22a481`: Within the directory of detection_engine/scripts execute the script: ```sh ./post_legacy_notification.sh 933ca720-1be1-11ec-a722-83da1c22a481 { "ok": "acknowledged" } ``` which is going to do a few things. See the file detection_engine/routes/rules/legacy_create_legacy_notification.ts for the definition of the route and what it does in full, but we should notice that we have now have actions on the rule: <img width="1050" alt="edit_rule" src="https://user-images.githubusercontent.com/1151048/148860221-4ee3b2b4-64a4-4e4e-b762-611b0e5db1d1.png"> If you query for legacy notifications you should see the one you have: ```sh GET .kibana/_search { "query": { "term": { "alert.alertTypeId": "siem.notifications" } } } ``` If you query for saved objects associated such as `siem-detection-engine-rule-actions`: ```sh GET .kibana/_search { "query": { "term": { "type": { "value": "siem-detection-engine-rule-actions" } } } } ``` You should also see 1. After you delete the rule, both should be gone if you repeat the query above now. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit 25d1c66)
FrankHassanabad
added a commit
to FrankHassanabad/kibana
that referenced
this issue
Jan 13, 2022
…otifications by migrating them before the deletion (elastic#122610) ## Summary Bug: elastic#122456, elastic#113055 Previously when we brought back the legacy notifications we did not migrate them on delete so the legacy notifications end up becoming dangling alerts that run forever with warns in your log files like so: This fixes the bug by migrating the actions first before deleting them in the routes of: * * That allows the action to be returned in the response body on successful delete and also deletes the legacy notification. I could not find a good way to turn off the legacy notification from within its self so for now I updated the logging message to include instructions on how to remove legacy notifications that are dangling or to upgrade/update from legacy to non-legacy in the log messages. Also, this changes the line of code for deleting legacy actions from: rule.id to siem.notifications **Manual testing** Either you can use the rule route endpoints or do a true upgrade test scenario. For these manual testing steps I am going to outline the developer fast way of adding a legacy rule and then deleting it from there. Create a rule and activate it normally within security_solution: <img width=1197 alt=create_rule src=https://user-images.githubusercontent.com/1151048/148859970-e7587c2f-de7a-4f4c-9280-4acdaeac03dd.png> Do not add actions to the rule at this point as we will first exercise the older legacy system. However, you want at least one action configured such as a slack notification: <img width=575 alt=define_connector src=https://user-images.githubusercontent.com/1151048/148859992-ad83aa8b-7c86-49ad-bae9-fcc291f6824a.png> Within dev tools do a query for all your actions and grab one of the _id of them without their prefix: Go to the file detection_engine/scripts/legacy_notifications/one_action.json and add this id to the file. Something like this: Mine was , so I will be copying the ID of : Query for an alert you want to add manually add back a legacy notification to it. Such as: Grab the without the alert prefix. For mine this was : Within the directory of detection_engine/scripts execute the script: which is going to do a few things. See the file detection_engine/routes/rules/legacy_create_legacy_notification.ts for the definition of the route and what it does in full, but we should notice that we have now have actions on the rule: <img width=1050 alt=edit_rule src=https://user-images.githubusercontent.com/1151048/148860221-4ee3b2b4-64a4-4e4e-b762-611b0e5db1d1.png> If you query for legacy notifications you should see the one you have: If you query for saved objects associated such as : You should also see 1. After you delete the rule, both should be gone if you repeat the query above now. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
FrankHassanabad
added a commit
that referenced
this issue
Jan 13, 2022
…otifications by migrating them before the deletion (#122610) (#122904) ## Summary Bug: #122456, #113055 Previously when we brought back the legacy notifications we did not migrate them on delete so the legacy notifications end up becoming dangling alerts that run forever with warns in your log files like so: ```sh [2022-01-06T10:09:16.593-07:00][ERROR][plugins.alerting] Executing Rule frank-test:siem.notifications:549ac8c0-6f11-11ec-b6be-193e9dcb2837 has resulted in Error: Saved object [alert/36fe8520-6e7d-11ec-83a2-150f18947658] not found ``` This fixes the bug by migrating the actions first before deleting them in the routes of: * `delete_rules_route.ts` * `delete_rules_bulk_route.ts` That allows the action to be returned in the response body on successful delete and also deletes the legacy notification. I could not find a good way to turn off the legacy notification from within its self so for now I updated the logging message to include instructions on how to remove legacy notifications that are dangling or to upgrade/update from legacy to non-legacy in the log messages. Also, this changes the line of code for deleting legacy actions from: ```ts rulesClient.find({ options: { hasReference: { <-- This currently will delete the first found rule that happens to have a reference of the `rule.id` That should be only legacy notifications today but tomorrow or later could be a different rule by accident. type: 'alert', id: rule.id, }, }, }), ``` to ```ts rulesClient.find({ options: { filter: 'alert.attributes.alertTypeId:(siem.notifications)', <--- This filter should ensure we only delete `siem.notifications` as extra safety measure. hasReference: { type: 'alert', id: rule.id, }, }, }), ``` **Manual testing** Either you can use the `internal` rule route endpoints or do a true upgrade test scenario. For these manual testing steps I am going to outline the developer fast way of adding a legacy rule and then deleting it from there. Create a rule and activate it normally within security_solution: <img width="1197" alt="create_rule" src="https://user-images.githubusercontent.com/1151048/148859970-e7587c2f-de7a-4f4c-9280-4acdaeac03dd.png"> Do not add actions to the rule at this point as we will first exercise the older legacy system. However, you want at least one action configured such as a slack notification: <img width="575" alt="define_connector" src="https://user-images.githubusercontent.com/1151048/148859992-ad83aa8b-7c86-49ad-bae9-fcc291f6824a.png"> Within dev tools do a query for all your actions and grab one of the _id of them without their prefix: ```sh GET .kibana/_search { "query": { "term": { "type": "action" } } } ``` Go to the file detection_engine/scripts/legacy_notifications/one_action.json and add this id to the file. Something like this: Mine was `_id : action:879e8ff0-1be1-11ec-a722-83da1c22a481`, so I will be copying the ID of `879e8ff0-1be1-11ec-a722-83da1c22a481`: ```json { "name": "Legacy notification with one action", "interval": "1m", <--- You can use whatever you want. Real values are "1h", "1d", "1w". I use "1m" for testing purposes. "actions": [ { "id": "879e8ff0-1be1-11ec-a722-83da1c22a481", <--- My action id "group": "default", "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "actionTypeId": ".slack" <--- I am a slack action id type. } ] } ``` Query for an alert you want to add manually add back a legacy notification to it. Such as: ```json GET .kibana/_search { "query": { "term": { "alert.alertTypeId": "siem.queryRule" } } } ``` Grab the `_id` without the alert prefix. For mine this was `933ca720-1be1-11ec-a722-83da1c22a481`: Within the directory of detection_engine/scripts execute the script: ```sh ./post_legacy_notification.sh 933ca720-1be1-11ec-a722-83da1c22a481 { "ok": "acknowledged" } ``` which is going to do a few things. See the file detection_engine/routes/rules/legacy_create_legacy_notification.ts for the definition of the route and what it does in full, but we should notice that we have now have actions on the rule: <img width="1050" alt="edit_rule" src="https://user-images.githubusercontent.com/1151048/148860221-4ee3b2b4-64a4-4e4e-b762-611b0e5db1d1.png"> If you query for legacy notifications you should see the one you have: ```sh GET .kibana/_search { "query": { "term": { "alert.alertTypeId": "siem.notifications" } } } ``` If you query for saved objects associated such as `siem-detection-engine-rule-actions`: ```sh GET .kibana/_search { "query": { "term": { "type": { "value": "siem-detection-engine-rule-actions" } } } } ``` You should also see 1. After you delete the rule, both should be gone if you repeat the query above now. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit 25d1c66)
FrankHassanabad
added a commit
that referenced
this issue
Jan 13, 2022
…egacy notifications by migrating them before the deletion (#122610) (#122905) * [Security Solutions] Fixes deleting of alerts to also delete legacy notifications by migrating them before the deletion (#122610) ## Summary Bug: #122456, #113055 Previously when we brought back the legacy notifications we did not migrate them on delete so the legacy notifications end up becoming dangling alerts that run forever with warns in your log files like so: This fixes the bug by migrating the actions first before deleting them in the routes of: * * That allows the action to be returned in the response body on successful delete and also deletes the legacy notification. I could not find a good way to turn off the legacy notification from within its self so for now I updated the logging message to include instructions on how to remove legacy notifications that are dangling or to upgrade/update from legacy to non-legacy in the log messages. Also, this changes the line of code for deleting legacy actions from: rule.id to siem.notifications **Manual testing** Either you can use the rule route endpoints or do a true upgrade test scenario. For these manual testing steps I am going to outline the developer fast way of adding a legacy rule and then deleting it from there. Create a rule and activate it normally within security_solution: <img width=1197 alt=create_rule src=https://user-images.githubusercontent.com/1151048/148859970-e7587c2f-de7a-4f4c-9280-4acdaeac03dd.png> Do not add actions to the rule at this point as we will first exercise the older legacy system. However, you want at least one action configured such as a slack notification: <img width=575 alt=define_connector src=https://user-images.githubusercontent.com/1151048/148859992-ad83aa8b-7c86-49ad-bae9-fcc291f6824a.png> Within dev tools do a query for all your actions and grab one of the _id of them without their prefix: Go to the file detection_engine/scripts/legacy_notifications/one_action.json and add this id to the file. Something like this: Mine was , so I will be copying the ID of : Query for an alert you want to add manually add back a legacy notification to it. Such as: Grab the without the alert prefix. For mine this was : Within the directory of detection_engine/scripts execute the script: which is going to do a few things. See the file detection_engine/routes/rules/legacy_create_legacy_notification.ts for the definition of the route and what it does in full, but we should notice that we have now have actions on the rule: <img width=1050 alt=edit_rule src=https://user-images.githubusercontent.com/1151048/148860221-4ee3b2b4-64a4-4e4e-b762-611b0e5db1d1.png> If you query for legacy notifications you should see the one you have: If you query for saved objects associated such as : You should also see 1. After you delete the rule, both should be gone if you repeat the query above now. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios * Removed the log entry for the backport
|
Fixed by linked PR - that piece of code where we needed to add the link is no longer there in favor of other more detailed log messages. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add hyper link to migration documentation for users for the Security Solution Legacy system to the code of
legacyRulesNotificationAlertTypeThe text was updated successfully, but these errors were encountered: