[Platform] Migrate siem-detection-engine-rule-actions saved object ids to references array #113278
Closed
2 tasks done
Assignees
Labels
Team:Security Solution Platform
Security Solution Platform Team
Comments
FrankHassanabad
added
7.16 candidate
Team:Security Solution Platform
FrankHassanabad
added a commit
that referenced
this issue
Oct 6, 2021
…lertId and actions to saved object references array (#113577) ## Summary Fixes #113278 * Migrates the legacy `siem-detection-engine-rule-actions` `ruleAlertId` and `actions` to saved object references arrays * Adds an e2e test for `siem-detection-engine-rule-actions` * Updates the types to work with the migrations and the new and old data structures. * Decouples and removes reliance on alerting within the types since we do not want development of alerting to get in the way of legacy things and have migration changes by accident. * Updates the REST interface and code to produce post migration data structures. Removes some types and code where w can since those parts are no longer needed/used. * Adds `actionRef` to the mapping Before migration you should see data structures like this if you query: ```json GET .kibana/_search { "query": { "term": { "type": { "value": "siem-detection-engine-rule-actions" } } } } ``` ```json { "siem-detection-engine-rule-actions": { "ruleAlertId": "fb1046a0-0452-11ec-9b15-d13d79d162f3", <-- ruleAlertId which we want in the references array and removed "actions": [ { "action_type_id": ".slack", "id": "f6e64c00-0452-11ec-9b15-d13d79d162f3", <-- id which we want in the references array and removed "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "group": "default" } ], "ruleThrottle": "7d", "alertThrottle": "7d" }, "type": "siem-detection-engine-rule-actions", "references": [], <-- Array is empty which instead needs the id's of alerts and actions "migrationVersion": { "siem-detection-engine-rule-actions": "7.11.2" }, "coreMigrationVersion": "7.14.0", "updated_at": "2021-09-15T22:18:48.369Z" } ``` After migration you should see data structures like this: ```json { "siem-detection-engine-rule-actions": { "actions": [ { "action_type_id": ".slack", "actionRef" : "action_0", <-- We use the name and "actionRef" to be consistent with kibana alerting "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "group": "default" } ], "ruleThrottle": "7d", "alertThrottle": "7d" }, "type": "siem-detection-engine-rule-actions", "references" : [ { "name" : "alert_0", <-- Name is "alert_0" "id" : "fb1046a0-0452-11ec-9b15-d13d79d162f3", <-- Alert id is now here "type" : "alert" <-- Type should be "alert" }, { "name" : "action_0", <-- Name is "action_0" and should be the same as kibana alerting names theirs for consistencty "id" : "f6e64c00-0452-11ec-9b15-d13d79d162f3", <-- Id of the action is now here. "type" : "action" <-- Type should be "action" } ], "migrationVersion": { "siem-detection-engine-rule-actions": "7.16.0" }, "coreMigrationVersion": "8.0.0", "updated_at": "2021-09-15T22:18:48.369Z" } ``` Manual testing --- There are e2e tests but for any manual testing or verification you can do the following: If you have a 7.14.0 system and can migrate it forward that is the most straight forward way to ensure this does migrate correctly and forward. You should see that the legacy notification system still operates as expected. If you are a developer off of master and want to test different scenarios then this section is for below as it is more involved and harder to do but goes into more depth: * Create a rule and activate it normally within security_solution * Do not add actions to the rule at this point as we are exercising the older legacy system. However, you want at least one action configured such as a slack notification. * Within dev tools do a query for all your actions and grab one of the `_id` of them without their prefix: ```json # See all your actions GET .kibana/_search { "query": { "term": { "type": "action" } } } ``` Mine was `"_id" : "action:879e8ff0-1be1-11ec-a722-83da1c22a481"`, so I will be copying the ID of `879e8ff0-1be1-11ec-a722-83da1c22a481` Go to the file `detection_engine/scripts/legacy_notifications/one_action.json` and add this id to the file. Something like this: ```json { "name": "Legacy notification with one action", "interval": "1m", <--- You can use whatever you want. Real values are "1h", "1d", "1w". I use "1m" for testing purposes. "actions": [ { "id": "879e8ff0-1be1-11ec-a722-83da1c22a481", <--- My action id "group": "default", "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "actionTypeId": ".slack" <--- I am a slack action id type. } ] } ``` Query for an alert you want to add manually add back a legacy notification to it. Such as: ```json # See all your siem.signals alert types and choose one GET .kibana/_search { "query": { "term": { "alert.alertTypeId": "siem.signals" } } } ``` Grab the `_id` without the alert prefix. For mine this was `933ca720-1be1-11ec-a722-83da1c22a481` Within the directory of detection_engine/scripts execute the script: ```json ./post_legacy_notification.sh 933ca720-1be1-11ec-a722-83da1c22a481 { "ok": "acknowledged" } ``` which is going to do a few things. See the file `detection_engine/routes/rules/legacy_create_legacy_notification.ts` for the definition of the route and what it does in full, but we should notice that we have now: Created a legacy side car action object of type `siem-detection-engine-rule-actions` you can see in dev tools: ```json # See the actions "side car" which are part of the legacy notification system. GET .kibana/_search { "query": { "term": { "type": { "value": "siem-detection-engine-rule-actions" } } } } ``` Take note that this actually creates the rule migrated since this PR updated the code to produce new side cars. So we have to use some scripting to change the actions to utilize the old format. However, before continuing you should verify that this does fire correctly and that the new format is working as expected. After that replace the structure with the older structure like so below and downgrade the migration version so that we can restart Kibana and ensure that this does migrate correctly forward: ```json # Get your id of your rules side car above and then use this script to downgrade the data structure POST .kibana/_update/siem-detection-engine-rule-actions:210f4c90-2233-11ec-98c6-ed2574588902 { "script" : { "source": """ ctx._source.migrationVersion['siem-detection-engine-rule-actions'] = "7.15.0"; ctx._source['siem-detection-engine-rule-actions'].actions[0].id = ctx._source.references[1].id; ctx._source['siem-detection-engine-rule-actions'].actions[0].remove('actionRef'); ctx._source['siem-detection-engine-rule-actions'].ruleAlertId = ctx._source.references[0].id; ctx._source.references.remove(0); ctx._source.references.remove(0); """, "lang": "painless" } } ``` Restart Kibana and now it should be migrated correctly and the system should fire the notifications as expected. You shouldn't see any errors in your console. In the scripts folder execute the `find_rules.sh` and expect to see actions like so in the rule with the `id` still in the REST interface and we shouldn't see `actionRef` within the actions: ```json "actions": [{ "id": "42534430-2092-11ec-99a6-05d79563c01a", "group": "default", "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "action_type_id": ".slack" }], ``` Take the rule id and query that as well using `./get_rule_by_id.sh` and verify that the action also looks the same and is present within the rule. You can also verify all of this within the UI's as well for rules to ensure the action is still present and as we expect it to be and work. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Oct 6, 2021
…lertId and actions to saved object references array (elastic#113577) ## Summary Fixes elastic#113278 * Migrates the legacy `siem-detection-engine-rule-actions` `ruleAlertId` and `actions` to saved object references arrays * Adds an e2e test for `siem-detection-engine-rule-actions` * Updates the types to work with the migrations and the new and old data structures. * Decouples and removes reliance on alerting within the types since we do not want development of alerting to get in the way of legacy things and have migration changes by accident. * Updates the REST interface and code to produce post migration data structures. Removes some types and code where w can since those parts are no longer needed/used. * Adds `actionRef` to the mapping Before migration you should see data structures like this if you query: ```json GET .kibana/_search { "query": { "term": { "type": { "value": "siem-detection-engine-rule-actions" } } } } ``` ```json { "siem-detection-engine-rule-actions": { "ruleAlertId": "fb1046a0-0452-11ec-9b15-d13d79d162f3", <-- ruleAlertId which we want in the references array and removed "actions": [ { "action_type_id": ".slack", "id": "f6e64c00-0452-11ec-9b15-d13d79d162f3", <-- id which we want in the references array and removed "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "group": "default" } ], "ruleThrottle": "7d", "alertThrottle": "7d" }, "type": "siem-detection-engine-rule-actions", "references": [], <-- Array is empty which instead needs the id's of alerts and actions "migrationVersion": { "siem-detection-engine-rule-actions": "7.11.2" }, "coreMigrationVersion": "7.14.0", "updated_at": "2021-09-15T22:18:48.369Z" } ``` After migration you should see data structures like this: ```json { "siem-detection-engine-rule-actions": { "actions": [ { "action_type_id": ".slack", "actionRef" : "action_0", <-- We use the name and "actionRef" to be consistent with kibana alerting "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "group": "default" } ], "ruleThrottle": "7d", "alertThrottle": "7d" }, "type": "siem-detection-engine-rule-actions", "references" : [ { "name" : "alert_0", <-- Name is "alert_0" "id" : "fb1046a0-0452-11ec-9b15-d13d79d162f3", <-- Alert id is now here "type" : "alert" <-- Type should be "alert" }, { "name" : "action_0", <-- Name is "action_0" and should be the same as kibana alerting names theirs for consistencty "id" : "f6e64c00-0452-11ec-9b15-d13d79d162f3", <-- Id of the action is now here. "type" : "action" <-- Type should be "action" } ], "migrationVersion": { "siem-detection-engine-rule-actions": "7.16.0" }, "coreMigrationVersion": "8.0.0", "updated_at": "2021-09-15T22:18:48.369Z" } ``` Manual testing --- There are e2e tests but for any manual testing or verification you can do the following: If you have a 7.14.0 system and can migrate it forward that is the most straight forward way to ensure this does migrate correctly and forward. You should see that the legacy notification system still operates as expected. If you are a developer off of master and want to test different scenarios then this section is for below as it is more involved and harder to do but goes into more depth: * Create a rule and activate it normally within security_solution * Do not add actions to the rule at this point as we are exercising the older legacy system. However, you want at least one action configured such as a slack notification. * Within dev tools do a query for all your actions and grab one of the `_id` of them without their prefix: ```json # See all your actions GET .kibana/_search { "query": { "term": { "type": "action" } } } ``` Mine was `"_id" : "action:879e8ff0-1be1-11ec-a722-83da1c22a481"`, so I will be copying the ID of `879e8ff0-1be1-11ec-a722-83da1c22a481` Go to the file `detection_engine/scripts/legacy_notifications/one_action.json` and add this id to the file. Something like this: ```json { "name": "Legacy notification with one action", "interval": "1m", <--- You can use whatever you want. Real values are "1h", "1d", "1w". I use "1m" for testing purposes. "actions": [ { "id": "879e8ff0-1be1-11ec-a722-83da1c22a481", <--- My action id "group": "default", "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "actionTypeId": ".slack" <--- I am a slack action id type. } ] } ``` Query for an alert you want to add manually add back a legacy notification to it. Such as: ```json # See all your siem.signals alert types and choose one GET .kibana/_search { "query": { "term": { "alert.alertTypeId": "siem.signals" } } } ``` Grab the `_id` without the alert prefix. For mine this was `933ca720-1be1-11ec-a722-83da1c22a481` Within the directory of detection_engine/scripts execute the script: ```json ./post_legacy_notification.sh 933ca720-1be1-11ec-a722-83da1c22a481 { "ok": "acknowledged" } ``` which is going to do a few things. See the file `detection_engine/routes/rules/legacy_create_legacy_notification.ts` for the definition of the route and what it does in full, but we should notice that we have now: Created a legacy side car action object of type `siem-detection-engine-rule-actions` you can see in dev tools: ```json # See the actions "side car" which are part of the legacy notification system. GET .kibana/_search { "query": { "term": { "type": { "value": "siem-detection-engine-rule-actions" } } } } ``` Take note that this actually creates the rule migrated since this PR updated the code to produce new side cars. So we have to use some scripting to change the actions to utilize the old format. However, before continuing you should verify that this does fire correctly and that the new format is working as expected. After that replace the structure with the older structure like so below and downgrade the migration version so that we can restart Kibana and ensure that this does migrate correctly forward: ```json # Get your id of your rules side car above and then use this script to downgrade the data structure POST .kibana/_update/siem-detection-engine-rule-actions:210f4c90-2233-11ec-98c6-ed2574588902 { "script" : { "source": """ ctx._source.migrationVersion['siem-detection-engine-rule-actions'] = "7.15.0"; ctx._source['siem-detection-engine-rule-actions'].actions[0].id = ctx._source.references[1].id; ctx._source['siem-detection-engine-rule-actions'].actions[0].remove('actionRef'); ctx._source['siem-detection-engine-rule-actions'].ruleAlertId = ctx._source.references[0].id; ctx._source.references.remove(0); ctx._source.references.remove(0); """, "lang": "painless" } } ``` Restart Kibana and now it should be migrated correctly and the system should fire the notifications as expected. You shouldn't see any errors in your console. In the scripts folder execute the `find_rules.sh` and expect to see actions like so in the rule with the `id` still in the REST interface and we shouldn't see `actionRef` within the actions: ```json "actions": [{ "id": "42534430-2092-11ec-99a6-05d79563c01a", "group": "default", "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "action_type_id": ".slack" }], ``` Take the rule id and query that as well using `./get_rule_by_id.sh` and verify that the action also looks the same and is present within the rule. You can also verify all of this within the UI's as well for rules to ensure the action is still present and as we expect it to be and work. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
kibanamachine
added a commit
that referenced
this issue
Oct 7, 2021
…lertId and actions to saved object references array (#113577) (#114201) ## Summary Fixes #113278 * Migrates the legacy `siem-detection-engine-rule-actions` `ruleAlertId` and `actions` to saved object references arrays * Adds an e2e test for `siem-detection-engine-rule-actions` * Updates the types to work with the migrations and the new and old data structures. * Decouples and removes reliance on alerting within the types since we do not want development of alerting to get in the way of legacy things and have migration changes by accident. * Updates the REST interface and code to produce post migration data structures. Removes some types and code where w can since those parts are no longer needed/used. * Adds `actionRef` to the mapping Before migration you should see data structures like this if you query: ```json GET .kibana/_search { "query": { "term": { "type": { "value": "siem-detection-engine-rule-actions" } } } } ``` ```json { "siem-detection-engine-rule-actions": { "ruleAlertId": "fb1046a0-0452-11ec-9b15-d13d79d162f3", <-- ruleAlertId which we want in the references array and removed "actions": [ { "action_type_id": ".slack", "id": "f6e64c00-0452-11ec-9b15-d13d79d162f3", <-- id which we want in the references array and removed "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "group": "default" } ], "ruleThrottle": "7d", "alertThrottle": "7d" }, "type": "siem-detection-engine-rule-actions", "references": [], <-- Array is empty which instead needs the id's of alerts and actions "migrationVersion": { "siem-detection-engine-rule-actions": "7.11.2" }, "coreMigrationVersion": "7.14.0", "updated_at": "2021-09-15T22:18:48.369Z" } ``` After migration you should see data structures like this: ```json { "siem-detection-engine-rule-actions": { "actions": [ { "action_type_id": ".slack", "actionRef" : "action_0", <-- We use the name and "actionRef" to be consistent with kibana alerting "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "group": "default" } ], "ruleThrottle": "7d", "alertThrottle": "7d" }, "type": "siem-detection-engine-rule-actions", "references" : [ { "name" : "alert_0", <-- Name is "alert_0" "id" : "fb1046a0-0452-11ec-9b15-d13d79d162f3", <-- Alert id is now here "type" : "alert" <-- Type should be "alert" }, { "name" : "action_0", <-- Name is "action_0" and should be the same as kibana alerting names theirs for consistencty "id" : "f6e64c00-0452-11ec-9b15-d13d79d162f3", <-- Id of the action is now here. "type" : "action" <-- Type should be "action" } ], "migrationVersion": { "siem-detection-engine-rule-actions": "7.16.0" }, "coreMigrationVersion": "8.0.0", "updated_at": "2021-09-15T22:18:48.369Z" } ``` Manual testing --- There are e2e tests but for any manual testing or verification you can do the following: If you have a 7.14.0 system and can migrate it forward that is the most straight forward way to ensure this does migrate correctly and forward. You should see that the legacy notification system still operates as expected. If you are a developer off of master and want to test different scenarios then this section is for below as it is more involved and harder to do but goes into more depth: * Create a rule and activate it normally within security_solution * Do not add actions to the rule at this point as we are exercising the older legacy system. However, you want at least one action configured such as a slack notification. * Within dev tools do a query for all your actions and grab one of the `_id` of them without their prefix: ```json # See all your actions GET .kibana/_search { "query": { "term": { "type": "action" } } } ``` Mine was `"_id" : "action:879e8ff0-1be1-11ec-a722-83da1c22a481"`, so I will be copying the ID of `879e8ff0-1be1-11ec-a722-83da1c22a481` Go to the file `detection_engine/scripts/legacy_notifications/one_action.json` and add this id to the file. Something like this: ```json { "name": "Legacy notification with one action", "interval": "1m", <--- You can use whatever you want. Real values are "1h", "1d", "1w". I use "1m" for testing purposes. "actions": [ { "id": "879e8ff0-1be1-11ec-a722-83da1c22a481", <--- My action id "group": "default", "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "actionTypeId": ".slack" <--- I am a slack action id type. } ] } ``` Query for an alert you want to add manually add back a legacy notification to it. Such as: ```json # See all your siem.signals alert types and choose one GET .kibana/_search { "query": { "term": { "alert.alertTypeId": "siem.signals" } } } ``` Grab the `_id` without the alert prefix. For mine this was `933ca720-1be1-11ec-a722-83da1c22a481` Within the directory of detection_engine/scripts execute the script: ```json ./post_legacy_notification.sh 933ca720-1be1-11ec-a722-83da1c22a481 { "ok": "acknowledged" } ``` which is going to do a few things. See the file `detection_engine/routes/rules/legacy_create_legacy_notification.ts` for the definition of the route and what it does in full, but we should notice that we have now: Created a legacy side car action object of type `siem-detection-engine-rule-actions` you can see in dev tools: ```json # See the actions "side car" which are part of the legacy notification system. GET .kibana/_search { "query": { "term": { "type": { "value": "siem-detection-engine-rule-actions" } } } } ``` Take note that this actually creates the rule migrated since this PR updated the code to produce new side cars. So we have to use some scripting to change the actions to utilize the old format. However, before continuing you should verify that this does fire correctly and that the new format is working as expected. After that replace the structure with the older structure like so below and downgrade the migration version so that we can restart Kibana and ensure that this does migrate correctly forward: ```json # Get your id of your rules side car above and then use this script to downgrade the data structure POST .kibana/_update/siem-detection-engine-rule-actions:210f4c90-2233-11ec-98c6-ed2574588902 { "script" : { "source": """ ctx._source.migrationVersion['siem-detection-engine-rule-actions'] = "7.15.0"; ctx._source['siem-detection-engine-rule-actions'].actions[0].id = ctx._source.references[1].id; ctx._source['siem-detection-engine-rule-actions'].actions[0].remove('actionRef'); ctx._source['siem-detection-engine-rule-actions'].ruleAlertId = ctx._source.references[0].id; ctx._source.references.remove(0); ctx._source.references.remove(0); """, "lang": "painless" } } ``` Restart Kibana and now it should be migrated correctly and the system should fire the notifications as expected. You shouldn't see any errors in your console. In the scripts folder execute the `find_rules.sh` and expect to see actions like so in the rule with the `id` still in the REST interface and we shouldn't see `actionRef` within the actions: ```json "actions": [{ "id": "42534430-2092-11ec-99a6-05d79563c01a", "group": "default", "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "action_type_id": ".slack" }], ``` Take the rule id and query that as well using `./get_rule_by_id.sh` and verify that the action also looks the same and is present within the rule. You can also verify all of this within the UI's as well for rules to ensure the action is still present and as we expect it to be and work. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Frank Hassanabad <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Summary
This ticket is a child of https://github.com/elastic/security-team/issues/1758. It tracks the work involved with exporting all SOs necessary to migrate the rule to the new actions SOs.
Acceptance Criteria
Test Cases
The text was updated successfully, but these errors were encountered: