[Security Solutions][Detection Engine] Adds exception lists to the saved object references when created or modified (part 1) #107064
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
FrankHassanabad
changed the title
FrankHassanabad
added
v8.0.0
v7.15.0
Team:Detections and Resp
FrankHassanabad
added
Feature:Rule Exceptions
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
FrankHassanabad
changed the title
|
@elasticmachine merge upstream |
|
Flakey test: #107911 Passes locally: |
|
@elasticmachine merge upstream |
This comment has been minimized.
This comment has been minimized.
spong
reviewed
Aug 9, 2021
...ty_solution/server/lib/detection_engine/signals/saved_object_references/inject_references.ts
Show resolved
Hide resolved
💚 Build SucceededMetrics [docs]
History
To update your PR or re-run it, just comment with: |
spong
approved these changes
Aug 11, 2021
There was a problem hiding this comment.
Paired with @FrankHassanabad on implementation final testing and everything LGTM! 👍
In testing it was discovered that we don't currently maintain a link (SO references[]) between exception list containers and items, so I went ahead in opened #108127 to cover that issue.
We had also tested what importing/exporting via the SO Mgmt UI would look like by toggling this configuration on the security rules:
And found we were able to successfully export both rules and their exceptions! 🙌 This gets us one step closer to consolidating Import/Export logic, however we'll still need to expose the SO type exception-list to the SO Mgmt UI (create this issue for that #108128).
Thank you for all your efforts here @FrankHassanabad, and for the significant test coverage and examples added for future developers to use these new references API's -- they're greatly appreciated!
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Aug 11, 2021
…ved object references when created or modified (part 1) (elastic#107064) ## Summary This is part 1 to addressing the issue seen here: elastic#101975 This part 1 wires up our rules to be able to `inject` and `extract` parameters from the saved object references. Follow up part 2 (not included here) will do the saved object migrations of existing rules to have the saved object references. The way the code is written it shouldn't interfere or blow up anything even though the existing rules have not been migrated since we do fallbacks and only log errors when we detect that the saved object references have not been migrated or have been deleted. Therefore this PR should be migration friendly in that you will only see an occasional error as it serializes and deserializes a non migrated rule without object references but still work both ways. Non-migrated rules or rules with deleted saved object references will self correct during the serialization phase when you edit a rule and save out the modification. This should be migration bug friendly as well in case something does not work out with migrations, we can still have users edit an existing rule to correct the bug. For manual testing, see the `README.md` in the folder. You should be able to create and modify existing rules and then see in their saved objects that they have `references` pointing to the top level exception list containers with this PR. * Adds the new folder in `detection_engine/signals/saved_object_references` with all the code needed * Adds a top level `README.md` about the functionality and tips for new programmers to add their own references * Adds a generic pattern for adding more saved object references within our rule set * Adds ~40 unit tests * Adds additional migration safe logic to de-couple this from required saved object migrations and hopefully helps mitigates any existing bugs within the stack or previous migration bugs a bit for us. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
💚 Backport successful
This backport PR will be merged automatically after passing CI. |
kibanamachine
added a commit
that referenced
this pull request
Aug 11, 2021
…ved object references when created or modified (part 1) (#107064) (#108225) ## Summary This is part 1 to addressing the issue seen here: #101975 This part 1 wires up our rules to be able to `inject` and `extract` parameters from the saved object references. Follow up part 2 (not included here) will do the saved object migrations of existing rules to have the saved object references. The way the code is written it shouldn't interfere or blow up anything even though the existing rules have not been migrated since we do fallbacks and only log errors when we detect that the saved object references have not been migrated or have been deleted. Therefore this PR should be migration friendly in that you will only see an occasional error as it serializes and deserializes a non migrated rule without object references but still work both ways. Non-migrated rules or rules with deleted saved object references will self correct during the serialization phase when you edit a rule and save out the modification. This should be migration bug friendly as well in case something does not work out with migrations, we can still have users edit an existing rule to correct the bug. For manual testing, see the `README.md` in the folder. You should be able to create and modify existing rules and then see in their saved objects that they have `references` pointing to the top level exception list containers with this PR. * Adds the new folder in `detection_engine/signals/saved_object_references` with all the code needed * Adds a top level `README.md` about the functionality and tips for new programmers to add their own references * Adds a generic pattern for adding more saved object references within our rule set * Adds ~40 unit tests * Adds additional migration safe logic to de-couple this from required saved object migrations and hopefully helps mitigates any existing bugs within the stack or previous migration bugs a bit for us. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Frank Hassanabad <[email protected]>
1 task
FrankHassanabad
added a commit
that referenced
this pull request
Aug 17, 2021
…ved object references (Part 2) (#108291) ## Summary This is part 2 to addressing the issue seen here: #101975 (Part 1 #107064) This adds the alerting migration scripts and unit tests for exception list containers on Kibana startup for `7.15.0` We only migrate if we find these conditions and cases: - `exceptionLists` are an `array` and not `null`, `undefined`, or malformed data. - The exceptionList item is an `object` and its `id` is a `string` and not `null`, `undefined`, or malformed data - The existing references do not already have an exceptionItem reference already found within it. We migrate on the common use case - The saved object references do not exist but we have exceptionList items with the id's to create the saved object references, 👍 so we migrate. - The alert contains no exception list items, in which case we have nothing to migrate We do these additional (mis-use) cases and steps as well. These should _NOT_ be common things that happen but we safe guard for them here: - If the migration is run twice we are idempotent and do _NOT_ add duplicates list items or remove items. - If the migration was partially successful but re-run a second time, we only add what is missing. Again no duplicates or removed items should occur. - If the `exceptionLists` contains invalid data shape or not enough information to migrate, we filter it out and ignore it - If the saved object references already exists and contains a different or foreign value, we will retain the foreign reference(s) and still migrate. ## Manual testing There are unit tests but for any manual testing or verification you can do the following: Create a few alerts through the `security_solution` application with exception lists <img width="1775" alt="Screen Shot 2021-08-11 at 5 42 31 PM" src="https://user-images.githubusercontent.com/1151048/129117377-61b17fcf-ad01-4405-bbfe-42d97a6f7654.png"> Use the dev tools to de-migrate as well as to test end to end like so: ```json # First get an "_id" with an exceptions list like below. Mine I found was: "alert:38482620-ef1b-11eb-ad71-7de7959be71c": GET .kibana/_search { "query": { "terms": { "alert.alertTypeId": [ "siem.signals" ] } }, "size": 10000 } ``` With Kibana running downgrade and remove the references as a test: ```json # Set saved object array references as empty arrays and set our migration version to be 7.14.0 POST .kibana/_update/alert:38482620-ef1b-11eb-ad71-7de7959be71c { "script" : { "source": """ ctx._source.migrationVersion.alert = "7.14.0"; ctx._source.references = [] """, "lang": "painless" } } # Double check the references is empty and the version is 7.14.0 GET .kibana/_doc/alert:38482620-ef1b-11eb-ad71-7de7959be71c ``` Reload the alert in the `security_solution` and notice you get these errors until you restart Kibana to cause a migration moving forward ```sh server log [17:35:16.914] [error][plugins][securitySolution] The saved object references were not found for our exception list when we were expecting to find it. Kibana migrations might not have run correctly or someone might have removed the saved object references manually. Returning the last known good exception list id which might not work. exceptionItem with its id being returned is: {"list_id":"endpoint_list","namespace_type":"agnostic","id":"endpoint_list","type":"endpoint"} server log [17:35:16.914] [error][plugins][securitySolution] Cannot get a saved object reference using an index which is larger than the saved object references. Index is:1 which is larger than the savedObjectReferences:[] server log [17:35:16.915] [error][plugins][securitySolution] The saved object references were not found for our exception list when we were expecting to find it. Kibana migrations might not have run correctly or someone might have removed the saved object references manually. Returning the last known good exception list id which might not work. exceptionItem with its id being returned is: {"list_id":"cd152d0d-3590-4a45-a478-eed04da7936b","namespace_type":"single","id":"50e3bd70-ef1b-11eb-ad71-7de7959be71c","type":"detection"} server log [17:35:16.940] [error][plugins][securitySolution] The saved object references were not found for our exception list when we were expecting to find it. Kibana migrations might not have run correctly or someone might have removed the saved object references manually. Returning the last known good exception list id which might not work. exceptionItem with its id being returned is: {"list_id":"endpoint_list","namespace_type":"agnostic","id":"endpoint_list","type":"endpoint"} server log [17:35:16.940] [error][plugins][securitySolution] Cannot get a saved object reference using an index which is larger than the saved object references. Index is:1 which is larger than the savedObjectReferences:[] server log [17:35:16.940] [error][plugins][securitySolution] The saved object references were not found for our exception list when we were expecting to find it. Kibana migrations might not have run correctly or someone might have removed the saved object references manually. Returning the last known good exception list id which might not work. exceptionItem with its id being returned is: {"list_id":"cd152d0d-3590-4a45-a478-eed04da7936b","namespace_type":"single","id":"50e3bd70-ef1b-11eb-ad71-7de7959be71c","type":"detection"} ``` Restart Kibana and you should no longer have errors in the Kibana console. If you do this query in dev tools ```json # Check that the `migrationVersion` is `7.15.0` and that we have a `references` array filled out with the correct structure GET .kibana/_doc/alert:38482620-ef1b-11eb-ad71-7de7959be71c ``` You should notice that you now have a `references` array filled out: ```json "references" : [ { "name" : "param:exceptionsList_0", "id" : "endpoint_list", "type" : "exception-list-agnostic" }, { "name" : "param:exceptionsList_1", "id" : "50e3bd70-ef1b-11eb-ad71-7de7959be71c", "type" : "exception-list" } ], "migrationVersion" : { "alert" : "7.15.0" } ``` For testing [idempotentence](https://en.wikipedia.org/wiki/Idempotence) Run just this to downgrade and restart Kibana and you should notice on a GET that we do not have anything extra in the references array: ```json # Set our migration version to be 7.14.0 only POST .kibana/_update/alert:38482620-ef1b-11eb-ad71-7de7959be71c { "script" : { "source": """ ctx._source.migrationVersion.alert = "7.14.0"; """, "lang": "painless" } } # Double check the `references` is still there, and we do not get errors or changes to `references` after we restart Kibana GET .kibana/_doc/alert:38482620-ef1b-11eb-ad71-7de7959be71c ``` For testing foreign keys: ```json # Set saved object array references to foreign keys and set our migration version to be 7.14.0 POST .kibana/_update/alert:38482620-ef1b-11eb-ad71-7de7959be71c { "script" : { "source": """ ctx._source.migrationVersion.alert = "7.14.0"; ctx._source.references = [["name" : "foreign", "id" : "123", "type" : "some-type"]]; """, "lang": "painless" } } ``` Restart, ensure no errors in Kibana console and do a get call to ensure we have the foreign mixed with valid values: ```json GET .kibana/_doc/alert:38482620-ef1b-11eb-ad71-7de7959be71c ``` Should return this data: ```json "type" : "alert", "references" : [ { "name" : "foreign", "id" : "123", "type" : "some-type" }, { "name" : "param:exceptionsList_0", "id" : "endpoint_list", "type" : "exception-list-agnostic" }, { "name" : "param:exceptionsList_1", "id" : "50e3bd70-ef1b-11eb-ad71-7de7959be71c", "type" : "exception-list" } ] ``` ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Aug 17, 2021
…ved object references (Part 2) (elastic#108291) ## Summary This is part 2 to addressing the issue seen here: elastic#101975 (Part 1 elastic#107064) This adds the alerting migration scripts and unit tests for exception list containers on Kibana startup for `7.15.0` We only migrate if we find these conditions and cases: - `exceptionLists` are an `array` and not `null`, `undefined`, or malformed data. - The exceptionList item is an `object` and its `id` is a `string` and not `null`, `undefined`, or malformed data - The existing references do not already have an exceptionItem reference already found within it. We migrate on the common use case - The saved object references do not exist but we have exceptionList items with the id's to create the saved object references, 👍 so we migrate. - The alert contains no exception list items, in which case we have nothing to migrate We do these additional (mis-use) cases and steps as well. These should _NOT_ be common things that happen but we safe guard for them here: - If the migration is run twice we are idempotent and do _NOT_ add duplicates list items or remove items. - If the migration was partially successful but re-run a second time, we only add what is missing. Again no duplicates or removed items should occur. - If the `exceptionLists` contains invalid data shape or not enough information to migrate, we filter it out and ignore it - If the saved object references already exists and contains a different or foreign value, we will retain the foreign reference(s) and still migrate. ## Manual testing There are unit tests but for any manual testing or verification you can do the following: Create a few alerts through the `security_solution` application with exception lists <img width="1775" alt="Screen Shot 2021-08-11 at 5 42 31 PM" src="https://user-images.githubusercontent.com/1151048/129117377-61b17fcf-ad01-4405-bbfe-42d97a6f7654.png"> Use the dev tools to de-migrate as well as to test end to end like so: ```json # First get an "_id" with an exceptions list like below. Mine I found was: "alert:38482620-ef1b-11eb-ad71-7de7959be71c": GET .kibana/_search { "query": { "terms": { "alert.alertTypeId": [ "siem.signals" ] } }, "size": 10000 } ``` With Kibana running downgrade and remove the references as a test: ```json # Set saved object array references as empty arrays and set our migration version to be 7.14.0 POST .kibana/_update/alert:38482620-ef1b-11eb-ad71-7de7959be71c { "script" : { "source": """ ctx._source.migrationVersion.alert = "7.14.0"; ctx._source.references = [] """, "lang": "painless" } } # Double check the references is empty and the version is 7.14.0 GET .kibana/_doc/alert:38482620-ef1b-11eb-ad71-7de7959be71c ``` Reload the alert in the `security_solution` and notice you get these errors until you restart Kibana to cause a migration moving forward ```sh server log [17:35:16.914] [error][plugins][securitySolution] The saved object references were not found for our exception list when we were expecting to find it. Kibana migrations might not have run correctly or someone might have removed the saved object references manually. Returning the last known good exception list id which might not work. exceptionItem with its id being returned is: {"list_id":"endpoint_list","namespace_type":"agnostic","id":"endpoint_list","type":"endpoint"} server log [17:35:16.914] [error][plugins][securitySolution] Cannot get a saved object reference using an index which is larger than the saved object references. Index is:1 which is larger than the savedObjectReferences:[] server log [17:35:16.915] [error][plugins][securitySolution] The saved object references were not found for our exception list when we were expecting to find it. Kibana migrations might not have run correctly or someone might have removed the saved object references manually. Returning the last known good exception list id which might not work. exceptionItem with its id being returned is: {"list_id":"cd152d0d-3590-4a45-a478-eed04da7936b","namespace_type":"single","id":"50e3bd70-ef1b-11eb-ad71-7de7959be71c","type":"detection"} server log [17:35:16.940] [error][plugins][securitySolution] The saved object references were not found for our exception list when we were expecting to find it. Kibana migrations might not have run correctly or someone might have removed the saved object references manually. Returning the last known good exception list id which might not work. exceptionItem with its id being returned is: {"list_id":"endpoint_list","namespace_type":"agnostic","id":"endpoint_list","type":"endpoint"} server log [17:35:16.940] [error][plugins][securitySolution] Cannot get a saved object reference using an index which is larger than the saved object references. Index is:1 which is larger than the savedObjectReferences:[] server log [17:35:16.940] [error][plugins][securitySolution] The saved object references were not found for our exception list when we were expecting to find it. Kibana migrations might not have run correctly or someone might have removed the saved object references manually. Returning the last known good exception list id which might not work. exceptionItem with its id being returned is: {"list_id":"cd152d0d-3590-4a45-a478-eed04da7936b","namespace_type":"single","id":"50e3bd70-ef1b-11eb-ad71-7de7959be71c","type":"detection"} ``` Restart Kibana and you should no longer have errors in the Kibana console. If you do this query in dev tools ```json # Check that the `migrationVersion` is `7.15.0` and that we have a `references` array filled out with the correct structure GET .kibana/_doc/alert:38482620-ef1b-11eb-ad71-7de7959be71c ``` You should notice that you now have a `references` array filled out: ```json "references" : [ { "name" : "param:exceptionsList_0", "id" : "endpoint_list", "type" : "exception-list-agnostic" }, { "name" : "param:exceptionsList_1", "id" : "50e3bd70-ef1b-11eb-ad71-7de7959be71c", "type" : "exception-list" } ], "migrationVersion" : { "alert" : "7.15.0" } ``` For testing [idempotentence](https://en.wikipedia.org/wiki/Idempotence) Run just this to downgrade and restart Kibana and you should notice on a GET that we do not have anything extra in the references array: ```json # Set our migration version to be 7.14.0 only POST .kibana/_update/alert:38482620-ef1b-11eb-ad71-7de7959be71c { "script" : { "source": """ ctx._source.migrationVersion.alert = "7.14.0"; """, "lang": "painless" } } # Double check the `references` is still there, and we do not get errors or changes to `references` after we restart Kibana GET .kibana/_doc/alert:38482620-ef1b-11eb-ad71-7de7959be71c ``` For testing foreign keys: ```json # Set saved object array references to foreign keys and set our migration version to be 7.14.0 POST .kibana/_update/alert:38482620-ef1b-11eb-ad71-7de7959be71c { "script" : { "source": """ ctx._source.migrationVersion.alert = "7.14.0"; ctx._source.references = [["name" : "foreign", "id" : "123", "type" : "some-type"]]; """, "lang": "painless" } } ``` Restart, ensure no errors in Kibana console and do a get call to ensure we have the foreign mixed with valid values: ```json GET .kibana/_doc/alert:38482620-ef1b-11eb-ad71-7de7959be71c ``` Should return this data: ```json "type" : "alert", "references" : [ { "name" : "foreign", "id" : "123", "type" : "some-type" }, { "name" : "param:exceptionsList_0", "id" : "endpoint_list", "type" : "exception-list-agnostic" }, { "name" : "param:exceptionsList_1", "id" : "50e3bd70-ef1b-11eb-ad71-7de7959be71c", "type" : "exception-list" } ] ``` ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
kibanamachine
added a commit
that referenced
this pull request
Aug 17, 2021
…ved object references (Part 2) (#108291) (#108991) ## Summary This is part 2 to addressing the issue seen here: #101975 (Part 1 #107064) This adds the alerting migration scripts and unit tests for exception list containers on Kibana startup for `7.15.0` We only migrate if we find these conditions and cases: - `exceptionLists` are an `array` and not `null`, `undefined`, or malformed data. - The exceptionList item is an `object` and its `id` is a `string` and not `null`, `undefined`, or malformed data - The existing references do not already have an exceptionItem reference already found within it. We migrate on the common use case - The saved object references do not exist but we have exceptionList items with the id's to create the saved object references, 👍 so we migrate. - The alert contains no exception list items, in which case we have nothing to migrate We do these additional (mis-use) cases and steps as well. These should _NOT_ be common things that happen but we safe guard for them here: - If the migration is run twice we are idempotent and do _NOT_ add duplicates list items or remove items. - If the migration was partially successful but re-run a second time, we only add what is missing. Again no duplicates or removed items should occur. - If the `exceptionLists` contains invalid data shape or not enough information to migrate, we filter it out and ignore it - If the saved object references already exists and contains a different or foreign value, we will retain the foreign reference(s) and still migrate. ## Manual testing There are unit tests but for any manual testing or verification you can do the following: Create a few alerts through the `security_solution` application with exception lists <img width="1775" alt="Screen Shot 2021-08-11 at 5 42 31 PM" src="https://user-images.githubusercontent.com/1151048/129117377-61b17fcf-ad01-4405-bbfe-42d97a6f7654.png"> Use the dev tools to de-migrate as well as to test end to end like so: ```json # First get an "_id" with an exceptions list like below. Mine I found was: "alert:38482620-ef1b-11eb-ad71-7de7959be71c": GET .kibana/_search { "query": { "terms": { "alert.alertTypeId": [ "siem.signals" ] } }, "size": 10000 } ``` With Kibana running downgrade and remove the references as a test: ```json # Set saved object array references as empty arrays and set our migration version to be 7.14.0 POST .kibana/_update/alert:38482620-ef1b-11eb-ad71-7de7959be71c { "script" : { "source": """ ctx._source.migrationVersion.alert = "7.14.0"; ctx._source.references = [] """, "lang": "painless" } } # Double check the references is empty and the version is 7.14.0 GET .kibana/_doc/alert:38482620-ef1b-11eb-ad71-7de7959be71c ``` Reload the alert in the `security_solution` and notice you get these errors until you restart Kibana to cause a migration moving forward ```sh server log [17:35:16.914] [error][plugins][securitySolution] The saved object references were not found for our exception list when we were expecting to find it. Kibana migrations might not have run correctly or someone might have removed the saved object references manually. Returning the last known good exception list id which might not work. exceptionItem with its id being returned is: {"list_id":"endpoint_list","namespace_type":"agnostic","id":"endpoint_list","type":"endpoint"} server log [17:35:16.914] [error][plugins][securitySolution] Cannot get a saved object reference using an index which is larger than the saved object references. Index is:1 which is larger than the savedObjectReferences:[] server log [17:35:16.915] [error][plugins][securitySolution] The saved object references were not found for our exception list when we were expecting to find it. Kibana migrations might not have run correctly or someone might have removed the saved object references manually. Returning the last known good exception list id which might not work. exceptionItem with its id being returned is: {"list_id":"cd152d0d-3590-4a45-a478-eed04da7936b","namespace_type":"single","id":"50e3bd70-ef1b-11eb-ad71-7de7959be71c","type":"detection"} server log [17:35:16.940] [error][plugins][securitySolution] The saved object references were not found for our exception list when we were expecting to find it. Kibana migrations might not have run correctly or someone might have removed the saved object references manually. Returning the last known good exception list id which might not work. exceptionItem with its id being returned is: {"list_id":"endpoint_list","namespace_type":"agnostic","id":"endpoint_list","type":"endpoint"} server log [17:35:16.940] [error][plugins][securitySolution] Cannot get a saved object reference using an index which is larger than the saved object references. Index is:1 which is larger than the savedObjectReferences:[] server log [17:35:16.940] [error][plugins][securitySolution] The saved object references were not found for our exception list when we were expecting to find it. Kibana migrations might not have run correctly or someone might have removed the saved object references manually. Returning the last known good exception list id which might not work. exceptionItem with its id being returned is: {"list_id":"cd152d0d-3590-4a45-a478-eed04da7936b","namespace_type":"single","id":"50e3bd70-ef1b-11eb-ad71-7de7959be71c","type":"detection"} ``` Restart Kibana and you should no longer have errors in the Kibana console. If you do this query in dev tools ```json # Check that the `migrationVersion` is `7.15.0` and that we have a `references` array filled out with the correct structure GET .kibana/_doc/alert:38482620-ef1b-11eb-ad71-7de7959be71c ``` You should notice that you now have a `references` array filled out: ```json "references" : [ { "name" : "param:exceptionsList_0", "id" : "endpoint_list", "type" : "exception-list-agnostic" }, { "name" : "param:exceptionsList_1", "id" : "50e3bd70-ef1b-11eb-ad71-7de7959be71c", "type" : "exception-list" } ], "migrationVersion" : { "alert" : "7.15.0" } ``` For testing [idempotentence](https://en.wikipedia.org/wiki/Idempotence) Run just this to downgrade and restart Kibana and you should notice on a GET that we do not have anything extra in the references array: ```json # Set our migration version to be 7.14.0 only POST .kibana/_update/alert:38482620-ef1b-11eb-ad71-7de7959be71c { "script" : { "source": """ ctx._source.migrationVersion.alert = "7.14.0"; """, "lang": "painless" } } # Double check the `references` is still there, and we do not get errors or changes to `references` after we restart Kibana GET .kibana/_doc/alert:38482620-ef1b-11eb-ad71-7de7959be71c ``` For testing foreign keys: ```json # Set saved object array references to foreign keys and set our migration version to be 7.14.0 POST .kibana/_update/alert:38482620-ef1b-11eb-ad71-7de7959be71c { "script" : { "source": """ ctx._source.migrationVersion.alert = "7.14.0"; ctx._source.references = [["name" : "foreign", "id" : "123", "type" : "some-type"]]; """, "lang": "painless" } } ``` Restart, ensure no errors in Kibana console and do a get call to ensure we have the foreign mixed with valid values: ```json GET .kibana/_doc/alert:38482620-ef1b-11eb-ad71-7de7959be71c ``` Should return this data: ```json "type" : "alert", "references" : [ { "name" : "foreign", "id" : "123", "type" : "some-type" }, { "name" : "param:exceptionsList_0", "id" : "endpoint_list", "type" : "exception-list-agnostic" }, { "name" : "param:exceptionsList_1", "id" : "50e3bd70-ef1b-11eb-ad71-7de7959be71c", "type" : "exception-list" } ] ``` ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Frank Hassanabad <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This is part 1 to addressing the issue seen here: #101975
This part 1 wires up our rules to be able to
injectandextractparameters from the saved object references. Follow up part 2 (not included here) will do the saved object migrations of existing rules to have the saved object references.The way the code is written it shouldn't interfere or blow up anything even though the existing rules have not been migrated since we do fallbacks and only log errors when we detect that the saved object references have not been migrated or have been deleted.
Therefore this PR should be migration friendly in that you will only see an occasional error as it serializes and deserializes a non migrated rule without object references but still work both ways. Non-migrated rules or rules with deleted saved object references will self correct during the serialization phase when you edit a rule and save out the modification. This should be migration bug friendly as well in case something does not work out with migrations, we can still have users edit an existing rule to correct the bug.
For manual testing, see the
README.mdin the folder. You should be able to create and modify existing rules and then see in their saved objects that they havereferencespointing to the top level exception list containers with this PR.detection_engine/signals/saved_object_referenceswith all the code neededREADME.mdabout the functionality and tips for new programmers to add their own referencesChecklist