Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[7.x] [Security Solutions][Detection Engine] Adds exception lists to the saved object references when created or modified (part 1) (#107064) #108225

Merged
merged 1 commit into from
Aug 11, 2021
Merged

[7.x] [Security Solutions][Detection Engine] Adds exception lists to the saved object references when created or modified (part 1) (#107064) #108225

merged 1 commit into from
Aug 11, 2021

Conversation

kibanamachine
Copy link
Contributor

Backports the following commits to 7.x:

@FrankHassanabad @kibanamachine
[Security Solutions][Detection Engine] Adds exception lists to the sa…
97c2d72 
…ved object references when created or modified (part 1) (elastic#107064)

## Summary

This is part 1 to addressing the issue seen here: elastic#101975

This part 1 wires up our rules to be able to `inject` and `extract` parameters from the saved object references. Follow up part 2 (not included here) will do the saved object migrations of existing rules to have the saved object references.

The way the code is written it shouldn't interfere or blow up anything even though the existing rules have not been migrated since we do fallbacks and only log errors when we detect that the saved object references have not been migrated or have been deleted.

Therefore this PR should be migration friendly in that you will only see an occasional error as it serializes and deserializes a non migrated rule without object references but still work both ways. Non-migrated rules or rules with deleted saved object references will self correct during the serialization phase when you edit a rule and save out the modification. This should be migration bug friendly as well in case something does not work out with migrations, we can still have users edit an existing rule to correct the bug.

For manual testing, see the `README.md` in the folder. You should be able to create and modify existing rules and then see in their saved objects that they have `references` pointing to the top level exception list containers with this PR.

* Adds the new folder in `detection_engine/signals/saved_object_references` with all the code needed
* Adds a top level `README.md` about the functionality and tips for new programmers to add their own references 
* Adds a generic pattern for adding more saved object references within our rule set
* Adds ~40 unit tests
* Adds additional migration safe logic to de-couple this from required saved object migrations and hopefully helps mitigates any existing bugs within the stack or previous migration bugs a bit for us.

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
@kibanamachine kibanamachine enabled auto-merge (squash) August 11, 2021 15:48
@kibanamachine kibanamachine mentioned this pull request Aug 11, 2021
Merged
1 task
@kibanamachine
Copy link
Contributor Author

💚 Build Succeeded

Metrics [docs]

✅ unchanged

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @FrankHassanabad

@kibanamachine kibanamachine merged commit d379ebc into elastic:7.x Aug 11, 2021
@FrankHassanabad FrankHassanabad mentioned this pull request Aug 13, 2021
Merged
1 task
FrankHassanabad added a commit that referenced this pull request Aug 16, 2021
@FrankHassanabad
[Security Solution][Detection Engine] Fixes agnostic type bug (#108610)
9fa62bf 
## Summary

Fixes agnostic type bug where in part 1 (#108225), I incorrectly used the same saved object type for both `single` and `agnostic`.

Before the references for SO's were:

```json
"references" : [
  {
    "name" : "param:exceptionsList_0",
    "id" : "endpoint_list",
    "type" : "exception-list" <--- This should have been "exception-list-agnostic" type
  },
  {
    "name" : "param:exceptionsList_1",
    "id" : "50e3bd70-ef1b-11eb-ad71-7de7959be71c",
    "type" : "exception-list"
  }
],
```

After:
```json
"references" : [
  {
    "name" : "param:exceptionsList_0",
    "id" : "endpoint_list",
    "type" : "exception-list-agnostic" <--- This should now be the "exception-list-agnostic" type
  },
  {
    "name" : "param:exceptionsList_1",
    "id" : "50e3bd70-ef1b-11eb-ad71-7de7959be71c",
    "type" : "exception-list"
  }
],
```

Manual testing: Add a new `security_solution` alert and exception list as well as an endpoint list to it. Then save it
<img width="1581" alt="Screen Shot 2021-08-13 at 5 00 39 PM" src="https://user-images.githubusercontent.com/1151048/129425847-78025aba-6d7a-4a5a-9d4f-950ec664596c.png">
<img width="1571" alt="Screen Shot 2021-08-13 at 5 00 47 PM" src="https://user-images.githubusercontent.com/1151048/129425848-42018331-cac6-4411-8153-3441a8af6f34.png">


Do this query in dev tools:
```json
GET .kibana-hassanabad19/_search
{
  "query": {
    "terms": {
      "alert.alertTypeId": [
        "siem.signals"
      ]
    }
  },
  "size": 10000
}
```

And check to ensure that the references look like the after picture where type has : `"type" : "exception-list-agnostic"` if we have an agnostic list. Ensure that on a page reload that the exception types are still there on the rule. Ensure that there are no errors in the console about not finding the correct SO type or anything else odd. 

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Aug 16, 2021
@FrankHassanabad @kibanamachine
[Security Solution][Detection Engine] Fixes agnostic type bug (elasti…
0b44c13 
…c#108610)

## Summary

Fixes agnostic type bug where in part 1 (elastic#108225), I incorrectly used the same saved object type for both `single` and `agnostic`.

Before the references for SO's were:

```json
"references" : [
  {
    "name" : "param:exceptionsList_0",
    "id" : "endpoint_list",
    "type" : "exception-list" <--- This should have been "exception-list-agnostic" type
  },
  {
    "name" : "param:exceptionsList_1",
    "id" : "50e3bd70-ef1b-11eb-ad71-7de7959be71c",
    "type" : "exception-list"
  }
],
```

After:
```json
"references" : [
  {
    "name" : "param:exceptionsList_0",
    "id" : "endpoint_list",
    "type" : "exception-list-agnostic" <--- This should now be the "exception-list-agnostic" type
  },
  {
    "name" : "param:exceptionsList_1",
    "id" : "50e3bd70-ef1b-11eb-ad71-7de7959be71c",
    "type" : "exception-list"
  }
],
```

Manual testing: Add a new `security_solution` alert and exception list as well as an endpoint list to it. Then save it
<img width="1581" alt="Screen Shot 2021-08-13 at 5 00 39 PM" src="https://user-images.githubusercontent.com/1151048/129425847-78025aba-6d7a-4a5a-9d4f-950ec664596c.png">
<img width="1571" alt="Screen Shot 2021-08-13 at 5 00 47 PM" src="https://user-images.githubusercontent.com/1151048/129425848-42018331-cac6-4411-8153-3441a8af6f34.png">


Do this query in dev tools:
```json
GET .kibana-hassanabad19/_search
{
  "query": {
    "terms": {
      "alert.alertTypeId": [
        "siem.signals"
      ]
    }
  },
  "size": 10000
}
```

And check to ensure that the references look like the after picture where type has : `"type" : "exception-list-agnostic"` if we have an agnostic list. Ensure that on a page reload that the exception types are still there on the rule. Ensure that there are no errors in the console about not finding the correct SO type or anything else odd. 

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
kibanamachine added a commit that referenced this pull request Aug 16, 2021
@kibanamachine @FrankHassanabad
[Security Solution][Detection Engine] Fixes agnostic type bug (#108610)…
bf61bcc 
… (#108745)

## Summary

Fixes agnostic type bug where in part 1 (#108225), I incorrectly used the same saved object type for both `single` and `agnostic`.

Before the references for SO's were:

```json
"references" : [
  {
    "name" : "param:exceptionsList_0",
    "id" : "endpoint_list",
    "type" : "exception-list" <--- This should have been "exception-list-agnostic" type
  },
  {
    "name" : "param:exceptionsList_1",
    "id" : "50e3bd70-ef1b-11eb-ad71-7de7959be71c",
    "type" : "exception-list"
  }
],
```

After:
```json
"references" : [
  {
    "name" : "param:exceptionsList_0",
    "id" : "endpoint_list",
    "type" : "exception-list-agnostic" <--- This should now be the "exception-list-agnostic" type
  },
  {
    "name" : "param:exceptionsList_1",
    "id" : "50e3bd70-ef1b-11eb-ad71-7de7959be71c",
    "type" : "exception-list"
  }
],
```

Manual testing: Add a new `security_solution` alert and exception list as well as an endpoint list to it. Then save it
<img width="1581" alt="Screen Shot 2021-08-13 at 5 00 39 PM" src="https://user-images.githubusercontent.com/1151048/129425847-78025aba-6d7a-4a5a-9d4f-950ec664596c.png">
<img width="1571" alt="Screen Shot 2021-08-13 at 5 00 47 PM" src="https://user-images.githubusercontent.com/1151048/129425848-42018331-cac6-4411-8153-3441a8af6f34.png">


Do this query in dev tools:
```json
GET .kibana-hassanabad19/_search
{
  "query": {
    "terms": {
      "alert.alertTypeId": [
        "siem.signals"
      ]
    }
  },
  "size": 10000
}
```

And check to ensure that the references look like the after picture where type has : `"type" : "exception-list-agnostic"` if we have an agnostic list. Ensure that on a page reload that the exception types are still there on the rule. Ensure that there are no errors in the console about not finding the correct SO type or anything else odd. 

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

Co-authored-by: Frank Hassanabad <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@FrankHassanabad FrankHassanabad

Labels
backport
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kibanamachine @FrankHassanabad