[Security Solution][Platform] - Exceptions export duplicates #116698

yctercero · 2021-10-28T18:11:54Z

Summary

Removes duplicate exception lists on rule export when multiple rules reference the same list.

Testing

Create an exception list:
cd x-pack/plugins/lists/server/scripts
./post_exception_list.sh

Add exception list items to it:
./post_x_exception_list_items.sh

Using the id of the exception list created above, update x-pack/plugins/security_solution/server/lib/detection_engine/scripts/rules/queries/references/query_with_single_exception_list.json so that it references your exception list.

{
  "name": "Rule 1",
  "description": "Sample rule with single exception list",
  "rule_id": "query-with-single-exception-list",
  "risk_score": 1,
  "severity": "high",
  "type": "query",
  "query": "host.name: *",
  "interval": "30s",
  "exceptions_list": [{ "id": "YOUR_LIST_ID_HERE", "list_id": "YOUR_LIST_ID_ID_HERE", "namespace_type": "single", "type": "detection" }]
}

Create rule with exception list:
cd x-pack/plugins/security_solution/server/lib/detection_engine/scripts
./post_rule.sh ./rules/queries/references/query_with_single_exception_list.json

Update x-pack/plugins/security_solution/server/lib/detection_engine/scripts/rules/queries/references/query_with_single_exception_list.json so that it has a different rule_id and name. and create a second rule referencing the same list.
./post_rule.sh ./rules/queries/references/query_with_single_exception_list.json

Export rule:
cd x-pack/plugins/security_solution/server/lib/detection_engine/scripts
./export_rules.sh

Sample Export

With duplicates

{"id":"db223c60-3817-11ec-bf3b-2b435a1e041e","updated_at":"2021-10-28T17:52:46.250Z","updated_by":"elastic","created_at":"2021-10-28T17:52:46.250Z","created_by":"elastic","name":"Rule 1","tags":[],"interval":"30s","enabled":true,"description":"Sample rule with single exception list","risk_score":1,"severity":"high","output_index":".siem-signals-default","author":[],"false_positives":[],"from":"now-6m","rule_id":"query-with-single-exception-list","max_signals":100,"risk_score_mapping":[],"severity_mapping":[],"threat":[],"to":"now","references":[],"version":1,"exceptions_list":[{"id":"40677cd0-3817-11ec-bf3b-2b435a1e041e","list_id":"simple_list","namespace_type":"single","type":"detection"}],"immutable":false,"type":"query","language":"kuery","query":"host.name: *","throttle":"no_actions","actions":[]}
{"id":"2aeda270-3818-11ec-bf3b-2b435a1e041e","updated_at":"2021-10-28T17:54:59.899Z","updated_by":"elastic","created_at":"2021-10-28T17:54:59.899Z","created_by":"elastic","name":"Rule 2","tags":[],"interval":"30s","enabled":true,"description":"Sample rule with single exception list","risk_score":1,"severity":"high","output_index":".siem-signals-default","author":[],"false_positives":[],"from":"now-6m","rule_id":"query-with-single-exception-list-2","max_signals":100,"risk_score_mapping":[],"severity_mapping":[],"threat":[],"to":"now","references":[],"version":1,"exceptions_list":[{"id":"40677cd0-3817-11ec-bf3b-2b435a1e041e","list_id":"simple_list","namespace_type":"single","type":"detection"}],"immutable":false,"type":"query","language":"kuery","query":"host.name: *","throttle":"no_actions","actions":[]}
{"_version":"WzI1MTUsMV0=","created_at":"2021-10-28T17:48:25.757Z","created_by":"elastic","description":"This is a sample endpoint type exception","id":"40677cd0-3817-11ec-bf3b-2b435a1e041e","immutable":false,"list_id":"simple_list","name":"Sample Endpoint Exception List","namespace_type":"single","os_types":[],"tags":["user added string for a tag","malware"],"tie_breaker_id":"ed212e9d-bc1f-40f6-a280-b8aad2b98478","type":"detection","updated_at":"2021-10-28T17:48:25.764Z","updated_by":"elastic","version":1}
{"_version":"WzI1MDksMV0=","comments":[],"created_at":"2021-10-28T17:50:05.108Z","created_by":"elastic","description":"This is a sample endpoint type exception","entries":[{"field":"actingProcess.file.signer","operator":"excluded","type":"exists"},{"field":"host.name","operator":"included","type":"match_any","value":["some host","another host"]}],"id":"7b9f3f40-3817-11ec-bf3b-2b435a1e041e","item_id":"9R5p1zpooec3LF8GjB4rOYN3KfnIuISq7yt5IuAgilB0sxPcgIaEi09unIGNcV0mmNyzw6eAD4MYX2BGg7nNc4LFASZn2KUAXlW8","list_id":"simple_list","name":"Sample Endpoint Exception List","namespace_type":"single","os_types":["linux"],"tags":["user added string for a tag","malware"],"tie_breaker_id":"b8417928-ab00-40fa-809c-b9002c320b76","type":"simple","updated_at":"2021-10-28T17:50:05.110Z","updated_by":"elastic"}
{"_version":"WzI1MTUsMV0=","created_at":"2021-10-28T17:48:25.757Z","created_by":"elastic","description":"This is a sample endpoint type exception","id":"40677cd0-3817-11ec-bf3b-2b435a1e041e","immutable":false,"list_id":"simple_list","name":"Sample Endpoint Exception List","namespace_type":"single","os_types":[],"tags":["user added string for a tag","malware"],"tie_breaker_id":"ed212e9d-bc1f-40f6-a280-b8aad2b98478","type":"detection","updated_at":"2021-10-28T17:48:25.764Z","updated_by":"elastic","version":1}
{"_version":"WzI1MDksMV0=","comments":[],"created_at":"2021-10-28T17:50:05.108Z","created_by":"elastic","description":"This is a sample endpoint type exception","entries":[{"field":"actingProcess.file.signer","operator":"excluded","type":"exists"},{"field":"host.name","operator":"included","type":"match_any","value":["some host","another host"]}],"id":"7b9f3f40-3817-11ec-bf3b-2b435a1e041e","item_id":"9R5p1zpooec3LF8GjB4rOYN3KfnIuISq7yt5IuAgilB0sxPcgIaEi09unIGNcV0mmNyzw6eAD4MYX2BGg7nNc4LFASZn2KUAXlW8","list_id":"simple_list","name":"Sample Endpoint Exception List","namespace_type":"single","os_types":["linux"],"tags":["user added string for a tag","malware"],"tie_breaker_id":"b8417928-ab00-40fa-809c-b9002c320b76","type":"simple","updated_at":"2021-10-28T17:50:05.110Z","updated_by":"elastic"}
{"exported_rules_count":2,"missing_rules":[],"missing_rules_count":0,"exported_exception_list_count":2,"exported_exception_list_item_count":2,"missing_exception_list_item_count":0,"missing_exception_list_items":[],"missing_exception_lists":[],"missing_exception_lists_count":0}

Without duplicates

{"id":"db223c60-3817-11ec-bf3b-2b435a1e041e","updated_at":"2021-10-28T17:52:46.250Z","updated_by":"elastic","created_at":"2021-10-28T17:52:46.250Z","created_by":"elastic","name":"Rule 1","tags":[],"interval":"30s","enabled":true,"description":"Sample rule with single exception list","risk_score":1,"severity":"high","output_index":".siem-signals-default","author":[],"false_positives":[],"from":"now-6m","rule_id":"query-with-single-exception-list","max_signals":100,"risk_score_mapping":[],"severity_mapping":[],"threat":[],"to":"now","references":[],"version":1,"exceptions_list":[{"id":"40677cd0-3817-11ec-bf3b-2b435a1e041e","list_id":"simple_list","namespace_type":"single","type":"detection"}],"immutable":false,"type":"query","language":"kuery","query":"host.name: *","throttle":"no_actions","actions":[]}
{"id":"2aeda270-3818-11ec-bf3b-2b435a1e041e","updated_at":"2021-10-28T17:54:59.899Z","updated_by":"elastic","created_at":"2021-10-28T17:54:59.899Z","created_by":"elastic","name":"Rule 2","tags":[],"interval":"30s","enabled":true,"description":"Sample rule with single exception list","risk_score":1,"severity":"high","output_index":".siem-signals-default","author":[],"false_positives":[],"from":"now-6m","rule_id":"query-with-single-exception-list-2","max_signals":100,"risk_score_mapping":[],"severity_mapping":[],"threat":[],"to":"now","references":[],"version":1,"exceptions_list":[{"id":"40677cd0-3817-11ec-bf3b-2b435a1e041e","list_id":"simple_list","namespace_type":"single","type":"detection"}],"immutable":false,"type":"query","language":"kuery","query":"host.name: *","throttle":"no_actions","actions":[]}
{"_version":"WzE0LDFd","created_at":"2021-10-28T17:48:25.757Z","created_by":"elastic","description":"This is a sample endpoint type exception","id":"40677cd0-3817-11ec-bf3b-2b435a1e041e","immutable":false,"list_id":"simple_list","name":"Sample Endpoint Exception List","namespace_type":"single","os_types":[],"tags":["user added string for a tag","malware"],"tie_breaker_id":"ed212e9d-bc1f-40f6-a280-b8aad2b98478","type":"detection","updated_at":"2021-10-28T17:48:25.764Z","updated_by":"elastic","version":1}
{"_version":"WzExNCwxXQ==","comments":[],"created_at":"2021-10-28T17:50:05.108Z","created_by":"elastic","description":"This is a sample endpoint type exception","entries":[{"field":"actingProcess.file.signer","operator":"excluded","type":"exists"},{"field":"host.name","operator":"included","type":"match_any","value":["some host","another host"]}],"id":"7b9f3f40-3817-11ec-bf3b-2b435a1e041e","item_id":"9R5p1zpooec3LF8GjB4rOYN3KfnIuISq7yt5IuAgilB0sxPcgIaEi09unIGNcV0mmNyzw6eAD4MYX2BGg7nNc4LFASZn2KUAXlW8","list_id":"simple_list","name":"Sample Endpoint Exception List","namespace_type":"single","os_types":["linux"],"tags":["user added string for a tag","malware"],"tie_breaker_id":"b8417928-ab00-40fa-809c-b9002c320b76","type":"simple","updated_at":"2021-10-28T17:50:05.110Z","updated_by":"elastic"}
{"exported_rules_count":2,"missing_rules":[],"missing_rules_count":0,"exported_exception_list_count":1,"exported_exception_list_item_count":1,"missing_exception_list_item_count":0,"missing_exception_list_items":[],"missing_exception_lists":[],"missing_exception_lists_count":0}

Checklist

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
This was checked for cross-browser compatibility

…rt_duplicates

elasticmachine · 2021-10-28T18:30:20Z

Pinging @elastic/security-solution (Team: SecuritySolution)

yctercero · 2021-10-29T18:18:04Z

@elasticmachine merge upstream

x-pack/plugins/lists/server/scripts/check_env_variables.sh

x-pack/plugins/security_solution/server/lib/detection_engine/scripts/check_env_variables.sh

FrankHassanabad · 2021-10-29T21:29:43Z

...ck/plugins/security_solution/server/lib/detection_engine/rules/get_export_rule_exceptions.ts

+      const listHash = createHash('sha256').update(JSON.stringify(list)).digest('hex');
+      if (!uniqueExceptionLists.has(listHash)) {
+        uniqueExceptionLists.add(listHash);
+        return !NON_EXPORTABLE_LIST_IDS.includes(list.list_id);


Why are you using a hash here and not just a unique identifier such as the list_id? Shouldn't that be enough for this to work. If two are the same list but they are different in ordering then the hash will begin to fail on you.

I had gone with this because I worried about possibilities of the list_id being the same once they become shareable. Although, at that point I think something else would be wrong. I'll update to just use the id as those should be globally unique.

…rt_duplicates

…ibana into exceptions_export_duplicates

kibanamachine · 2021-11-01T22:51:40Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💔 Build #2911 failed 4354a6c
💔 Build #2672 failed e28c458

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @yctercero

FrankHassanabad

Looked it over 👍 , did not test run it though.

## Summary Addresses elastic#116329 Removes duplicate exception lists on rule export when multiple rules reference the same list.

kibanamachine · 2021-11-02T17:59:33Z

💚 Backport successful

Status	Branch	Result
✅	8.0
✅	7.16

The backport PRs will be merged automatically after passing CI.

## Summary Addresses #116329 Removes duplicate exception lists on rule export when multiple rules reference the same list. Co-authored-by: Yara Tercero <[email protected]>

yctercero added 3 commits October 26, 2021 20:50

add logic to remove duplicate exception list references on export

b46765c

Merge branch 'main' of github.com:elastic/kibana into exceptions_expo…

ae4a34c

…rt_duplicates

update check env now that we cant specify task manager and kib index

e28c458

yctercero self-assigned this Oct 28, 2021

yctercero marked this pull request as ready for review October 28, 2021 18:30

yctercero requested review from a team as code owners October 28, 2021 18:30

yctercero enabled auto-merge (squash) October 28, 2021 18:30

yctercero changed the title ~~Exceptions export duplicates~~ [Security Solution][Platform] - Exceptions export duplicates Oct 28, 2021

yctercero mentioned this pull request Oct 29, 2021

[Security Solution][Platform] - 7.16/8.0 Sanity Check #116336

Closed

21 tasks

brianseeders changed the base branch from main to master October 29, 2021 15:17

Merge branch 'main' into exceptions_export_duplicates

4354a6c

FrankHassanabad reviewed Oct 29, 2021

View reviewed changes

x-pack/plugins/lists/server/scripts/check_env_variables.sh Show resolved Hide resolved

FrankHassanabad reviewed Oct 29, 2021

View reviewed changes

x-pack/plugins/security_solution/server/lib/detection_engine/scripts/check_env_variables.sh Show resolved Hide resolved

FrankHassanabad reviewed Oct 29, 2021

View reviewed changes

yctercero added 3 commits November 1, 2021 13:36

Merge branch 'main' of github.com:elastic/kibana into exceptions_expo…

d590488

…rt_duplicates

update dedupe to use list id

b027d08

Merge branch 'exceptions_export_duplicates' of github.com:yctercero/k…

9199f68

…ibana into exceptions_export_duplicates

FrankHassanabad approved these changes Nov 2, 2021

View reviewed changes

yctercero merged commit b52a9ab into elastic:main Nov 2, 2021

kibanamachine mentioned this pull request Nov 2, 2021

[8.0] Exceptions export duplicates (#116698) #117197

Merged

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Nov 2, 2021

Exceptions export duplicates (elastic#116698)

29632be

## Summary Addresses elastic#116329 Removes duplicate exception lists on rule export when multiple rules reference the same list.

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Nov 2, 2021

Exceptions export duplicates (elastic#116698)

b4854fb

## Summary Addresses elastic#116329 Removes duplicate exception lists on rule export when multiple rules reference the same list.

kibanamachine mentioned this pull request Nov 2, 2021

[7.16] Exceptions export duplicates (#116698) #117198

Merged

yctercero deleted the exceptions_export_duplicates branch August 4, 2022 18:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Platform] - Exceptions export duplicates #116698

[Security Solution][Platform] - Exceptions export duplicates #116698

yctercero commented Oct 28, 2021 •

edited

Loading

elasticmachine commented Oct 28, 2021

yctercero commented Oct 29, 2021

FrankHassanabad Oct 29, 2021

yctercero Nov 1, 2021

kibanamachine commented Nov 1, 2021

FrankHassanabad left a comment

kibanamachine commented Nov 2, 2021

[Security Solution][Platform] - Exceptions export duplicates #116698

[Security Solution][Platform] - Exceptions export duplicates #116698

Conversation

yctercero commented Oct 28, 2021 • edited Loading

Summary

Testing

Sample Export

Checklist

elasticmachine commented Oct 28, 2021

yctercero commented Oct 29, 2021

FrankHassanabad Oct 29, 2021

Choose a reason for hiding this comment

yctercero Nov 1, 2021

Choose a reason for hiding this comment

kibanamachine commented Nov 1, 2021

💚 Build Succeeded

Metrics [docs]

History

FrankHassanabad left a comment

Choose a reason for hiding this comment

kibanamachine commented Nov 2, 2021

💚 Backport successful

yctercero commented Oct 28, 2021 •

edited

Loading