[SIEM][Detection Engine] Allow simple value based lists in the REST interface #60022

FrankHassanabad · 2020-03-12T15:32:11Z

Meta ticket tracking everything
#59715

Parent issue this is split from
#59603

Describe the feature:
Allow value based lists to be added to rules such as this below:

Note, that adding lists are not supported yet in the "type" but the "type" will be of "value". This task should include the first feature flag for lists using a process env variable (for ease) and all the tests for adding this first part server side of list values.

{
  "name": "Query with a value list",
  "description": "Query with value lists",
  "rule_id": "query-with-value-lists",
  "risk_score": 1,
  "severity": "high",
  "type": "query",
  "query": "user.name: root or user.name: admin",
  "lists": [
    {
      "field": "source.ip",
      "boolean_operator": "and",
      "values": [
        {
          "name": "127.0.0.1",
          "type": "value"
        }
      ]
    },
    {
      "field": "host.name",
      "boolean_operator": "and not",
      "values": [
        {
          "name": "server-name-01",
          "type": "value"
        }
      ]
    }
  ]
}

elasticmachine · 2020-03-12T15:32:20Z

Pinging @elastic/siem (Team:SIEM)

…he REST interfaces ## Summary * #60022 * Adds the feature flag for simple list values * Adds the boolean filters of "and", "and not" to further filter based on simple values * Adds unit tests and e2e tests for the values. * Most tests can include the simple list values but some have to be skipped until we move those to more functions or just enable simple list values as a permanent feature. * DOES NOT FILTER ON THE VALUES JUST YET (That will be a follow on PR) ## Testing: To turn on/off the feature flag do this with an env variable (set this in your .bashrc/.zshrc): ```ts export ELASTIC_XPACK_SIEM_LISTS_FEATURE=true ``` Expect to see this error in the console when the environment variable is set: ```ts server log [11:41:16.245] [error][plugins][siem] You have activated the lists feature flag which is NOT currently supported for SIEM! You should turn this feature flag off immediately by un-setting the environment variable: ELASTIC_XPACK_SIEM_LISTS_FEATURE and restarting Kibana ``` Expect create and update to work when the environment variable is set and look like this: ```ts ./update_rule.sh ./rules/updates/update_list.json { "created_at": "2020-03-15T17:42:37.074Z", "updated_at": "2020-03-15T17:54:22.427Z", "created_by": "yo", "description": "Query with a list", "enabled": true, "false_positives": [], "from": "now-6m", "id": "c602e3f6-713b-4f43-9bdd-b60fbfead1c5", "immutable": false, "interval": "5m", "rule_id": "query-with-list", "language": "kuery", "output_index": ".siem-signals-hassanabad-frank-default", "max_signals": 100, "risk_score": 1, "name": "Query with a list", "query": "user.name: root or user.name: admin", "references": [], "severity": "high", "updated_by": "yo", "tags": [], "to": "now", "type": "query", "threat": [], "version": 6, "lists": [ { "field": "source.ip", "boolean_operator": "and", "values": [ { "name": "127.0.0.1", "type": "value" } ] }, { "field": "host.name", "boolean_operator": "and not", "values": [ { "name": "rock01", "type": "value" } ] } ], "status": "succeeded", "status_date": "2020-03-15T17:42:40.718Z", "last_success_at": "2020-03-15T17:42:40.718Z", "last_success_message": "succeeded" } ``` ```ts ./post_rule.sh ./rules/queries/query_with_list.json { "created_at": "2020-03-15T17:42:37.074Z", "updated_at": "2020-03-15T17:42:37.116Z", "created_by": "yo", "description": "Query with a list", "enabled": true, "false_positives": [], "from": "now-6m", "id": "c602e3f6-713b-4f43-9bdd-b60fbfead1c5", "immutable": false, "interval": "5m", "rule_id": "query-with-list", "language": "kuery", "output_index": ".siem-signals-hassanabad-frank-default", "max_signals": 100, "risk_score": 1, "name": "Query with a list", "query": "user.name: root or user.name: admin", "references": [], "severity": "high", "updated_by": "yo", "tags": [], "to": "now", "type": "query", "threat": [], "version": 1, "lists": [ { "field": "source.ip", "boolean_operator": "and", "values": [ { "name": "127.0.0.1", "type": "value" } ] }, { "field": "host.name", "boolean_operator": "and not", "values": [ { "name": "rock01", "type": "value" }, { "name": "mothra", "type": "value" } ] } ] } ``` ```ts ./patch_rule.sh ./rules/patches/update_list.json { "created_at": "2020-03-15T18:02:52.434Z", "updated_at": "2020-03-15T18:02:57.675Z", "created_by": "yo", "description": "Query with a list", "enabled": true, "false_positives": [], "from": "now-6m", "id": "40b7c2fb-83b4-4820-bf7c-056f3a631126", "immutable": false, "interval": "5m", "rule_id": "query-with-list", "language": "kuery", "output_index": ".siem-signals-hassanabad-frank-default", "max_signals": 100, "risk_score": 1, "name": "Query with a list", "query": "user.name: root or user.name: admin", "references": [], "severity": "high", "updated_by": "yo", "tags": [], "to": "now", "type": "query", "threat": [], "version": 1, "lists": [ { "field": "source.ip", "boolean_operator": "and", "values": [ { "name": "127.0.0.1", "type": "value" } ] }, { "field": "host.name", "boolean_operator": "and not", "values": [ { "name": "rock01", "type": "value" }, { "name": "mothra", "type": "value" } ] } ], "status": "succeeded", "status_date": "2020-03-15T18:02:56.426Z", "last_success_at": "2020-03-15T18:02:56.426Z", "last_success_message": "succeeded" } ``` ```ts ./get_rule_by_rule_id.sh query-with-list { "created_at": "2020-03-15T18:10:07.657Z", "updated_at": "2020-03-15T18:10:08.479Z", "created_by": "yo", "description": "Query with a list", "enabled": true, "false_positives": [], "from": "now-6m", "id": "9854162b-003c-47be-af59-8c3c9545aafa", "immutable": false, "interval": "5m", "rule_id": "query-with-list", "language": "kuery", "output_index": ".siem-signals-hassanabad-frank-default", "max_signals": 100, "risk_score": 1, "name": "Query with a list", "query": "user.name: root or user.name: admin", "references": [], "severity": "high", "updated_by": "yo", "tags": [], "to": "now", "type": "query", "threat": [], "version": 1, "lists": [ { "field": "source.ip", "boolean_operator": "and", "values": [ { "name": "127.0.0.1", "type": "value" } ] }, { "field": "host.name", "boolean_operator": "and not", "values": [ { "name": "rock01", "type": "value" }, { "name": "mothra", "type": "value" } ] } ], "status": "going to run", "status_date": "2020-03-15T18:10:10.738Z" } ``` Expect these errors when the environment variable is not set: ```ts ./post_rule.sh ./rules/queries/query_with_list.json { "statusCode": 400, "error": "Bad Request", "message": "[request body]: child \"lists\" fails because [\"lists\" is not allowed]" } ``` ```ts ./update_rule.sh ./rules/queries/query_with_list.json { "statusCode": 400, "error": "Bad Request", "message": "[request body]: child \"lists\" fails because [\"lists\" is not allowed]" } ``` ```ts ./patch_rule.sh ./rules/patches/update_list.json { "statusCode": 400, "error": "Bad Request", "message": "[request body]: child \"lists\" fails because [\"lists\" is not allowed]" } ``` Expect that this is _backwards_ compatible with the feature flag but not necessarily _forwards_ compatible. This means: * You can have older data that never had lists and it will show up as an empty list when you query it. (backwards compatible) * You _might_ have lists and remove the env. variable and get back items as if the list was not there for (forwards compatible) * You can export without lists, flip on the env flag and import with newer lists feature (backwards compatible) * You can export lists and it will _not_ work with an older system (not forwards compatible) ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios

…he REST interfaces ## Summary * elastic#60022 * Adds the feature flag for simple list values * Adds the boolean filters of "and", "and not" to further filter based on simple values * Adds unit tests and e2e tests for the values. * Most tests can include the simple list values but some have to be skipped until we move those to more functions or just enable simple list values as a permanent feature. * DOES NOT FILTER ON THE VALUES JUST YET (That will be a follow on PR) ## Testing: To turn on/off the feature flag do this with an env variable (set this in your .bashrc/.zshrc): ```ts export ELASTIC_XPACK_SIEM_LISTS_FEATURE=true ``` Expect to see this error in the console when the environment variable is set: ```ts server log [11:41:16.245] [error][plugins][siem] You have activated the lists feature flag which is NOT currently supported for SIEM! You should turn this feature flag off immediately by un-setting the environment variable: ELASTIC_XPACK_SIEM_LISTS_FEATURE and restarting Kibana ``` Expect create and update to work when the environment variable is set and look like this: ```ts ./update_rule.sh ./rules/updates/update_list.json { "created_at": "2020-03-15T17:42:37.074Z", "updated_at": "2020-03-15T17:54:22.427Z", "created_by": "yo", "description": "Query with a list", "enabled": true, "false_positives": [], "from": "now-6m", "id": "c602e3f6-713b-4f43-9bdd-b60fbfead1c5", "immutable": false, "interval": "5m", "rule_id": "query-with-list", "language": "kuery", "output_index": ".siem-signals-hassanabad-frank-default", "max_signals": 100, "risk_score": 1, "name": "Query with a list", "query": "user.name: root or user.name: admin", "references": [], "severity": "high", "updated_by": "yo", "tags": [], "to": "now", "type": "query", "threat": [], "version": 6, "lists": [ { "field": "source.ip", "boolean_operator": "and", "values": [ { "name": "127.0.0.1", "type": "value" } ] }, { "field": "host.name", "boolean_operator": "and not", "values": [ { "name": "rock01", "type": "value" } ] } ], "status": "succeeded", "status_date": "2020-03-15T17:42:40.718Z", "last_success_at": "2020-03-15T17:42:40.718Z", "last_success_message": "succeeded" } ``` ```ts ./post_rule.sh ./rules/queries/query_with_list.json { "created_at": "2020-03-15T17:42:37.074Z", "updated_at": "2020-03-15T17:42:37.116Z", "created_by": "yo", "description": "Query with a list", "enabled": true, "false_positives": [], "from": "now-6m", "id": "c602e3f6-713b-4f43-9bdd-b60fbfead1c5", "immutable": false, "interval": "5m", "rule_id": "query-with-list", "language": "kuery", "output_index": ".siem-signals-hassanabad-frank-default", "max_signals": 100, "risk_score": 1, "name": "Query with a list", "query": "user.name: root or user.name: admin", "references": [], "severity": "high", "updated_by": "yo", "tags": [], "to": "now", "type": "query", "threat": [], "version": 1, "lists": [ { "field": "source.ip", "boolean_operator": "and", "values": [ { "name": "127.0.0.1", "type": "value" } ] }, { "field": "host.name", "boolean_operator": "and not", "values": [ { "name": "rock01", "type": "value" }, { "name": "mothra", "type": "value" } ] } ] } ``` ```ts ./patch_rule.sh ./rules/patches/update_list.json { "created_at": "2020-03-15T18:02:52.434Z", "updated_at": "2020-03-15T18:02:57.675Z", "created_by": "yo", "description": "Query with a list", "enabled": true, "false_positives": [], "from": "now-6m", "id": "40b7c2fb-83b4-4820-bf7c-056f3a631126", "immutable": false, "interval": "5m", "rule_id": "query-with-list", "language": "kuery", "output_index": ".siem-signals-hassanabad-frank-default", "max_signals": 100, "risk_score": 1, "name": "Query with a list", "query": "user.name: root or user.name: admin", "references": [], "severity": "high", "updated_by": "yo", "tags": [], "to": "now", "type": "query", "threat": [], "version": 1, "lists": [ { "field": "source.ip", "boolean_operator": "and", "values": [ { "name": "127.0.0.1", "type": "value" } ] }, { "field": "host.name", "boolean_operator": "and not", "values": [ { "name": "rock01", "type": "value" }, { "name": "mothra", "type": "value" } ] } ], "status": "succeeded", "status_date": "2020-03-15T18:02:56.426Z", "last_success_at": "2020-03-15T18:02:56.426Z", "last_success_message": "succeeded" } ``` ```ts ./get_rule_by_rule_id.sh query-with-list { "created_at": "2020-03-15T18:10:07.657Z", "updated_at": "2020-03-15T18:10:08.479Z", "created_by": "yo", "description": "Query with a list", "enabled": true, "false_positives": [], "from": "now-6m", "id": "9854162b-003c-47be-af59-8c3c9545aafa", "immutable": false, "interval": "5m", "rule_id": "query-with-list", "language": "kuery", "output_index": ".siem-signals-hassanabad-frank-default", "max_signals": 100, "risk_score": 1, "name": "Query with a list", "query": "user.name: root or user.name: admin", "references": [], "severity": "high", "updated_by": "yo", "tags": [], "to": "now", "type": "query", "threat": [], "version": 1, "lists": [ { "field": "source.ip", "boolean_operator": "and", "values": [ { "name": "127.0.0.1", "type": "value" } ] }, { "field": "host.name", "boolean_operator": "and not", "values": [ { "name": "rock01", "type": "value" }, { "name": "mothra", "type": "value" } ] } ], "status": "going to run", "status_date": "2020-03-15T18:10:10.738Z" } ``` Expect these errors when the environment variable is not set: ```ts ./post_rule.sh ./rules/queries/query_with_list.json { "statusCode": 400, "error": "Bad Request", "message": "[request body]: child \"lists\" fails because [\"lists\" is not allowed]" } ``` ```ts ./update_rule.sh ./rules/queries/query_with_list.json { "statusCode": 400, "error": "Bad Request", "message": "[request body]: child \"lists\" fails because [\"lists\" is not allowed]" } ``` ```ts ./patch_rule.sh ./rules/patches/update_list.json { "statusCode": 400, "error": "Bad Request", "message": "[request body]: child \"lists\" fails because [\"lists\" is not allowed]" } ``` Expect that this is _backwards_ compatible with the feature flag but not necessarily _forwards_ compatible. This means: * You can have older data that never had lists and it will show up as an empty list when you query it. (backwards compatible) * You _might_ have lists and remove the env. variable and get back items as if the list was not there for (forwards compatible) * You can export without lists, flip on the env flag and import with newer lists feature (backwards compatible) * You can export lists and it will _not_ work with an older system (not forwards compatible) ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios

…he REST interfaces (#60678) ## Summary * #60022 * Adds the feature flag for simple list values * Adds the boolean filters of "and", "and not" to further filter based on simple values * Adds unit tests and e2e tests for the values. * Most tests can include the simple list values but some have to be skipped until we move those to more functions or just enable simple list values as a permanent feature. * DOES NOT FILTER ON THE VALUES JUST YET (That will be a follow on PR) ## Testing: To turn on/off the feature flag do this with an env variable (set this in your .bashrc/.zshrc): ```ts export ELASTIC_XPACK_SIEM_LISTS_FEATURE=true ``` Expect to see this error in the console when the environment variable is set: ```ts server log [11:41:16.245] [error][plugins][siem] You have activated the lists feature flag which is NOT currently supported for SIEM! You should turn this feature flag off immediately by un-setting the environment variable: ELASTIC_XPACK_SIEM_LISTS_FEATURE and restarting Kibana ``` Expect create and update to work when the environment variable is set and look like this: ```ts ./update_rule.sh ./rules/updates/update_list.json { "created_at": "2020-03-15T17:42:37.074Z", "updated_at": "2020-03-15T17:54:22.427Z", "created_by": "yo", "description": "Query with a list", "enabled": true, "false_positives": [], "from": "now-6m", "id": "c602e3f6-713b-4f43-9bdd-b60fbfead1c5", "immutable": false, "interval": "5m", "rule_id": "query-with-list", "language": "kuery", "output_index": ".siem-signals-hassanabad-frank-default", "max_signals": 100, "risk_score": 1, "name": "Query with a list", "query": "user.name: root or user.name: admin", "references": [], "severity": "high", "updated_by": "yo", "tags": [], "to": "now", "type": "query", "threat": [], "version": 6, "lists": [ { "field": "source.ip", "boolean_operator": "and", "values": [ { "name": "127.0.0.1", "type": "value" } ] }, { "field": "host.name", "boolean_operator": "and not", "values": [ { "name": "rock01", "type": "value" } ] } ], "status": "succeeded", "status_date": "2020-03-15T17:42:40.718Z", "last_success_at": "2020-03-15T17:42:40.718Z", "last_success_message": "succeeded" } ``` ```ts ./post_rule.sh ./rules/queries/query_with_list.json { "created_at": "2020-03-15T17:42:37.074Z", "updated_at": "2020-03-15T17:42:37.116Z", "created_by": "yo", "description": "Query with a list", "enabled": true, "false_positives": [], "from": "now-6m", "id": "c602e3f6-713b-4f43-9bdd-b60fbfead1c5", "immutable": false, "interval": "5m", "rule_id": "query-with-list", "language": "kuery", "output_index": ".siem-signals-hassanabad-frank-default", "max_signals": 100, "risk_score": 1, "name": "Query with a list", "query": "user.name: root or user.name: admin", "references": [], "severity": "high", "updated_by": "yo", "tags": [], "to": "now", "type": "query", "threat": [], "version": 1, "lists": [ { "field": "source.ip", "boolean_operator": "and", "values": [ { "name": "127.0.0.1", "type": "value" } ] }, { "field": "host.name", "boolean_operator": "and not", "values": [ { "name": "rock01", "type": "value" }, { "name": "mothra", "type": "value" } ] } ] } ``` ```ts ./patch_rule.sh ./rules/patches/update_list.json { "created_at": "2020-03-15T18:02:52.434Z", "updated_at": "2020-03-15T18:02:57.675Z", "created_by": "yo", "description": "Query with a list", "enabled": true, "false_positives": [], "from": "now-6m", "id": "40b7c2fb-83b4-4820-bf7c-056f3a631126", "immutable": false, "interval": "5m", "rule_id": "query-with-list", "language": "kuery", "output_index": ".siem-signals-hassanabad-frank-default", "max_signals": 100, "risk_score": 1, "name": "Query with a list", "query": "user.name: root or user.name: admin", "references": [], "severity": "high", "updated_by": "yo", "tags": [], "to": "now", "type": "query", "threat": [], "version": 1, "lists": [ { "field": "source.ip", "boolean_operator": "and", "values": [ { "name": "127.0.0.1", "type": "value" } ] }, { "field": "host.name", "boolean_operator": "and not", "values": [ { "name": "rock01", "type": "value" }, { "name": "mothra", "type": "value" } ] } ], "status": "succeeded", "status_date": "2020-03-15T18:02:56.426Z", "last_success_at": "2020-03-15T18:02:56.426Z", "last_success_message": "succeeded" } ``` ```ts ./get_rule_by_rule_id.sh query-with-list { "created_at": "2020-03-15T18:10:07.657Z", "updated_at": "2020-03-15T18:10:08.479Z", "created_by": "yo", "description": "Query with a list", "enabled": true, "false_positives": [], "from": "now-6m", "id": "9854162b-003c-47be-af59-8c3c9545aafa", "immutable": false, "interval": "5m", "rule_id": "query-with-list", "language": "kuery", "output_index": ".siem-signals-hassanabad-frank-default", "max_signals": 100, "risk_score": 1, "name": "Query with a list", "query": "user.name: root or user.name: admin", "references": [], "severity": "high", "updated_by": "yo", "tags": [], "to": "now", "type": "query", "threat": [], "version": 1, "lists": [ { "field": "source.ip", "boolean_operator": "and", "values": [ { "name": "127.0.0.1", "type": "value" } ] }, { "field": "host.name", "boolean_operator": "and not", "values": [ { "name": "rock01", "type": "value" }, { "name": "mothra", "type": "value" } ] } ], "status": "going to run", "status_date": "2020-03-15T18:10:10.738Z" } ``` Expect these errors when the environment variable is not set: ```ts ./post_rule.sh ./rules/queries/query_with_list.json { "statusCode": 400, "error": "Bad Request", "message": "[request body]: child \"lists\" fails because [\"lists\" is not allowed]" } ``` ```ts ./update_rule.sh ./rules/queries/query_with_list.json { "statusCode": 400, "error": "Bad Request", "message": "[request body]: child \"lists\" fails because [\"lists\" is not allowed]" } ``` ```ts ./patch_rule.sh ./rules/patches/update_list.json { "statusCode": 400, "error": "Bad Request", "message": "[request body]: child \"lists\" fails because [\"lists\" is not allowed]" } ``` Expect that this is _backwards_ compatible with the feature flag but not necessarily _forwards_ compatible. This means: * You can have older data that never had lists and it will show up as an empty list when you query it. (backwards compatible) * You _might_ have lists and remove the env. variable and get back items as if the list was not there for (forwards compatible) * You can export without lists, flip on the env flag and import with newer lists feature (backwards compatible) * You can export lists and it will _not_ work with an older system (not forwards compatible) ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios

) Summary - #60022 - Follow up on #60171 - Modifies boolean filters to enum of "included" and "excluded" - Adds operator types of enum "match", "match_all", "list", and "exists" - Adds values properties to include those for "list" - DOES NOT FILTER ON THE VALUES JUST YET (That will be a follow on PR)

…stic#62320) Summary - elastic#60022 - Follow up on elastic#60171 - Modifies boolean filters to enum of "included" and "excluded" - Adds operator types of enum "match", "match_all", "list", and "exists" - Adds values properties to include those for "list" - DOES NOT FILTER ON THE VALUES JUST YET (That will be a follow on PR)

) (#62956) Summary - #60022 - Follow up on #60171 - Modifies boolean filters to enum of "included" and "excluded" - Adds operator types of enum "match", "match_all", "list", and "exists" - Adds values properties to include those for "list" - DOES NOT FILTER ON THE VALUES JUST YET (That will be a follow on PR)

FrankHassanabad · 2020-07-29T16:44:58Z

"Reviewed by Frank Hassanabad on 7/29/2020", This looks complete and will be shippable at this point. Going to close this and associated tickets. Any new bugs/features should be new tickets.

Other notes are that this feature has been superseded by efforts to make exception lists a join and no longer data within an array.

FrankHassanabad added the Team:SIEM label Mar 12, 2020

FrankHassanabad self-assigned this Mar 12, 2020

yctercero mentioned this issue Mar 12, 2020

[SIEM][Detection Engine] - Value based lists #59603

Closed

4 tasks

FrankHassanabad changed the title ~~[SIEM][Detection Engine] Allow value based lists to the REST interface~~ [SIEM][Detection Engine] Allow simple value based lists in the REST interface Mar 12, 2020

This was referenced Mar 12, 2020

[SIEM][Detection Engine] Lists Meta Ticket #59715

Closed

[SIEM][Detection Engine] Adds lists feature flag and list values to the REST interfaces #60171

Merged

yctercero mentioned this issue Apr 2, 2020

[SIEM][Detection Engine] - Update list values in REST interfaces #62320

Merged

1 task

FrankHassanabad closed this as completed Jul 29, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SIEM][Detection Engine] Allow simple value based lists in the REST interface #60022

[SIEM][Detection Engine] Allow simple value based lists in the REST interface #60022

FrankHassanabad commented Mar 12, 2020 •

edited

Loading

elasticmachine commented Mar 12, 2020

FrankHassanabad commented Jul 29, 2020

[SIEM][Detection Engine] Allow simple value based lists in the REST interface #60022

[SIEM][Detection Engine] Allow simple value based lists in the REST interface #60022

Comments

FrankHassanabad commented Mar 12, 2020 • edited Loading

elasticmachine commented Mar 12, 2020

FrankHassanabad commented Jul 29, 2020

FrankHassanabad commented Mar 12, 2020 •

edited

Loading