[SIEM][Detection Engine] Adds lists feature flag and list values to the REST interfaces

[FrankHassanabad]

Summary

• [SIEM][Detection Engine] Allow simple value based lists in the REST interface #60022
• Adds the feature flag for simple list values
• Adds the boolean filters of "and", "and not" to further filter based on simple values
• Adds unit tests and e2e tests for the values.
• Most tests can include the simple list values but some have to be skipped until we move those to more functions or just enable simple list values as a permanent feature.
• DOES NOT FILTER ON THE VALUES JUST YET (That will be a follow on PR)

Testing:

To turn on/off the feature flag do this with an env variable (set this in your .bashrc/.zshrc):

export ELASTIC_XPACK_SIEM_LISTS_FEATURE=true

Expect to see this error in the console when the environment variable is set:

server    log   [11:41:16.245] [error][plugins][siem] You have activated the lists feature flag which is NOT currently supported for SIEM! You should turn this feature flag off immediately by un-setting the environment variable: ELASTIC_XPACK_SIEM_LISTS_FEATURE and restarting Kibana

Expect create and update to work when the environment variable is set and look like this:

./update_rule.sh ./rules/updates/update_list.json 
{
  "created_at": "2020-03-15T17:42:37.074Z",
  "updated_at": "2020-03-15T17:54:22.427Z",
  "created_by": "yo",
  "description": "Query with a list",
  "enabled": true,
  "false_positives": [],
  "from": "now-6m",
  "id": "c602e3f6-713b-4f43-9bdd-b60fbfead1c5",
  "immutable": false,
  "interval": "5m",
  "rule_id": "query-with-list",
  "language": "kuery",
  "output_index": ".siem-signals-hassanabad-frank-default",
  "max_signals": 100,
  "risk_score": 1,
  "name": "Query with a list",
  "query": "user.name: root or user.name: admin",
  "references": [],
  "severity": "high",
  "updated_by": "yo",
  "tags": [],
  "to": "now",
  "type": "query",
  "threat": [],
  "version": 6,
  "lists": [
    {
      "field": "source.ip",
      "boolean_operator": "and",
      "values": [
        {
          "name": "127.0.0.1",
          "type": "value"
        }
      ]
    },
    {
      "field": "host.name",
      "boolean_operator": "and not",
      "values": [
        {
          "name": "rock01",
          "type": "value"
        }
      ]
    }
  ],
  "status": "succeeded",
  "status_date": "2020-03-15T17:42:40.718Z",
  "last_success_at": "2020-03-15T17:42:40.718Z",
  "last_success_message": "succeeded"
}

./post_rule.sh ./rules/queries/query_with_list.json 
{
  "created_at": "2020-03-15T17:42:37.074Z",
  "updated_at": "2020-03-15T17:42:37.116Z",
  "created_by": "yo",
  "description": "Query with a list",
  "enabled": true,
  "false_positives": [],
  "from": "now-6m",
  "id": "c602e3f6-713b-4f43-9bdd-b60fbfead1c5",
  "immutable": false,
  "interval": "5m",
  "rule_id": "query-with-list",
  "language": "kuery",
  "output_index": ".siem-signals-hassanabad-frank-default",
  "max_signals": 100,
  "risk_score": 1,
  "name": "Query with a list",
  "query": "user.name: root or user.name: admin",
  "references": [],
  "severity": "high",
  "updated_by": "yo",
  "tags": [],
  "to": "now",
  "type": "query",
  "threat": [],
  "version": 1,
  "lists": [
    {
      "field": "source.ip",
      "boolean_operator": "and",
      "values": [
        {
          "name": "127.0.0.1",
          "type": "value"
        }
      ]
    },
    {
      "field": "host.name",
      "boolean_operator": "and not",
      "values": [
        {
          "name": "rock01",
          "type": "value"
        },
        {
          "name": "mothra",
          "type": "value"
        }
      ]
    }
  ]
}

./patch_rule.sh ./rules/patches/update_list.json   
{
  "created_at": "2020-03-15T18:02:52.434Z",
  "updated_at": "2020-03-15T18:02:57.675Z",
  "created_by": "yo",
  "description": "Query with a list",
  "enabled": true,
  "false_positives": [],
  "from": "now-6m",
  "id": "40b7c2fb-83b4-4820-bf7c-056f3a631126",
  "immutable": false,
  "interval": "5m",
  "rule_id": "query-with-list",
  "language": "kuery",
  "output_index": ".siem-signals-hassanabad-frank-default",
  "max_signals": 100,
  "risk_score": 1,
  "name": "Query with a list",
  "query": "user.name: root or user.name: admin",
  "references": [],
  "severity": "high",
  "updated_by": "yo",
  "tags": [],
  "to": "now",
  "type": "query",
  "threat": [],
  "version": 1,
  "lists": [
    {
      "field": "source.ip",
      "boolean_operator": "and",
      "values": [
        {
          "name": "127.0.0.1",
          "type": "value"
        }
      ]
    },
    {
      "field": "host.name",
      "boolean_operator": "and not",
      "values": [
        {
          "name": "rock01",
          "type": "value"
        },
        {
          "name": "mothra",
          "type": "value"
        }
      ]
    }
  ],
  "status": "succeeded",
  "status_date": "2020-03-15T18:02:56.426Z",
  "last_success_at": "2020-03-15T18:02:56.426Z",
  "last_success_message": "succeeded"
}

./get_rule_by_rule_id.sh query-with-list
{
  "created_at": "2020-03-15T18:10:07.657Z",
  "updated_at": "2020-03-15T18:10:08.479Z",
  "created_by": "yo",
  "description": "Query with a list",
  "enabled": true,
  "false_positives": [],
  "from": "now-6m",
  "id": "9854162b-003c-47be-af59-8c3c9545aafa",
  "immutable": false,
  "interval": "5m",
  "rule_id": "query-with-list",
  "language": "kuery",
  "output_index": ".siem-signals-hassanabad-frank-default",
  "max_signals": 100,
  "risk_score": 1,
  "name": "Query with a list",
  "query": "user.name: root or user.name: admin",
  "references": [],
  "severity": "high",
  "updated_by": "yo",
  "tags": [],
  "to": "now",
  "type": "query",
  "threat": [],
  "version": 1,
  "lists": [
    {
      "field": "source.ip",
      "boolean_operator": "and",
      "values": [
        {
          "name": "127.0.0.1",
          "type": "value"
        }
      ]
    },
    {
      "field": "host.name",
      "boolean_operator": "and not",
      "values": [
        {
          "name": "rock01",
          "type": "value"
        },
        {
          "name": "mothra",
          "type": "value"
        }
      ]
    }
  ],
  "status": "going to run",
  "status_date": "2020-03-15T18:10:10.738Z"
}

Expect these errors when the environment variable is not set:

./post_rule.sh ./rules/queries/query_with_list.json 
{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "[request body]: child \"lists\" fails because [\"lists\" is not allowed]"
}

./update_rule.sh ./rules/queries/query_with_list.json
{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "[request body]: child \"lists\" fails because [\"lists\" is not allowed]"
}

./patch_rule.sh ./rules/patches/update_list.json
{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "[request body]: child \"lists\" fails because [\"lists\" is not allowed]"
}

Expect that this is backwards compatible with the feature flag but not necessarily forwards compatible. This means:

• You can have older data that never had lists and it will show up as an empty list when you query it. (backwards compatible)

• You might have lists and remove the env. variable and get back items as if the list was not there for (forwards compatible)

• You can export without lists, flip on the env flag and import with newer lists feature (backwards compatible)

• You can export lists and it will not work with an older system (not forwards compatible)

Checklist

Delete any items that are not applicable to this PR.

• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[yctercero]

Curious, what's the .default([]) do after forbidden? I assumed that by adding forbidden, only undefined was allowed?

[FrankHassanabad]

It says that it is forbidden but the value will be an empty array for 7.7 on input. This keeps downstream logic and types simple to be just an empty array for 7.7 with the feature flag even off. That makes it where I don't have to use the feature flag very often and I don't have to worry about it accepting an undefined/null since it is an array.

Arrays are typically empty arrays and defaulted when they are empty compared to a value that is a non-array which would be undefined/null

[yctercero]

LGTM! Had pretty minor comments. Pulled down and ran through all the tests you described in your summary 👍.

[dhurley14]

All the changes look good. Ran locally and the steps worked well. LGTM.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 2cc0602

History

• 💚 Build #34237 succeeded 4c74754
• 💚 Build #34065 succeeded 78e165b
• 💔 Build #34046 failed 84c1ba6
• 💚 Build #33816 succeeded 606841c
• 💚 Build #33565 succeeded ca24a96

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream