[SIEM][Detection Engine] - Value based lists

[yctercero]

Meta ticket tracking this
#59715

** Split/child ticket for just values**
#60022

⚠️ POC Ticket (Proof of Concept) ⚠️

As a user, add user generated list type values through UI or REST API.

• REST endpoints for value based CRUD (Create Read Update Delete)
• Create Saved Object mappings for list type values
  • Include some meta data on types
• Use a feature flag for env var / kibana.yml to prevent this from being exposed in 7.7 🙅‍♂
• Add tests to ensure feature flag is not set

Unknowns

• Would it be one document or multiple documents (one document per list)?
• Do we constrain list size to begin with?

This is looking unlikely at this phase but rather we keep lists separate from the query to make it possible if we need to do chunked querying it is simpler:

• How do we add to KQL and Lucene AST (integrating lists into queries)? See this ticket and add that feature through: [SIEM][Detection Engine] Add list syntax to KQL, lucene #59716

Research

• Someone has to look at the existing integration with MISP, Zeek, Suricata from beats and how we can eventually "point" to existing lists?
• De-dupes (for future). Duplicates can be between different lists or even in the same list.
• ES storage of large values (max size appears to be 2GB)

Mock REST request for adding a set of lists to an existing rule

{
  "name": "Query with a value list",
  "description": "Query with value lists",
  "rule_id": "query-with-value-lists",
  "risk_score": 1,
  "severity": "high",
  "type": "query",
  "query": "user.name: root or user.name: admin",
  "lists": [
    {
      "field": "source.ip",
      "boolean_operator": "and",
      "values": [
        {
          "name": "127.0.0.1",
          "type": "value"
        }
      ]
    },
    {
      "field": "host.name",
      "boolean_operator": "and not",
      "values": [
        {
          "name": "server-name-01",
          "type": "value"
        },
        {
          "name": "known_hosts",
          "type": "list"
        }
      ]
    }
  ]
}

Example of some lists within ES from the REST interface.

Other things this will have would be the classic fields from saved objects.

{
	"id": "1",
	"type": "homogenous"
	"data_type": "ip"
	"name": "BAD IPS"
	"ip": ['127.0.0.1', '127.0.0.2', ...]
}
{
	"id": "2",
	"type": "homogenous"
	"data_type": "ip"
	"name": "BAD IPS"
	"ip": ['127.0.0.2', ...]
}
{
	"id": "3",
	"type": "homogenous"
	"data_type": "string"
	"name": "Executives"
	"string": ['frank', 'yara']
}
{
	"id": "4",
	"type": "homogenous"
	"data_type": "date"
	"name": "Some dates"
	"meta": { "format": "YYYY/MM/DD" }
	"string": ['2001/01/01']
}

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[willemdh]

+1, also maybe this feature has some overlap with ml filter lists?

[FrankHassanabad]

"Reviewed by Frank Hassanabad on 7/29/2020", This looks complete and will be shippable at this point. Going to close this and associated tickets. Any new bugs/features should be new tickets. Note this work is superseded by exception lists being a join data type.