[Security Solution][Detections] Adds Indicator path config for indicator match rules

[rylnd]

Summary

This adds a new rule field, threat_indicator_path, that applies only to indicator match rules. Much like an ingest processor, users can use this field to define where their threat indicator can be found on their indicator documents. The value defaults to threat.indicator, the proposed ECS path, if unspecified.

Regardless of where the enrichment fields are found, they will be placed within the prospective ECS field, threat.indicator, as a nested object in the generated detection alert.

This configuration is critical for 7.12 to allow users to receive enrichments from the new filebeat modules, which currently do not conform to the proposed ECS but instead use threatintel.indicator.

Reviewer Notes:

In general: create an indicator index, create a rule, generate some events, and observe the resultant, enriched signals. Specifically:

1. Generate a custom indicator index:

   PUT /custom_indicators
   {
     "mappings": {
       "properties": {
         "@timestamp": {
           "type": "date"
         },
         "custom_indicator": {
           "properties": {
             "path": {
               "properties": {
                 "domain": {
                   "type": "keyword"
                 },
                 "description": {
                   "type": "text"
                 }
               }
             }
           }
         }
       }
     }
   }

2. Create some indicator data. The simplest approach here is to create an indicator(s) with some known value (e.g. your computer's hostname) that we can leverage later. In dev tools:

   POST custom_indicators/_doc
   {
     "@timestamp": "2021-02-10T00:41:06.527Z",
     "custom_indicator": {
       "path": {
         "domain": "my-laptop-hostname.local",
         "description": "This should also appear in the alert's threat.indicator!"
       }
     }
   }

3. Generate source events with values matching your indicator fields above. As an example, I run auditbeat, knowing that it will generate events with our laptop's hostname in host.name. However, you can also manually insert some documents into an index of your own creation; just ensure that they contain values that will cause the rule to generate a signal.

4. Create an indicator match rule against your indicator index and event data:

5. Peep those enriched signals:
   Note: we're temporarily using dev tools here due to some ongoing fields display issues, and the _source filter is for demo cleanliness, please remove

   GET .siem-signals-default/_search
   {
     "query": {
       "term": {
         "signal.rule.name": {
           "value": "Review Rule Name"
         }
       }
     },
     "_source": "threat.indicator"
   }

In the interest of getting this reviewed/merged I'm going to defer integration/UI tests to a subsequent PR.

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Unit or functional tests were updated or added to match the most common scenarios
• Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
• Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
  (https://github.com/elastic/kibana/blob/c29adfef29e921cc447d2a5ed06ac2047ceab552/src/dev/build/tasks/os_packages/docker_generator/resources/bin/kibana-docker)
• This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
• This was checked for cross-browser compatibility

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[rylnd]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: af0e0f1

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	7.5MB	7.5MB	+1.7KB

History

• 💔 Build #106706 failed 5da3b12
• 💔 Build #106310 failed c5004db
• 💔 Build #106302 failed d012c6c272147583eec69534fbeb59fc6a17b730

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[FrankHassanabad]

👍 You have some testing! Thanks

[FrankHassanabad]

nit: Suggest using a constant everywhere and put it in the common/constants area we have

[FrankHassanabad]

LGTM! Good straight forward plumbing.

[MikePaquette]

Thanks @rylnd for pulling this together.

Copy: any suggestions for something more descriptive than “Threat Indicator Path”? That’s the name of the underlying field, but it occurred to me that something like “Indicator Path Override” might be more self-explanatory, although that brought up further questions about threat/indicator terminology and how we switched from “threat match” to “indicator” match. Also looking for suggestions on the description text as well.

Generally the dialog in the rule create/edit steps should follow the naming of the rule type, which in this case is "indicator match", so I would vote for using "Indicator" rather than "Threat Indicator." (We expect that this rule can/will be used for matching other indicators besides threat indicators.)

Question, should we use "path" in this context? in ECS, the "path" concept is used for entities such as file system paths, but not generally for fields, field sets, and objects in an ECS-compatible document. In this case, it seems like we are asking the user to specify "where is the field set we should use to find the attributes of the indicator we're matching?" Thus I'm wondering if "Indicator field set" would be better.

The description could be adjusted accordingly, like "Specify the field set in which attributes of the indicator are located. These attributes will be used to enrich alerts that are generated by this rule. Defaults to threat.indicator.

bigger questions
I'm just sipping my first cup of tea, but trying to wrap my head around whether we actually need a separate field for this information in the first place? Is it truly an override? Would it make a simpler user experience if we derived it from the Indicator index field specified in Step 1? For example, if the rule_author specifies threatintel.indicator.ip as the indicator index field, we'd just use threatintel.indicator prefix as the path/field set from which to pull the indicator attributes when enriching the signal document. Similarly, if the data complied with ECS, and the rule_author specified threat.indicator.ip as the indicator index field, then we'd pull the enrichment information from the threat.indicator.* field set. Then having this new specification as an override would make sense - normally not needed, but if there's a use case, we're not thinking about, could give flexibility.

Also, do we need an override field to specify the similar path/field set where the enrichment information will be stored in the alert document? Again, keeping in mind that this rule type is a generic indicator match, and future ECS field sets might define other indicator enrichments that would also be stored in the alert document, we should allow that flexibility. Again, we could infer this value based on what is specified in the indicator index field, but allow an override. For example, if the rule_author specifies threatintel.indicator.ip or threat.indicator.ip as the indicator index field, we'd use threat.indicator.* as the path/field set to copy the indicator attributes to when enriching the signal document.

[MikePaquette]

@rylnd wondering if you had the chance to see my comments and questions above?

[rylnd]

Question, should we use "path" in this context? in ECS, the "path" concept is used for entities such as file system paths, but not generally for fields, field sets, and objects in an ECS-compatible document. In this case, it seems like we are asking the user to specify "where is the field set we should use to find the attributes of the indicator we're matching?" Thus I'm wondering if "Indicator field set" would be better.

"path" was fresh in my mind as that's what's used for nested queries, but I appreciate using terminology from ECS if possible👍 . My understanding of "field set" is it's literally "a set of fields," often (always?) sharing a common ancestor, rather than the location of that common ancestor as we're trying to describe.

bigger questions
I'm just sipping my first cup of tea, but trying to wrap my head around whether we actually need a separate field for this information in the first place? Is it truly an override? Would it make a simpler user experience if we derived it from the Indicator index field specified in Step 1? For example, if the rule_author specifies threatintel.indicator.ip as the indicator index field, we'd just use threatintel.indicator prefix as the path/field set from which to pull the indicator attributes when enriching the signal document. Similarly, if the data complied with ECS, and the rule_author specified threat.indicator.ip as the indicator index field, then we'd pull the enrichment information from the threat.indicator.* field set. Then having this new specification as an override would make sense - normally not needed, but if there's a use case, we're not thinking about, could give flexibility.

I really like this idea but I'm not sure it's possible without further restricting the structure of an indicator document. Currently the indicator fieldset can exist anywhere in a document (i.e. has n levels above it), and can itself have arbitrary data within it (n levels below it). I don't think we could derive threat.indicator from something like threat.indicator.file.hash unless we restricted the definition to e.g. "fieldset must end with indicator" or something like that.

Additionally, such a derivation would force documents to colocate both indicator values and enrichment metadata; while the current examples assume that the rule is matching within the indicator fieldset (e.g. threat.indicator.domain), there's nothing enforcing this behavior and the user can match on any arbitrary field within the indicator document.

Also, do we need an override field to specify the similar path/field set where the enrichment information will be stored in the alert document? Again, keeping in mind that this rule type is a generic indicator match, and future ECS field sets might define other indicator enrichments that would also be stored in the alert document, we should allow that flexibility. Again, we could infer this value based on what is specified in the indicator index field, but allow an override. For example, if the rule_author specifies threatintel.indicator.ip or threat.indicator.ip as the indicator index field, we'd use threat.indicator.* as the path/field set to copy the indicator attributes to when enriching the signal document.

This is a great point! As we start to provide other enrichment types we may very well need more configuration (or simply more rule types?). For now, the thought was that these rules would be used primarily for CTI and would thus populate threat.indicator.

TL;DR I'm leaning toward "Indicator path override"