[RFC] Threat Enrichment - Stage 1 #1400

rylnd · 2021-05-06T23:01:00Z

Much of this was taken from what was deleted from #1293 and is in various stages of completion. Will annotate and iterate accordingly.

RFC Preview

Notes

I've stuck with threat.enrichments for this field name, as it is agnostic of any "matching" verbiage that may not make sense universally, but that brings up its own question (see below). Of note, other potential names that were discussed:

threat.matches
threat.indicators
threat.indicator_matches

Outstanding Questions

Is matched.type sufficient to cover all enrichment cases?
- matched.type: "manual" or matched.type: "analyst" would cover the case where an analyst manually finds a matching indicator
- Moving forward as is; see summary below
- Would it ever be valid to enrich an event without a field/value to reference? E.g. if an analyst suspects a potential correlation?
How do I indicate that the threat.indicator fields are nested (as a nested array) under threat.enrichments?

Much of this was taken from what was deleted from elastic#1293 and is in various stages of completion. Will annotate and iterate on the PR.

rylnd · 2021-05-06T23:11:26Z

rfcs/text/0021-threat-enrichment.md

+              particular enrichment. If multiple matches for this indicator object, this could
+              be a list */
+        "matched": {
+          "atomic": "0c415dd718e3b3728707d579cf8214f54c2942e964975a5f925e0b82fea644b4",


I'm not convinced that matched.atomic and matched.field are sufficient to cover more sophisticated matching mechanisms than an exact match: if e.g. a user wanted to write an indicator match rule with a partial match, or if there were a more sophisticated indicator that itself represented a wildcard/regex value, then the value of the indicator would not be identical to the value from the event and so we may want two fields here.

Similarly, we would not be able to reproduce the exact conditions of the match with only one field value. While I've been told that the indicator field being matched upon should be self-evident, it seems safer to explicitly state it in another field, and I wanted to bring it up one more time before I shut up about it 😉 .

In general, I'm viewing these matched.* fields as the answer to both HOW and WHY a given event was enriched, so keep that in mind and/or correct me on that thinking.

a user wanted to write an indicator match rule with a partial match, or if there were a more sophisticated indicator that itself represented a wildcard/regex value, then the value of the indicator would not be identical to the value from the event

How are you envisioning this? Are you thinking that there is a mechanical way needed to match like fields. I can think of a use-case, DGAs.

So, if you know an aggressor uses a specific domain structure for C2 (like abc123\.12345abcdef\.xyz), you'd want to be able to match if they use

abc123\.12345abcdef\.xyz plo958\.59874qwersd\.xyz lje456\.01258iekduh\.xyz

So, you'd need a way to match url.full:/[aA-zZ]{3}[0-9]{3}\.[0-9]{5}[aA-zZ]{6}\.xyz MATCHES threat.indicator.url.full?

Or am I off?

@peasead right, the idea is that with anything other than an exact match, the values for the LHS and RHS of the match are going to be different, and we don't have fields to account for both of those right now.

IPs is another example, where one could specify a CIDR block instead of a single IP address.

What is SOP for DGAs within threat intel, currently? Is there any attempt to generalize the pattern/algorithm generating those values?

I'm not saying it's a must now.

There are some experimental machine learning jobs for DGAs.

I think we can continue with the implicit "exact value match" semantics for this RFC. If in the future we need to support the aforementioned functionality, I think that we can do so with the addition of two new fields as discussed.

rylnd · 2021-05-06T23:12:04Z

rfcs/text/0021-threat-enrichment.md

+}
+```
+
+### Proposed enrichment pipeline mechanics pseudocode


@dcode @peasead I know this section is out of date, but I had some trouble grokking and could use some help on this one.

@dcode, @rylnd and I chatted about this, but may still need some specifics if you could provide those.

@rylnd did you get the information you needed on this?

@peasead I think I finally grokked and updated that section appropriately. If you and @dcode can verify that what I wrote still makes sense, lemme know 👍

rfcs/text/0021/threat.yml

rfcs/text/0021-threat-enrichment.md

peasead

LGTM. Excited to see this taking shape.

ebeahan

Thanks for continuing to move this forward, @rylnd!

Do we have a sponsor for this RFC yet?

rfcs/text/0021-threat-enrichment.md

Co-authored-by: Eric Beahan <[email protected]>

rfcs/text/0021-threat-enrichment.md

* Removes unnecessary field renames, as fields no longer conflict * Adds a clause for setting a default array value for `threat.enrichments`

This is in fact redundant, but still useful.

* master: Stage 2 changes for RFC 0018 - extending the `threat.*` field set (elastic#1438) Remove deprecated `host.user.*` fields (elastic#1439) Explicitly include user identifiers in `related.user` field description (elastic#1420) Set the merge date on RFC 0018 stage 2 (elastic#1429) [RFC] Extend Threat Fieldset - Stage 2 Proposal (elastic#1395) [Tooling] Add --exclude flag to Generator to support field removal testing (elastic#1411) Add `host.user.*` deprecation notice in field reuse description (elastic#1422) Stage 2 changes for RFC 0015 - `elf` header (elastic#1410) Stage 3 changes for RFC 0012 - `orchestrator` field set (elastic#1417) Support `match_only_text` in Go code generator (elastic#1418) Stage 3 Orchestrator RFC (elastic#1343) moving into folder (elastic#1416) removing use-cases (elastic#1405) removing --oss (elastic#1404) Set the merge date on RFC 0015 stage 2 (elastic#1409) Consolidate `Breaking changes` sections in `CHANGELOG.next` (elastic#1408) RFC-Stage-0: Proposal to add a "ticket" schema / field definition to ECS (elastic#1383) [RFC] `match_only_text` type migration - Stage 0 (elastic#1396) Client port is wrongly documented (elastic#1402) (elastic#1406)

rylnd · 2021-05-28T22:11:36Z

Updated all outstanding threads; I think this is ready for review!

@devonakerr has agreed to be sponsor on this one, as well.

ebeahan

LGTM. Thanks, @rylnd, for the iterations here!

Can we capture @devonakerr as the sponsor under the People section in the document?

rylnd · 2021-06-03T19:13:30Z

rfcs/text/0021-threat-enrichment.md

+        "event": {
+          "provider": "Abuse.ch",
+          "dataset": "threatintel.abusemalware",
+          "module": "threatintel"


note: add event.reference here as well

devonakerr

Tidy PR, Ryland - LGTM

ebeahan · 2021-06-09T15:10:28Z

@rylnd Am I good to set the date and merge this PR?

You had this note; I wasn't sure if you wanted to make those changes here or in a later stage PR.

rylnd · 2021-06-09T18:02:20Z

@ebeahan I'll make that change right now, and then we should be good to merge here 👍

This provides a more complete example.

rylnd · 2021-06-09T18:06:19Z

@ebeahan all good on my end 👍

rylnd added 2 commits May 6, 2021 17:41

Adds initial Stage 1 information

f03979b

Much of this was taken from what was deleted from elastic#1293 and is in various stages of completion. Will annotate and iterate on the PR.

Undo noisy markdown style changes

f4a4465

rylnd added the RFC label May 6, 2021

Add Stage 1 PR link

1780349

rylnd commented May 6, 2021

View reviewed changes

rfcs/text/0021/threat.yml Show resolved Hide resolved

rylnd mentioned this pull request May 12, 2021

[DOCS] Updating "Manage detection alerts" topic elastic/security-docs#666

Merged

rylnd commented May 12, 2021

View reviewed changes

rfcs/text/0021-threat-enrichment.md Outdated Show resolved Hide resolved

ebeahan requested review from dcode, peasead and ebeahan May 13, 2021 13:45

peasead approved these changes May 18, 2021

View reviewed changes

ebeahan reviewed May 20, 2021

View reviewed changes

rfcs/text/0021-threat-enrichment.md Outdated Show resolved Hide resolved

rfcs/text/0021-threat-enrichment.md Outdated Show resolved Hide resolved

rfcs/text/0021-threat-enrichment.md Outdated Show resolved Hide resolved

Link to filebeat module

552b8f9

Co-authored-by: Eric Beahan <[email protected]>

rylnd mentioned this pull request May 24, 2021

[Security Solution][Detections] Alerts mappings conflict with experimental ECS threat fields elastic/kibana#100510

Open

rylnd added 3 commits May 24, 2021 16:59

Link to public kibana issue

deff482

Fully qualify our proposed fields table

2082ed9

Adds/updates missing enrichment fields in YML form

456a4d5

rylnd commented May 24, 2021

View reviewed changes

rfcs/text/0021-threat-enrichment.md Outdated Show resolved Hide resolved

rylnd added 4 commits May 28, 2021 16:57

Update enrichment pipeline pseudocode

1ad31d8

* Removes unnecessary field renames, as fields no longer conflict * Adds a clause for setting a default array value for `threat.enrichments`

Remove resolved TODO

bb3f5b5

This is in fact redundant, but still useful.

Add event fields to enrichment fieldset

9433c18

rylnd self-assigned this May 28, 2021

rylnd added 2 commits June 1, 2021 22:27

Merge branch 'master' into threat-enrichment-stage-1

421faa2

Merge branch 'master' into threat-enrichment-stage-1

f0421f5

ebeahan approved these changes Jun 2, 2021

View reviewed changes

ebeahan requested a review from devonakerr June 2, 2021 18:08

Add Devon K. as RFC sponsor

5bbf5f0

rylnd commented Jun 3, 2021

View reviewed changes

devonakerr approved these changes Jun 9, 2021

View reviewed changes

rylnd added 2 commits June 9, 2021 13:04

Add event.reference to example enrichment document

82f7c4a

This provides a more complete example.

Merge branch 'master' into threat-enrichment-stage-1

15df541

rylnd mentioned this pull request Jun 10, 2021

[Security Solution][CTI] Event enrichment search strategy elastic/kibana#101553

Merged

2 tasks

set advancement date for stage 1

f8fc9ce

ebeahan merged commit 676a9fe into elastic:master Jun 10, 2021

ebeahan mentioned this pull request Jun 14, 2021

Stage 1 changes for RFC 0021 - threat enrichment #1457

Merged

rylnd mentioned this pull request Jun 15, 2021

[RFC] Threat Enrichment - Stage 2 #1460

Merged

2 tasks

rylnd deleted the threat-enrichment-stage-1 branch June 22, 2021 15:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC] Threat Enrichment - Stage 1 #1400

[RFC] Threat Enrichment - Stage 1 #1400

rylnd commented May 6, 2021 •

edited

Loading

rylnd May 6, 2021

peasead May 17, 2021

rylnd May 24, 2021

peasead May 27, 2021

rylnd Jun 1, 2021

rylnd May 6, 2021

peasead May 17, 2021

peasead May 27, 2021

rylnd May 28, 2021

peasead left a comment

ebeahan left a comment

rylnd commented May 28, 2021

ebeahan left a comment

rylnd Jun 3, 2021

devonakerr left a comment

ebeahan commented Jun 9, 2021

rylnd commented Jun 9, 2021

rylnd commented Jun 9, 2021

[RFC] Threat Enrichment - Stage 1 #1400

[RFC] Threat Enrichment - Stage 1 #1400

Conversation

rylnd commented May 6, 2021 • edited Loading

Notes

Outstanding Questions

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peasead left a comment

Choose a reason for hiding this comment

ebeahan left a comment

Choose a reason for hiding this comment

rylnd commented May 28, 2021

ebeahan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devonakerr left a comment

Choose a reason for hiding this comment

ebeahan commented Jun 9, 2021

rylnd commented Jun 9, 2021

rylnd commented Jun 9, 2021

rylnd commented May 6, 2021 •

edited

Loading