New event categorization values to support threat intel use cases

[ebeahan]

Summary

This PR introduces three categorization field allowed values.

These fields were initially proposed in RFC 008 - threat intelligence fields and will be added as beta values to align with RFC 0008 currently being at stage 2.

event.kind: enrichment

Introduced here to support threat enrichment, but it will not be limited to that use case. Available for any ECS event that will be used to enrich others.

event.category: threat

Categorize events that contain details about threat actors' tactics, techniques, and procedures.

event.type: indicator

Sub-bucket threat enrichment indicators of compromise.

Questions for reviewers

• Do these descriptions capture the intent of each new value accurately?
• Particularly with event.catgory:threat, I limited the scope to cybersecurity threats in the description. Is this too narrow?
• Should the threat category include additional expected event types beyond indicator?

NOTE: Opening PR as draft until the functionality is added into the ECS tooling to include the beta tooltip for event categorization values Added in #1511

[peasead]

Suggested change

	- description: Relating to cybersecurity threats. Use this category to visualize
	- description: Use this category to visualize

[peasead]

Reference: https://github.com/elastic/ecs/pull/1510/files#r676902191

[peasead]

Suggested change

	- description: Relating to cybersecurity threats. Use this category to visualize
	- description: Use this category to visualize

[peasead]

Reference: https://github.com/elastic/ecs/pull/1510/files#r676902191

[peasead]

Suggested change

	- description: Relating to cybersecurity threats. Use this category to visualize
	- description: Use this category to visualize

[peasead]

Suggested change

	Relating to cybersecurity threats. Use this category to visualize and analyze events
	Use this category to visualize and analyze events

[peasead]

Suggested change

	- description: Relating to cybersecurity threats. Use this category to visualize
	- description: Use this category to visualize

[rylnd]

I had a similar thought to @peasead about generalizing the definition of threat, so I'm glad that's covered here 👍 .

The only other note I had here, broadly, was about the use of event.kind: enrichment. Coupled with the other sub-categories introduced here, it seems potentially redundant to define a new top-level event.kind solely for an intended use case.

I think the argument could be made that the data generated by filebeat's threatintel modules is more general than enrichment: it's intelligence; something (potentially) noteworthy that was observed and recorded. That might be a broad enough definition to simply reuse event, or maybe we generalize to intelligence or even observable, but IMO enrichment describes a use case as opposed to a category of data.

As an example, the dashboards that ship with filebeat threatintel modules describe the population of ingested data, and do not touch on enrichment at all, because that's a downstream process (brief aside: I could_ see event.kind: enrichment being used to capture an actual enrichment event, e.g. "event A was enriched with indicator B")

[peasead]

I had a similar thought to @peasead about generalizing the definition of threat, so I'm glad that's covered here 👍 .

The only other note I had here, broadly, was about the use of event.kind: enrichment. Coupled with the other sub-categories introduced here, it seems potentially redundant to define a new top-level event.kind solely for an intended use case.

I think the argument could be made that the data generated by filebeat's threatintel modules is more general than enrichment: it's intelligence; something (potentially) noteworthy that was observed and recorded. That might be a broad enough definition to simply reuse event, or maybe we generalize to intelligence or even observable, but IMO enrichment describes a use case as opposed to a category of data.

As an example, the dashboards that ship with filebeat threatintel modules describe the population of ingested data, and do not touch on enrichment at all, because that's a downstream process (brief aside: I could_ see event.kind: enrichment being used to capture an actual enrichment event, e.g. "event A was enriched with indicator B")

To be honest, I never quite understood the event.kind: enrichment for the data provided in a threat feed. I always thought event.kind: enrichment should be intelligence coupled with an observable or, at a minimum, multiple data points combined together...ideally all three.

I assumed it was for some schema requirement I had missed. I continually forgot to ask about it in the right audience.

I think that event.kind: event makes the most sense. I don't think observable is right because nothing has been observed and it's not generally "intelligence" until it is contextually relevant to the local environment.

You mentioned it's not in the dashboards, so that would be a pretty simple and non-impactful change (enrichments -> event), right?

[ebeahan]

@rylnd @peasead No objections from my end if we pause on introducing event.kind: enrichment.

However, the threatintel Beats modules are already actively using event.kind: enrichment: https://github.com/elastic/beats/blob/v7.13.4/x-pack/filebeat/module/threatintel/abusemalware/test/abusechmalware.ndjson.log-expected.json#L5.

Ideally, those modules will use valid ECS categorization values. If there's a change in direction, the Beats modules will also need to reflect that change, I believe.

[rylnd]

Ideally, those modules will use valid ECS categorization values. If there's a change in direction, the Beats modules will also need to reflect that change, I believe.

Yep, that's a big consideration that I should have mentioned. If we need some new value for event.kind I think that enrichment is acceptable, but I wanted to ensure that a) we do need a new event.kind value and that b) enrichment isn't a significant departure from the other values (usage vs categorization as I mentioned above).

@P1llus @dcode would you care to weigh in?

[dcode]

I think event.kind: enrichment makes a lot of sense for a general use case. The documents do not generally describe something that happened, but rather some additional knowledge we can combine with something that happened. So, it doesn't make sense to call the threatintel documents "events".

At the same time, there's plenty of use cases that it could open up: named geopoints/shapes, organizational department data for a given username or IP, CVE info (!)... I think once these mechanics are worked out, there's so much value that could be added.

I also agree with @peasead that observable and intelligence doesn't make sense.

[MikePaquette]

Hi @rylnd Thanks for moving this PR forward, and for sanity-checking with all of us on the event.kind value.

I am +1 for moving forward with the proposed event.kind:enrichment and its proposed definition.

a) I wanted to ensure that we do need a new event.kind value

We do. As @dcode mentions above, none of the existing allowed values is appropriate for additional knowledge we can combine with something that happened (e.g., events, metrics) to provide beneficial information and context to the analyst.

b) [I wanted to ensure that] enrichment isn't a significant departure from the other values (usage vs categorization as I mentioned above)

I don't think it's inconsistent, as it seems to "fit" quite well in this example, threat intel, and some other envisioned for the future, such as, again suggested by @dcode, named geopoints/shapes, organizational department data for a given username or IP, CVE info.

Brainstorming hypothetical future categorization triples to demonstrate consistent usage:

• event.kind:enrichment, event.category:threat, event.type:indicator (this example)
• event.kind:enrichment, event.category:geo, event.type:point (future geo points)
• event.kind:enrichment, event.category:geo, event.type:shape (future geo shapes)
• event.kind:enrichment, event.category:user, event.type:??? (future user data)
• event.kind:enrichment, event.category:vulnerability, event.type:cve (future CVE lists)
• etc.

[peasead]

Great feedback. TL;DR we're good with event.kind:enrichment.

I always understood indicators would be combined with other events to make enrichments and intelligence, I just wasn't aware "evil.com" would be considered an event.kind:enrichment when it was data provided by a feed and not enriched. Just data. I assumed once "evil.com" + geo (in the examples) happens, then the document would be the event.kind:enrichment.

This is why I didn't bring it up before as I assumed it was a schema implementation I wasn't thinking about properly. I assumed event.kind was for what data was not what it could be.

That's why I held my nose and suggested event.kind:event, because I wasn't sure that raw data provided by a threat feed met the bar for a new event.kind.

[ebeahan]

Thanks, all, for the discussion. Looks like we have a consensus to move forward with these additions.

I also propose these changes are backported to the 1.11 release branch even though it's frozen:

1. We're already using these values in the threatintel module, and I think it's better to have them formally established in ECS sooner than wait for 1.12.
2. The accompanying threat.* changes (Stage 2 changes for RFC 0008 - threat indicator fields #1471, Stage 2 changes for RFC 0021 - Threat enrichments #1478, Threat intelligence RFCs stage 2 addenda #1504) are already included in 1.11
3. There are no field changes made with this PR. It's only adding the new allowed values to the appropriate sections of the ECS docs.

Not expecting objections, but I did want to raise and open the opportunity for any feedback on including these changes for 1.11.

[ebeahan]

I'd appreciate a final 👀 and ✅ if there's no additional feedback.