[threat intelligence] logs-ti_abusech_latest.dest_threatfox-1 missing event.module field -> Threat Intel IP Match rule not working

[Danouchka]

Description
Abuse CH threat intel integration (2.0.0) is pulling ip addresses ioc in the data stream logs_ti-abusech.threatfox-default indice
The transform logs-ti_abusech.latest_threatfox-default-0.1.0 is pulling from there the iocs and send them in a destination index named logs-ti_abusech_latest.dest_threatfox-1.
But in this index the field event.module is missing and its missing because it's not in the mappings

Problem
the Threat Intel IP Match rule is looking in logs_ti-* with the following query :
@timestamp >= "now-30d/d" and event.module:(threatintel or ti_*) and threat.indicator.ip:* and not labels.is_ioc_transform_source:"true" and event.category:"threat" and event.kind:"enrichment" and event.type:"indicator"

As a result, since event.module is missing in the index, no match will be ever done for ip iocs coming from abuse ch threatfox

Same issues for other transforms below. The event.module field is missing in dest indexes and all Threat Intel * Math rules cannot trigger any alerts

• logs-ti_abusech.latest_malwarebazaar-default-0.1.0
• logs-ti_abusech.latest_malware-default-0.1.0
• logs-ti_abusech.latest_url-default-0.1.0
• ...

Many thanks for the correction

[Danouchka]

PS: using 8.13.4 on Elastic Cloud

[Danouchka]

cc @elastic/security-threat-intelligence cc @elastic/threat-research-and-detection-engineering

[kcreddy]

I was able to replicate this behaviour ( i.e., missing event.module inside destination index).

The destination index field event.module is defined in https://github.com/elastic/integrations/blob/main/packages/ti_abusech/elasticsearch/transform/latest_threatfox/fields/base-fields.yml#L10-L13.

Mapping difference between source-index and destination-index showed some fields missing from destination indices:

74,79c75,76
<       "properties": {
<         "is_ioc_transform_source": {
<           "type": "constant_keyword",
<           "value": "true"
<         }
<       }
---
>       "dynamic": true,
>       "type": "object"
429,440d425
<             }
<           }
<         },
<         "feed": {
<           "properties": {
<             "name": {
<               "type": "constant_keyword",
<               "value": "AbuseCH Threat Fox"
<             },
<             "dashboard_id": {
<               "type": "constant_keyword",
<               "value": "ti_abusech-c0d8d1f0-3b20-11ec-ae50-2fdf1e96c6a6"
479,482d463
<         "module": {
<           "type": "constant_keyword",
<           "value": "ti_abusech"
<         },
490,493d470
<         },
<         "dataset": {
<           "type": "constant_keyword",
<           "value": "ti_abusech.threatfox"
497c474,475
<   }

Note that, while is_ioc_transform_source field is intentionally removed to facilitate IOC expiration, other missing fields were all defined inside the same file https://github.com/elastic/integrations/blob/main/packages/ti_abusech/elasticsearch/transform/latest_threatfox/fields/base-fields.yml.

1. Also noticed that all the fields missing from the destination mapping i.e., event.module, event.dataset, threat.feed.name and threat.feed.dashboard_id are of same type constant_keyword while they are defined as keyword inside ECS. Could this mapping conflict be dropping them?
2. Even though event.dataset is missing from mappings, it is present inside the destination index document. The other difference I noticed is that except for event.dataset, the 3 other fields namely event.module, threat.feed.name, and threat.feed.dashboard_id were inside fields and not inside _source in the source indices.

@jsoriano / @andrewkroh Do you think transform is dropping these fields because it doesn't query source index's runtime fields? Is there anyway to mitigate this? I think one way to temporarily fix this is to create a pipeline (adding event.module) and add to transform's definition. I created this PR: #10042 for the same. Please let me know your thoughts.

[andrewkroh]

constant_keyword fields with a static value in the mapping may optionally include that same value in the _source, but it's better to omit the value to save on storage. Elastic Agent happens to be setting some of those fields in the _source (it sets various "dataset" fields), but not all of them.

@jsoriano / @andrewkroh Do you think transform is dropping these fields because it doesn't query source index's runtime fields?

It's reading the _source which doesn't include the static constant_keyword fields that are only present in the mapping. So it's not dropping them, instead it never saw them.

Is there anyway to mitigate this?

other missing fields were all defined inside the same file https://github.com/elastic/integrations/blob/main/packages/ti_abusech/elasticsearch/transform/latest_threatfox/fields/base-fields.yml.

I think you have done the correct thing which is to define the same constant_keyword mappings within the template for the transform index. But when I look at the logs-ti_abusech.latest_threatfox-template@package that is installed by TI AbuseCH 2.0.0 under stack 8.13.4, it does not have any of these fields. That seems like a bug in Fleet, unless there is some good reason that it's not including them in the template.

[andrewkroh]

If I move these fields from base-fields.yml to fields.yml I get a different template. For transforms, it seems like there is some lack of merging between fields in different files.

diff <(cat ~/Downloads/ti_abusech_mappings_orig.json | flatten-json) <(cat ~/Downloads/ti_abusech_mappings.json | flatten-json)
54,64c54,57
< .properties.event.properties.category.ignore_above = 1024
< .properties.event.properties.category.type = "keyword"
< .properties.event.properties.created.type = "date"
< .properties.event.properties.ingested.type = "date"
< .properties.event.properties.kind.ignore_above = 1024
< .properties.event.properties.kind.type = "keyword"
< .properties.event.properties.original.doc_values = false
< .properties.event.properties.original.index = false
< .properties.event.properties.original.type = "keyword"
< .properties.event.properties.type.ignore_above = 1024
< .properties.event.properties.type.type = "keyword"
---
> .properties.event.properties.dataset.type = "constant_keyword"
> .properties.event.properties.dataset.value = "ti_abusech.threatfox"
> .properties.event.properties.module.type = "constant_keyword"
> .properties.event.properties.module.value = "ti_abusech"
110,168c103,106
< .properties.threat.properties.indicator.properties.confidence.ignore_above = 1024
< .properties.threat.properties.indicator.properties.confidence.type = "keyword"
< .properties.threat.properties.indicator.properties.description.ignore_above = 1024
< .properties.threat.properties.indicator.properties.description.type = "keyword"
< .properties.threat.properties.indicator.properties.file.properties.elf.properties.telfhash.ignore_above = 1024
< .properties.threat.properties.indicator.properties.file.properties.elf.properties.telfhash.type = "keyword"
< .properties.threat.properties.indicator.properties.file.properties.extension.ignore_above = 1024
< .properties.threat.properties.indicator.properties.file.properties.extension.type = "keyword"
< .properties.threat.properties.indicator.properties.file.properties.hash.properties.md5.ignore_above = 1024
< .properties.threat.properties.indicator.properties.file.properties.hash.properties.md5.type = "keyword"
< .properties.threat.properties.indicator.properties.file.properties.hash.properties.sha1.ignore_above = 1024
< .properties.threat.properties.indicator.properties.file.properties.hash.properties.sha1.type = "keyword"
< .properties.threat.properties.indicator.properties.file.properties.hash.properties.sha256.ignore_above = 1024
< .properties.threat.properties.indicator.properties.file.properties.hash.properties.sha256.type = "keyword"
< .properties.threat.properties.indicator.properties.file.properties.hash.properties.sha384.ignore_above = 1024
< .properties.threat.properties.indicator.properties.file.properties.hash.properties.sha384.type = "keyword"
< .properties.threat.properties.indicator.properties.file.properties.hash.properties.ssdeep.ignore_above = 1024
< .properties.threat.properties.indicator.properties.file.properties.hash.properties.ssdeep.type = "keyword"
< .properties.threat.properties.indicator.properties.file.properties.hash.properties.tlsh.ignore_above = 1024
< .properties.threat.properties.indicator.properties.file.properties.hash.properties.tlsh.type = "keyword"
< .properties.threat.properties.indicator.properties.file.properties.mime_type.ignore_above = 1024
< .properties.threat.properties.indicator.properties.file.properties.mime_type.type = "keyword"
< .properties.threat.properties.indicator.properties.file.properties.name.ignore_above = 1024
< .properties.threat.properties.indicator.properties.file.properties.name.type = "keyword"
< .properties.threat.properties.indicator.properties.file.properties.pe.properties.imphash.ignore_above = 1024
< .properties.threat.properties.indicator.properties.file.properties.pe.properties.imphash.type = "keyword"
< .properties.threat.properties.indicator.properties.file.properties.size.type = "long"
< .properties.threat.properties.indicator.properties.file.properties.type.ignore_above = 1024
< .properties.threat.properties.indicator.properties.file.properties.type.type = "keyword"
< .properties.threat.properties.indicator.properties.first_seen.type = "date"
< .properties.threat.properties.indicator.properties.ip.type = "ip"
< .properties.threat.properties.indicator.properties.last_seen.type = "date"
< .properties.threat.properties.indicator.properties.name.ignore_above = 1024
< .properties.threat.properties.indicator.properties.name.type = "keyword"
< .properties.threat.properties.indicator.properties.port.type = "long"
< .properties.threat.properties.indicator.properties.provider.ignore_above = 1024
< .properties.threat.properties.indicator.properties.provider.type = "keyword"
< .properties.threat.properties.indicator.properties.reference.ignore_above = 1024
< .properties.threat.properties.indicator.properties.reference.type = "keyword"
< .properties.threat.properties.indicator.properties.type.ignore_above = 1024
< .properties.threat.properties.indicator.properties.type.type = "keyword"
< .properties.threat.properties.indicator.properties.url.properties.domain.ignore_above = 1024
< .properties.threat.properties.indicator.properties.url.properties.domain.type = "keyword"
< .properties.threat.properties.indicator.properties.url.properties.extension.ignore_above = 1024
< .properties.threat.properties.indicator.properties.url.properties.extension.type = "keyword"
< .properties.threat.properties.indicator.properties.url.properties.original.fields.text.type = "match_only_text"
< .properties.threat.properties.indicator.properties.url.properties.original.ignore_above = 1024
< .properties.threat.properties.indicator.properties.url.properties.original.type = "wildcard"
< .properties.threat.properties.indicator.properties.url.properties.path.ignore_above = 1024
< .properties.threat.properties.indicator.properties.url.properties.path.type = "wildcard"
< .properties.threat.properties.indicator.properties.url.properties.port.type = "long"
< .properties.threat.properties.indicator.properties.url.properties.scheme.ignore_above = 1024
< .properties.threat.properties.indicator.properties.url.properties.scheme.type = "keyword"
< .properties.threat.properties.software.properties.alias.ignore_above = 1024
< .properties.threat.properties.software.properties.alias.type = "keyword"
< .properties.threat.properties.software.properties.name.ignore_above = 1024
< .properties.threat.properties.software.properties.name.type = "keyword"
< .properties.threat.properties.software.properties.reference.ignore_above = 1024
< .properties.threat.properties.software.properties.reference.type = "keyword"
---
> .properties.threat.properties.feed.properties.dashboard_id.type = "constant_keyword"
> .properties.threat.properties.feed.properties.dashboard_id.value = "ti_abusech-c0d8d1f0-3b20-11ec-ae50-2fdf1e96c6a6"
> .properties.threat.properties.feed.properties.name.type = "constant_keyword"
> .properties.threat.properties.feed.properties.name.value = "AbuseCH Threat Fox"

ti_abusech_mappings.json
ti_abusech_mappings_orig.json

[kcreddy]

For transforms, tt seems like there is some lack of merging between fields in different files.

Thats what seems to be happening here.

The issue is with how the field mappings are built for the transform. It is fixed in elastic/kibana#177608 which aims at 8.14. As most ti_* packages have kibana.version constrained to ^8.12.0, a temporary fix could be made moving all fields having same prefix into same file.

For AbuseCH, this would mean moving all the 4 fields event.module, event.dataset, threat.feed.name, and threat.feed.dashboard_id into ecs.yml which already has event.* and threat.* fields. Created #10049 for having this fix.
@andrewkroh feel free to review/let me know your thoughts on this temporary fix. I can replicate this for other ti_* packages.

[kcreddy]

Fixed by #10049

Also another bug was identified in malware datastream, linking here: elastic/kibana#184759.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)