Fleet transform creating extra field mappings inside destination index

[kcreddy]

Kibana version: 8.14.0-SNAPSHOT

Elasticsearch version: 8.14.0-SNAPSHOT

Original install method (e.g. download page, yum, from source, etc.): elastic-package stack up --version=8.14.0-SNAPSHOT

Description of the problem including expected versus actual behavior: For ti_abusech package, the 4 latest transforms exist (1 for each datastream) to copy only the latest indicators from source datastream into the destination index. The latest_malware transform is incorrectly creating extra field mappings inside destination index that are not defined in any of its fields definitions. Interestingly, these extra field mappings are coming from another transform latest_malwarebazaar field definitions. This could be due to a bug in Fleet transform which is taking field definitions based on a pattern, malware vs malwarebazaar?

Here's a diff of the mappings that show extra fields inside malware's destination index. Notice extra field mappings such as abusech.malwarebazaar.* that shouldn't exist.

diff <(cat ~/abusech-malware-source.json | flatten-json) <(cat ~/abusech-malware-dest.json | flatten-json)
abusech-malware-source.json
abusech-malware-dest.json

> .properties.abusech.properties.malwarebazaar.properties.ioc_expiration_duration.ignore_above = 1024
> .properties.abusech.properties.malwarebazaar.properties.ioc_expiration_duration.type = "keyword"
> .properties.abusech.properties.malwarebazaar.properties.anonymous.type = "long"
> .properties.abusech.properties.malwarebazaar.properties.dhash_icon.ignore_above = 1024
> .properties.abusech.properties.malwarebazaar.properties.dhash_icon.type = "keyword"
> .properties.abusech.properties.malwarebazaar.properties.code_sign.properties.subject_cn.ignore_above = 1024
> .properties.abusech.properties.malwarebazaar.properties.code_sign.properties.subject_cn.type = "keyword"
> .properties.abusech.properties.malwarebazaar.properties.code_sign.properties.issuer_cn.ignore_above = 1024
> .properties.abusech.properties.malwarebazaar.properties.code_sign.properties.issuer_cn.type = "keyword"
> .properties.abusech.properties.malwarebazaar.properties.code_sign.properties.valid_to.ignore_above = 1024
> .properties.abusech.properties.malwarebazaar.properties.code_sign.properties.valid_to.type = "keyword"
> .properties.abusech.properties.malwarebazaar.properties.code_sign.properties.thumbprint.ignore_above = 1024
> .properties.abusech.properties.malwarebazaar.properties.code_sign.properties.thumbprint.type = "keyword"
> .properties.abusech.properties.malwarebazaar.properties.code_sign.properties.valid_from.type = "date"
> .properties.abusech.properties.malwarebazaar.properties.code_sign.properties.cscb_reason.ignore_above = 1024
> .properties.abusech.properties.malwarebazaar.properties.code_sign.properties.cscb_reason.type = "keyword"
> .properties.abusech.properties.malwarebazaar.properties.code_sign.properties.serial_number.ignore_above = 1024
> .properties.abusech.properties.malwarebazaar.properties.code_sign.properties.serial_number.type = "keyword"
> .properties.abusech.properties.malwarebazaar.properties.code_sign.properties.thumbprint_algorithm.ignore_above = 1024
> .properties.abusech.properties.malwarebazaar.properties.code_sign.properties.thumbprint_algorithm.type = "keyword"
> .properties.abusech.properties.malwarebazaar.properties.code_sign.properties.algorithm.ignore_above = 1024
> .properties.abusech.properties.malwarebazaar.properties.code_sign.properties.algorithm.type = "keyword"
> .properties.abusech.properties.malwarebazaar.properties.code_sign.properties.cscb_listed.type = "boolean"
> .properties.abusech.properties.malwarebazaar.properties.deleted_at.type = "date"
> .properties.abusech.properties.malwarebazaar.properties.intelligence.properties.mail.properties.Generic.ignore_above = 1024
> .properties.abusech.properties.malwarebazaar.properties.intelligence.properties.mail.properties.Generic.type = "keyword"
> .properties.abusech.properties.malwarebazaar.properties.intelligence.properties.mail.properties.IT.ignore_above = 1024
> .properties.abusech.properties.malwarebazaar.properties.intelligence.properties.mail.properties.IT.type = "keyword"
> .properties.abusech.properties.malwarebazaar.properties.intelligence.properties.downloads.type = "long"
> .properties.abusech.properties.malwarebazaar.properties.intelligence.properties.uploads.type = "long"
94a132,133
> .properties.threat.properties.indicator.properties.geo.properties.country_iso_code.ignore_above = 1024
> .properties.threat.properties.indicator.properties.geo.properties.country_iso_code.type = "keyword"
95a135,146
> .properties.threat.properties.indicator.properties.file.properties.x509.properties.not_after.type = "date"
> .properties.threat.properties.indicator.properties.file.properties.x509.properties.subject.properties.common_name.ignore_above = 1024
> .properties.threat.properties.indicator.properties.file.properties.x509.properties.subject.properties.common_name.type = "keyword"
> .properties.threat.properties.indicator.properties.file.properties.x509.properties.not_before.type = "date"
> .properties.threat.properties.indicator.properties.file.properties.x509.properties.public_key_algorithm.ignore_above = 1024
> .properties.threat.properties.indicator.properties.file.properties.x509.properties.public_key_algorithm.type = "keyword"
> .properties.threat.properties.indicator.properties.file.properties.x509.properties.serial_number.ignore_above = 1024
> .properties.threat.properties.indicator.properties.file.properties.x509.properties.serial_number.type = "keyword"
> .properties.threat.properties.indicator.properties.file.properties.x509.properties.issuer.properties.common_name.ignore_above = 1024
> .properties.threat.properties.indicator.properties.file.properties.x509.properties.issuer.properties.common_name.type = "keyword"
> .properties.threat.properties.indicator.properties.file.properties.extension.ignore_above = 1024
> .properties.threat.properties.indicator.properties.file.properties.extension.type = "keyword"
98a150,153
> .properties.threat.properties.indicator.properties.file.properties.mime_type.ignore_above = 1024
> .properties.threat.properties.indicator.properties.file.properties.mime_type.type = "keyword"
> .properties.threat.properties.indicator.properties.file.properties.name.ignore_above = 1024
> .properties.threat.properties.indicator.properties.file.properties.name.type = "keyword"
100a156,159
> .properties.threat.properties.indicator.properties.file.properties.hash.properties.sha1.ignore_above = 1024
> .properties.threat.properties.indicator.properties.file.properties.hash.properties.sha1.type = "keyword"
> .properties.threat.properties.indicator.properties.file.properties.hash.properties.sha384.ignore_above = 1024
> .properties.threat.properties.indicator.properties.file.properties.hash.properties.sha384.type = "keyword"
108a168,170
> .properties.threat.properties.indicator.properties.file.properties.elf.properties.telfhash.ignore_above = 1024
> .properties.threat.properties.indicator.properties.file.properties.elf.properties.telfhash.type = "keyword"
> .properties.threat.properties.indicator.properties.last_seen.type = "date"
118a181,182
> .properties.threat.properties.software.properties.alias.ignore_above = 1024
> .properties.threat.properties.software.properties.alias.type = "keyword"

Steps to reproduce:

1. Bring up the Elastic Stack using elastic-package stack up --version=8.14.0-SNAPSHOT -d -v
2. Install AbuseCH package enabling AbuseCH Malware payloads using Elastic Agent.
3. In Dev Tools, run GET logs-ti_abusech_latest.dest_malware-1/_mapping to get mappings of destination index created by the transform logs-ti_abusech.latest_malware-default-0.1.0 .
4. Verify that fields named abusech.malwarebazaar.* exist which were never defined by the transform field definitions.. There are also many threat.* ECS fields such as threat.software.alias which are present in the mappings, but were never added in the field definitions.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)