[meta] Update to ECS 1.2 to 1.4

[andrewkroh]

ECS 1.2 has been released with these changes. Beats should be updated where possible.

ECS Changes (copied from changelog)

• Added threat.* fields to apply a taxonomy to events and alerts.
• Added fields in log.* to allow for full Syslog mapping.
• Added package.* to installed software packages.
• Added registered_domain to url, source, destination, client, and server.
• Added top_level_domain field to url, dns.question, source, destination, client, and server.
• Added group.domain field.
• Added url.extension.
• Added observer.name and observer.product.
• Added dns.question.subdomain field.
• Added error.stack_trace field.
• Added log.origin.file.name, log.origin.function and log.origin.file.line fields.
• Added service.node.name to allow distinction between different nodes of the same service running on the same host.
• Added error.type field.

Changes to Beats

• Import latest ECS fields.yml to libbeat/_meta. Add ECS 1.2.0 fields.ecs.yml #14052

  • Removing any duplicate field definitions.
  • Test beat export template for each Beat.

• Update Auditbeat system/package metricset with new package fields.

• Add dns.question.top_level_domain.

  • Packetbeat
  • Filebeat Suricata
  • Filebeat Zeek
  • Winlogbeat Sysmon DNS
  • Filebeat CoreDNS

• Syslog fields

  • Filebeat syslog input

• TLS fields to Filebeat filesets [Filebeat] Add ECS TLS fields to existing filesets #15757

  • zeek/ssl (maybe other zeek filesets?)
  • aws/elb
  • aws/s3access

• Upgrade activemq module to ECS 1.4 [Filebeat] Upgrade activemq module to ECS 1.4 #16151

• Update apache module to support ECS 1.4 [Filebeat] Update apache/access fileset to support ECS 1.4 fields #16032

• Upgrade auditd module to ECS 1.4 [Filebeat] Upgrade auditd module to ECS 1.4 #16153

• Upgrade aws module to ECS 1.4 [Filebeat] Upgrade aws module to ECS 1.4 #16154

• Upgrade azure module to ECS 1.4 [Filebeat] Upgrade azure module to ECS 1.4 #16155

• Upgrade cef module to ECS 1.4 [Filebeat] Upgrade cef module to ECS 1.4 #16157

• Update cisco module to ECS 1.4 [Filebeat] Update cisco module to ECS 1.4 #16028

• Update elasticsearch module to ECS 1.4 [Filebeat] Upgrade elasticsearch module to ECS 1.4 #16160

• Update envoyproxy module to ECS 1.4 [Filebeat] Upgrade envoyproxy module to ECS 1.4 #16161

• Update googlecloud module to ECS 1.4 [Filebeat] Upgrade googlecloud module to ECS 1.4 #16030

• Update haproxy module to ECS 1.4 [Filebeat] Upgrade haproxy module to ECS 1.4 #16162

• Update ibmmq module to ECS 1.4 [Filebeat] Upgrade ibmmq module to ECS 1.4 #16163

• Update icinga module to ECS 1.4 [Filebeat] Upgrade icinga module to ECS 1.4 #16164

• Update iis module to ECS 1.4 [Filebeat] Upgrade iis module to ECS 1.4 #16165

• Update iptables module to ECS 1.4 [Filebeat] Upgrade iptables module to ECS 1.4 #16166

• Update kafka module to ECS 1.4 [Filebeat] Upgrade kafka module to ECS 1.4 #16167

• Update kibana module to ECS 1.4 [Filebeat] Upgrade kibana module to ECS 1.4 #16168

• Update logstash module to ECS 1.4 [Filebeat] Upgrade logstash module to ECS 1.4 #16169

• Update misp module to ECS 1.4 [Filebeat] Upgrade misp module to ECS 1.4 #16026

• Update mongodb module to ECS 1.4 [Filebeat] Upgrade mongodb module to ECS 1.4 #16170

• Update mssql module to ECS 1.4 [Filebeat] Upgrade mssql module to ECS 1.4 #16171

• Update mysql module to ECS 1.4 [Filebeat] Upgrade mysql module to ECS 1.4 #16172

• Update nats module to ECS 1.4 [Filebeat] Upgrade nats module to ECS 1.4 #16173

• Update netflow module to ECS 1.4 [Filebeat][Netflow] Populate new ECS fields for netflow #16135

• Update nginx module to ECS 1.4 [Filebeat] Upgrade nginx module to ECS 1.4 #16174

• Update osquery module to ECS 1.4 [Filebeat] Upgrade osquery module to ECS 1.4 #16176

• Update panw module to ECS 1.4 [Filebeat] Upgrade panw module to ECS 1.4 #16025

• Update postgresql module to ECS 1.4 [Filebeat] Upgrade postgresql module to ECS 1.4 #16177

• Update rabbitmq module to ECS 1.4 [Filebeat] Upgrade rabbitmq module to ECS 1.4 #16178

• Update redis module to ECS 1.4 [Filebeat] Upgrade redis module to ECS 1.4 #16179

• Update santa module to ECS 1.4 [Filebeat] Upgrade santa module to ECS 1.4 #16180

• Update suricata module to ECS 1.4 [Filebeat] Upgrade suricata module to ECS 1.4 #16181

• Update system module to ECS 1.4 [Filebeat] Upgrade system module to ECS 1.4 #16031

• Update traefik module to ECS 1.4 [Filebeat] Upgrade traefik module to ECS 1.4 #16183

• Update zeek module to ECS 1.4 [Filebeat] Update zeek module to ECS 1.4 #16029

• Please add addition things that need updated.

• Make sure modules populate source.address and destination.address any time the source/destination.ip field is used to allow for default field searches on IPs to work.

• Update Auditbeat network.direction values. [Auditbeat] auditd network.direction should use inbound/outbound #12445

• Add file.extension to Auditbeat FIM. [Auditbeat] Add file.extension field #7138

ECS 1.3 Changes

• Added vulnerability.* fields to represent vulnerability information. RabbitMQ output #581
• Added event.ingested as the ingest timestamp. Filebeat should start even when all input files are inaccessable. #582
• Added package.reference. filebeat: silently dies when Configfile is broken. #585
• Added package.build_version. Document bulk_max_size for all outputs, not just Elasticsearch #586
• Added package.type. Filebeat performance when sending to Logstash #587
• Added host.domain field. Fix for bad cross-document links #591
• Added process.command_line. Python system tests on Windows do not generate code coverage data #599
• Added process.exit_code. Replace magic value with magic const #600
• Added fields in tls.* to support analysis of TLS protocol events. remove some printf from reader #606
• Added process.parent.*. New protocol support: SSL #612
• Added process.args_count. Add missing nil check to pgsql module #615

ECS 1.4 Changes

• Added default text analyzer as a multi-field to user_agent.original. Add ability to queue/spool to disk #575
• Added file.attributes. Metricbeat Prototype #611
• Added file.drive_letter. Update glide dependencies #620
• Added rule fields. Move beat specific logic to docker-compose #665
• Added default text analyzer as a multi-field to around 25 more fields. Added nagioscheckbeat #680
• Added registry.* fieldset for the Windows registry. Excessive disk usage - filebeat #673
• Publish initial list of allowed values for the categorization fields (previously reserved) event.kind, event.category, event.type and event.outcome. Publisher options cleanup #684, Modify TLS description to clarify use of TCP #691, Add hearbeat to filebeat #692
• Added related.user Update example in the readme #694

[webmat]

Is this on track for 7.5 FF?

[cwurm]

We also need to govendor for ecs.version.

[andrewkroh]

@cwurm Thanks for the reminder. I pushed an update to #14052 with the vendor changes.

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[nemhods]

The new Winlogbeat 7.6.0 publishes events that claim to be ECS 1.4.0.
However, the security processor (https://github.com/elastic/beats/blob/master/x-pack/winlogbeat/module/security/config/winlogbeat-security.js) assigns illegal values to the field event.type:

• "authentication_success"/""authentication_failure"" should be expressed as event.type:access (?) and event.outcome:success/failure
• "process_start" and "process_end" should be "start" and "end"

I thought I'd report that here, or should i create a post on discuss.elastic.co?

[tsg]

Thanks for the report @nemhods. Reporting here is enough, and the adoption of ECS 1.4 among Beats is still in-progress targetting 7.7, so I hope we'll solve this by then.

[rw-access]

Checking in on this issue.
Will process_start/process_end be updated to start/end what about other event types? Should we (cc @elastic/security-intelligence-analytics) work with beats and ECS if necessary to help get these through? @randomuserid knows a few of these well

The more we follow ECS, the simpler it is for our rules to work across multiple data sources. Otherwise, we have to OR together the invalid values of fields event.type or rely on other fields like event.action which are implementation defined.

Update: I just checked and saw that both values are currently populated, which works fine. If @randomuserid finds more inconsistencies, I'll have him add them here.

[webmat]

Yes, the fields are arrays primarily to support events that fall into multiple categories.

But the array fields also let us keep bwc with the earlier values, prior to releasing the official list. 👍