[SIEM] Non-ECS fields in SIEM

[cwurm]

I went through the SIEM app code to find any fields that are not in ECS. Some we can fix today, some we have to wait for ECS to add them.

The SIEM app uses fields in many places, so it's likely I missed some, but the main ones are hopefully here.

Autonomous System

ASN info is now in ECS master (ECS docs). There are several places where we use out of data field names. This is something we can fix today.

• ip_details/query_overview.dsl.ts (code):

  1. autonomous_system - should be as. Alternatively, we can use the Ingest Pipeline Simulate API to populate Location and ASN (https://github.com/elastic/siem-team/issues/310).

• field_renderers.tsx (code)

  1. autonomous_system.as_org - should be as.organization.name
  2. autonomous_system.asn - should be as.number

TLS

We don't have TLS in ECS yet, so the TLS table on the IP Details page and the TLS Handshakes KPI on the Network page are using other fields such as those currently filled by the Packetbeat TLS protocol decoder (Packetbeat docs).

• ip_details/query_tls.dsl.ts (code):

  1. tls.server_certificate.fingerprint.sha1
  2. tls.server_certificate.issuer.common_name
  3. tls.server_certificate.subject.common_name
  4. tls.server_certificate.alternative_names
  5. tls.server_certificate.not_after
  6. tls.fingerprints.ja3.hash

• kpi_network/query_tls_handshakes.dsl.ts (code)

  1. tls.version
  2. suricata.eve.tls.version
  3. zeek.ssl.version

• ja3_fingerprint/index.tsx (code)

  1. tls.fingerprints.ja3.hash

• certificate_fingerprint/index.tsx (code)

  1. tls.client_certificate.fingerprint.sha1
  2. tls.server_certificate.fingerprint.sha1

DNS

DNS is almost merged into ECS (elastic/ecs#438). Some of the fields the SIEM app now uses are in the PR, and some are not. We will first have to change the data sources (Packetbeat, Filebeat, Winlogbeat) and then we can look into how and when to change the queries.

• kpi_network/query_dns.dsl.ts (code)

  1. dns.question.name
  2. suricata.eve.dns.type: query
  3. zeek.dns.query

• network/query_dns.dsl.ts (code)

  1. dns.question.etld_plus_one
  2. dns.question.type: PTR
  3. dns.question.name

Event Categories

The Overview page is trying to query for which data is available. In the absence of a canonical "this is this kind of data" field in ECS it uses various combinations that are agent-specific. This makes it hard to treat third-party data (even data from Logstash) the same as Beats data.

Same for the Uncommon Processes table trying to determine what a process is. For that, Winlogbeat has recently added event.category=process/event.type: process_start/process_end (elastic/beats#13047).

• overview/query.dsl.ts (code)

  1. type: flow
  2. type: dns
  3. service.type: suricata
  4. service.type: zeek
  5. event.dataset: socket
  6. agent.type: filebeat AND input.type: netflow
  7. agent.type: filebeat AND event.module: panw
  8. agent.type: filebeat AND event.module: cisco
  9. agent.type: packetbeat AND network.protocol: tls
  10. event.module: auditd
  11. event.module: file_integrity
  12. agent.type: winlogbeat
  13. event.module: system AND event.dataset: login
  14. event.module: system AND event.dataset: package
  15. event.module: system AND event.dataset: process
  16. event.module: system AND event.dataset: user
  17. agent.type: filebeat

• uncommon_processes/query.dsl.ts (code)

  1. agent.type: filebeat AND event.module: auditd AND event.action: executed
  2. agent.type: auditbeat AND event.module: system AND event.dataset: process
  3. agent.type: winlogbeat AND event.code: 4688
  4. winlog.event_id: 1 AND winlog.channel: Microsoft-Windows-Sysmon/Operational

/cc @webmat @MikePaquette

[elasticmachine]

Pinging @elastic/siem

[MikePaquette]

thanks @cwurm for pulling this together.

[stephmilovic]

autonomous_system to as ecs: #43925

[andrew-goldstein]

Endgame

Endgame row renderers, added to 7.5 with this PR, use the following non-ECS fields:

endgame.exit_code
endgame.file_name
endgame.file_path
endgame.logon_type
endgame.parent_process_name
endgame.pid
endgame.process_name
endgame.subject_domain_name
endgame.subject_logon_id
endgame.subject_user_name
endgame.target_domain_name
endgame.target_logon_id
endgame.target_user_name

Note that some of these fields, such as endgame.subject* and endgame.target* do not have ECS equivalents as of this writing (Elastic stack 7.5 / ECS 1.2). See Flesh out support for multiple users in an event #589 for more details.

[andrewkroh]

An update on TLS: As of ECS 1.4 the necessary TLS fields are present. In Packetbeat 7.6 we made the change to adopt ECS for all of the TLS fields. Now the SIEM UI needs updated.

[antcodd]

#60584 in 7.7 doesn't seem to have updated the TLS fields used in row renderers, such as certificate fingerprints, it only updated the main table.