[SIEM] Change DNS fields for ECS

[andrewkroh]

The SIEM app currently uses DNS data on the Network page and the field names are based on Packetbeat. ECS has now added fields for DNS and the app should be updated to utilize the ECS fields rather than Packetbeat fields.

One change I know that's necessary is to update dns.question.etld_plus_one to dns.question.registered_domain. But we should double check the other fields and areas of the SIEM app.

Work to update Beats is in-progress.

[elasticmachine]

Pinging @elastic/siem

[cwurm]

We use DNS fields in two places: Network KPIs, and the Top DNS domains table (see #43649). Luckily, only dns.question.etld_plus_one seems to have changed.

Network KPI: DNS queries

We use dns.question.name:* OR suricata.eve.dns.type: query OR zeek.dns.query:* to determine the count of DNS events for the Network KPIs. We don't necessarily have to change it now, on the assumption that data sources will slowly migrate to have dns.question.name.

Network: Top DNS domains

We use:

1. dns.question.etld_plus_one - changed in ECS to dns.question.registered_domain
2. dns.question.type: PTR - in ECS as-is
3. dns.question.name - in ECS as-is

The problem is that if we change from querying dns.question.etld_plus_one to dns.question.registered_domain it will no longer work with existing Packetbeat data.

For 7.x, would filling dns.question.registered_domain but maintaining dns.question.etld_plus_one as an alias in the Beats templates work? Then we could keep using the current query in the UI.

[webmat]

For 7.x, would filling dns.question.registered_domain but maintaining dns.question.etld_plus_one as an alias in the Beats templates work?

Yes, creating this alias in Beats 7.4 will work for counts and aggregations as we expect. Whenever the document themselves are read however (as opposed to aggregation results) -- for example in a table listing -- we'll have to make sure we look for dns.question.registered_domain, and if not there, fall back to dns.question.etld_plus_one.

[webmat]

Clarifying my comment from yesterday. I think we should keep both fields populated until 8.0, not put in an alias just yet.

The alias only helps for operations that depend on the mapping, like aggs and searches. But whenever the source document is being used directly (table listings, further processing in the pipeline, etc) the field dns.question.etld_plus_one would be gone.

I think this is a significant enough breaking change, especially since it's a very useful field.

So I'd rather have both fields present.

If users think it's wasteful, they can always remove the old one. We should update the old field definition to say it's being deprecated in favor of dns.question.registered_domain.

[cwurm]

As discussed, let's change from dns.question.etld_plus_one to dns.question.registered_domain in the query and the draggable.

[webmat]

I think this can be closed, right?

[cwurm]

Yes, closing.