Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Suricata for ECS DNS #13329

Merged
merged 4 commits into from
Aug 27, 2019
Merged

Update Suricata for ECS DNS #13329

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f8ba4d3
Update Suricata for ECS DNS
andrewkroh Aug 23, 2019
0f55384
Add registered_domain
andrewkroh Aug 26, 2019
1f15061
event.original, ensure destination.port is type long
andrewkroh Aug 26, 2019
ec003d8
Merge branch 'master' into feature/fb/suricata-ecs-dns
andrewkroh Aug 27, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Add module for ingesting IBM MQ logs. {pull}8782[8782]
- Add S3 input to retrieve logs from AWS S3 buckets. {pull}12640[12640] {issue}12582[12582]
- Add aws module s3access metricset. {pull}13170[13170] {issue}12880[12880]
- Update Suricata module to populate ECS DNS fields and handle EVE DNS version 2. {issue}13320[13320] {pull}13329[13329]
- Update PAN-OS fileset to use the ECS NAT fields. {issue}13320[13320] {pull}13330[13330]

*Heartbeat*
Expand Down
187 changes: 178 additions & 9 deletions x-pack/filebeat/module/suricata/eve/config/eve.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,184 @@ paths:
exclude_files: [".gz$"]
tags: {{.tags}}

json.keys_under_root: false

{{ if .community_id }}
processors:
- community_id:
- rename:
fields:
- {from: message, to: event.original}
- decode_json_fields:
fields: [event.original]
target: suricata.eve
- convert:
ignore_missing: true
ignore_failure: true
mode: rename
fields:
- {from: suricata.eve.src_ip, to: source.address}
- {from: suricata.eve.src_port, to: source.port, type: long}
- {from: suricata.eve.dest_ip, to: destination.address}
- {from: suricata.eve.dest_port, to: destination.port, type: long}
- {from: suricata.eve.proto, to: network.transport}
- convert:
ignore_missing: true
ignore_failure: true
mode: copy
fields:
source_ip: json.src_ip
source_port: json.src_port
destination_ip: json.dest_ip
destination_port: json.dest_port
transport: json.proto
- {from: source.address, to: source.ip, type: ip}
- {from: destination.address, to: destination.ip, type: ip}
- {from: '@timestamp', to: event.created}
- timestamp:
field: suricata.eve.timestamp
layouts:
- '2006-01-02T15:04:05.999999999Z0700' # ISO8601
- drop_fields:
fields:
- suricata.eve.timestamp
{{ if .community_id }}
- community_id:
{{ end }}
- if:
equals.suricata.eve.event_type: dns
then:
- convert:
ignore_missing: true
ignore_failure: true
mode: copy
fields:
- {from: suricata.eve.dns.id, to: dns.id, type: string}
- {from: suricata.eve.dns.rcode, to: dns.response_code}
- {from: suricata.eve.dns.type, to: dns.type}
- convert:
when.equals.dns.type: query
ignore_missing: true
ignore_failure: true
mode: copy
fields:
- {from: suricata.eve.dns.rrname, to: dns.question.name}
- {from: suricata.eve.dns.rrtype, to: dns.question.type}
# Handle the version=1 EVE DNS answer format. Each JSON event contains
# a single resource record from the DNS response.
- script:
when.and:
- equals.dns.type: answer
- not.has_fields: [suricata.eve.dns.version]
id: suricata_dns_answers_v1
lang: javascript
source: >
function process(evt) {
var name = evt.Get("suricata.eve.dns.rrname");
var data = evt.Get("suricata.eve.dns.rdata");
var type = evt.Get("suricata.eve.dns.rrtype");
var ttl = evt.Get("suricata.eve.dns.ttl");

var answer = {};
if (name) {
answer.name = name;
}
if (data) {
answer.data = data;
}
if (type) {
answer.type = type;
}
if (ttl) {
answer.ttl = ttl;
}

if (Object.keys(answer).length === 0) {
return;
}
evt.Put("dns.answers", [answer]);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's a DNS version 1 event, this will continue to create a single event per DNS answer still, correct?

I guess for backwards compatibility for users that use Suricata as a source, we don't really have a choice to keep it like that. But the semantics for DNS answers will not match ECS, therefore I'm not sure we'll be able to depend on Suricata DNS version 1 as a source for DNS answers.

I think for now SIEM only looks at queries, however. Correct?

cc @cwurm

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SIEM app has two DNS queries:

  1. Top DNS domains table: Terms agg on dns.question.etld_plus_one, sub-aggs on dns.question.name, source.bytes, and destination.bytes
  2. DNS queries KPI: Checks if dns.question.name exists among other things.

Since there is no filter on just queries or just answers I think we will miscount if there are several documents (for query and answer or one per answer).

# Handle the version=2 EVE DNS answer format.
- if:
and:
- equals.dns.type: answer
- equals.suricata.eve.dns.version: 2
then:
- convert:
ignore_missing: true
ignore_failure: true
mode: copy
fields:
- {from: suricata.eve.dns.rrname, to: dns.question.name}
- {from: suricata.eve.dns.rrtype, to: dns.question.type}
- script:
id: suricata_dns_answers_v2
lang: javascript
source: >
function transformDetailedAnswers(evt) {
var answers = evt.Get("suricata.eve.dns.answers");
if (!answers) {
return;
}
evt.Delete("suricata.eve.dns.answers");

var resolvedIps = [];
for (var i = 0; i < answers.length; i++) {
var answer = answers[i];

// Rename properties.
var name = answer["rrname"];
delete answer["rrname"];
var type = answer["rrtype"];
delete answer["rrtype"];
var data = answer["rdata"];
delete answer["rdata"];

answer["name"] = name;
answer["type"] = type;
answer["data"] = data;

// Append IP addresses to dns.resolved_ip.
if (type === "A" || type === "AAAA") {
resolvedIps.push(data);
}
}
evt.Put("dns.answers", answers);
if (resolvedIps.length > 0) {
evt.Put("dns.resolved_ip", resolvedIps);
}
}

function addDnsHeaderFlags(evt) {
var flag = evt.Get("suricata.eve.dns.aa");
if (flag === true) {
evt.AppendTo("dns.header_flags", "AA");
}

flag = evt.Get("suricata.eve.dns.tc");
if (flag === true) {
evt.AppendTo("dns.header_flags", "TC");
}

flag = evt.Get("suricata.eve.dns.rd");
if (flag === true) {
evt.AppendTo("dns.header_flags", "RD");
}

flag = evt.Get("suricata.eve.dns.ra");
if (flag === true) {
evt.AppendTo("dns.header_flags", "RA");
}
}

function process(evt) {
transformDetailedAnswers(evt);
addDnsHeaderFlags(evt);
}
- registered_domain:
ignore_missing: true
ignore_failure: true
field: dns.question.name
target_field: dns.question.registered_domain
- drop_fields:
ignore_missing: true
fields:
- suricata.eve.dns.aa
- suricata.eve.dns.tc
- suricata.eve.dns.rd
- suricata.eve.dns.ra
- suricata.eve.dns.qr
- suricata.eve.dns.version
- suricata.eve.dns.flags
- suricata.eve.dns.grouped
Loading