[Filebeat] [SIEM] Fileset for Cisco FTD logs #13286

adriansr · 2019-08-19T15:48:13Z

This adds a new ftd fileset to the cisco module for parsing Firepower Threat Defense logs.

As the FTD logs are a superset of the Cisco ASA logs, this PR introduces a shared ingest pipeline that is used both by the new ftd and the existing asa filesets.

As a side effect of this, it improves the existing ASA fileset so that it uses custom syslog message decoding instead of relying on Filebeat's syslog input, which has caused compatibility problems with some ASA devices.

Closes #12690

elasticmachine · 2019-08-19T17:24:55Z

Pinging @elastic/siem

adriansr · 2019-08-19T17:33:13Z

x-pack/filebeat/module/cisco/asa/config/input.yml

@@ -1,8 +1,8 @@
 {{ if eq .input "syslog" }}



I left the input configuration as syslog in order to not break existing installs. Does it makes sense, or it should align with the underlying input name?

I think that's a good decision for existing users and consistent with other modules.

x-pack/filebeat/module/cisco/ftd/_meta/fields.yml

x-pack/filebeat/module/cisco/ftd/test/security-connection.log-expected.json

webmat

Having reviewed the ECS 1.1 import, I knew which files to skip in this PR, so I started reviewing it already. I haven't had time to finish my review yet, but thought I'd share the two things I've seen so far.

I'll continue my review on Monday :-)

webmat · 2019-08-23T19:45:17Z

x-pack/filebeat/module/cisco/ftd/test/no-type-id.log-expected.json

+        "input.type": "log",
+        "log.level": "debug",
+        "log.offset": 0,
+        "log.original": "ApplicationProtocol: http, Client: webserver, DstIP: 10.8.12.47, SrcIP: 10.1.123.45, Message: Intrusion attempt",


Sorry, there's confusion between log.original and event.original 🤦‍♂

Based on elastic/ecs#127, I think this should go to event.original.

cc @ruflin

Renamed, but will keep log.original for 7.x to avoid introducing a breaking change in the ASA fileset.

This module is in Beta, correct? I think it's worth preserving for 7.3.x if it was already out in 7.3, but I wouldn't keep log.original until 8.0.

The problem is that this FTD module builds on top of the ASA module which came out in 7.2. Both FTD and ASA share the same pipeline because FTD logs are just ASA logs with 5 extra message types.

I understand it came out in a previous version.

But there's often confusion between log.original and event.original. Even I am struggling with the purpose of log.original, as it seems very wasteful to keep two versions of the full original event, one with slight modifications vs the other.

So I'm afraid if we keep log.original for the life of 7.x (roughly another year), it will just help carry the confusion forward.

That's why I was bringing up the fact that since it's in Beta, perhaps we can introduce this breaking change and try to keep alignment on using event.original.

x-pack/filebeat/module/cisco/ftd/_meta/fields.yml

adriansr · 2019-08-27T13:27:59Z

@webmat can you have another look?

webmat · 2019-08-27T13:29:29Z

Yes, today. This has been rebased over the ECS 1.1.0 import?

adriansr · 2019-08-27T15:17:10Z

Yes, it's rebased.

webmat

Ok, I have a lot of comments. Not 100% need to be addressed, there's a few instances of calling out to ECS contributors.

There's also a few places where I may be wishing for something to be parsed out in more detail, but perhaps it's too deep in the message to be extracted at this time. This is fine.

But I've still noticed many things I think should be modified.

I'm not 100% done with this review. Note to self:

asa-ftd-pipeline.yml
security-mappings.csv

Speaking of mappings, I only realized the mappings were generated based on the CSV very late in my review. So there's a lot of comments on the mappings scattered around the PR (docs, expected JSON, etc). Sorry :-\

webmat · 2019-08-27T15:08:50Z

x-pack/filebeat/module/cisco/ftd/test/dns.log-expected.json

+        "event.duration": "0",
+        "event.module": "cisco",
+        "event.original": "%FTD-1-430003: AccessControlRuleAction: Allow, SrcIP: 10.0.1.20, DstIP: 8.8.8.8, SrcPort: 57379, DstPort: 53, Protocol: udp, IngressInterface: inside, EgressInterface: outside, IngressZone: input-zone, EgressZone: output-zone, ACPolicy: default, AccessControlRuleName: Intrusion-Rule, Prefilter Policy: Default Prefilter Policy, User: No Authentication Required, Client: DNS client, ApplicationProtocol: DNS, ConnectionDuration: 0, InitiatorPackets: 1, ResponderPackets: 1, InitiatorBytes: 93, ResponderBytes: 145, NAPPolicy: Balanced Security and Connectivity, DNSQuery: elastic.co, DNSRecordType: a host address, DNS_TTL: 70",
+        "event.outcome": "allow",


event.outcome is reserved. This is risky, as it may change.

But since the module is still in Beta, I suppose it's fine.

That's bad. event.outcome is in use by plenty of modules in SIEM.

Yes, I know. I pushed back on reserving fields that everybody would want to use for that reason. When that didn't fly, I proposed we introduce the equivalent 4 fields for the raw value at the same time, so that people could record these values there, and I got push back on that. 🤷‍♂

I'll open a PR for the raw fields soon after this FF and try to push it through. It won't change anything for this PR and the modules using the reserved fields, but at least there will be a sensible way to resolve this from then on.

At that time we can figure out what we do with all this.

webmat · 2019-08-27T15:11:01Z

x-pack/filebeat/module/cisco/ftd/test/dns.log-expected.json

+            "cisco-ftd"
+        ],
+        "user.id": "No Authentication Required",
+        "user.name": "No Authentication Required"


I would skip the mapping to user.[id|name] when it's a generic value like this.

Can be fixed later, though.

x-pack/filebeat/module/cisco/ftd/test/dns.log-expected.json

filebeat/docs/modules/cisco.asciidoc

webmat · 2019-08-27T18:25:49Z

x-pack/filebeat/module/cisco/ftd/test/security-connection.log-expected.json

+        "dns.response_code": "NOERROR",
+        "event.action": "connection-started",
+        "event.code": "430002",
+        "event.dataset": "cisco.ftd",


With so many different types of logs emitted by Cisco FTD, I'm actually wondering if "event.dataset": "cisco.ftd" is granular enough...

Not a blocker for this PR, but @ruflin what would you think about having the Cisco FTD datasets being more granular?

"cisco.ftd.dns" for x-pack/filebeat/module/cisco/ftd/test/dns.log

"cisco.ftd.management" for x-pack/filebeat/module/cisco/ftd/test/firepower-management.log

"cisco.ftd.intrusion" for x-pack/filebeat/module/cisco/ftd/test/intrusion.log

"cisco.ftd.connection" for x-pack/filebeat/module/cisco/ftd/test/security-connection.log

"cisco.ftd.malware" for x-pack/filebeat/module/cisco/ftd/test/security-file-malware.log

x-pack/filebeat/module/cisco/ftd/test/security-connection.log-expected.json

x-pack/filebeat/module/cisco/ftd/test/security-file-malware.log-expected.json

x-pack/filebeat/module/cisco/shared/ingest/asa-ftd-pipeline.yml

adriansr · 2019-08-27T23:25:38Z

Thanks @webmat! you've found plenty of important issues.

I think I've addressed all your comments

This adds a new `ftd` fileset to the `cisco` module for parsing Firepower Threat Defense logs. As the FTD logs are a superset of the Cisco ASA logs, this PR introduces a shared ingest pipeline that is used both by the new `ftd` and the existing `asa` filesets. As a side effect of this, it improves the existing ASA fileset so that it uses custom syslog message decoding instead of relying on Filebeat's syslog input, which has caused compatibility problems with some ASA devices. Closes elastic#12690

This is a breaking change so it will be included in master but not in 7.4.

Perform this mappings in the CSV instead of a final processor in the pipeline so that the docs get the right mappings.

It was directly using the value from ConnectionDuration, which is seconds not nanos. Fixed and now populates event.{start,end} too.

Append, don't set, to error.message don't remove _temp_

MikePaquette · 2019-08-28T10:14:34Z

Nice work with this review @webmat and @adriansr

webmat

Ok, I think the two big outstanding issues are outside of the scope of this PR.

I didn't realize the reserved event.outcome was already being populated by multiple modules already. No need to change anything around this, for this PR, this will have to be resolved outside of this PR. discussion
I think the confusion between log.original and event.original can be resolved in a fix after FF, if we decide to make this breaking change. Let's not block this PR because of that. The Cisco module is already using log.original before this PR, in any case. discussion

Last point I wonder if we could tweak is making event.dataset more fine grained (discussion). But since in general this is being set by Filebeat directly, perhaps this is out of scope for this PR. I do think a more fine grained event.dataset would be useful for users, though. WDYT?

If that last one's out of scope, all of my other comments have been addressed. Great work, @adriansr!

andrewkroh · 2019-08-21T14:45:02Z

x-pack/filebeat/module/cisco/asa/config/input.yml

@@ -1,8 +1,8 @@
 {{ if eq .input "syslog" }}



I think that's a good decision for existing users and consistent with other modules.

adriansr · 2019-08-28T18:17:56Z

Oh dear, I forgot to git push before merge.

I will send a new PR with the extra comment to the generator and the section rename in the docs.

webmat · 2019-08-28T19:41:17Z

Yeah I think #13379 covers everything

* Rename docs section to Field Mappings * Add explanatory comment to the generator

adriansr requested a review from a team as a code owner August 19, 2019 15:48

adriansr force-pushed the feature_fb_cisco_ftd branch 4 times, most recently from 0f63699 to 5244bf3 Compare August 19, 2019 17:23

adriansr requested review from a team and removed request for a team August 19, 2019 17:24

adriansr added Filebeat Filebeat review Team:SIEM labels Aug 19, 2019

adriansr added enhancement x-pack Issues and pull requests for X-Pack features. labels Aug 19, 2019

adriansr commented Aug 19, 2019

View reviewed changes

adriansr force-pushed the feature_fb_cisco_ftd branch from e68c7ea to ab27199 Compare August 23, 2019 11:24

adriansr requested a review from a team as a code owner August 23, 2019 11:24

adriansr commented Aug 23, 2019

View reviewed changes

x-pack/filebeat/module/cisco/ftd/test/security-connection.log-expected.json Outdated Show resolved Hide resolved

webmat reviewed Aug 23, 2019

View reviewed changes

x-pack/filebeat/module/cisco/ftd/test/security-connection.log-expected.json Outdated Show resolved Hide resolved

webmat reviewed Aug 23, 2019

View reviewed changes

adriansr force-pushed the feature_fb_cisco_ftd branch 3 times, most recently from 6c25f50 to 19740f2 Compare August 27, 2019 01:01

adriansr requested a review from webmat August 27, 2019 13:26