[Filebeat] Update crowdstrike module

[andrewstucki]

What does this PR do?

I've been in the crowdstrike module recently anyway and noticed that there was an open issue reporting some parsing errors. I went ahead and just added some fixes for them.

One thing to note--due to normalizing all timestamps to UNIX_MS this is technically a breaking change. Do we want to be more conservative about the normalization?

Checklist

• My code follows the style guidelines of this project
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Closes Crowdstrike Filebeat Module: Parsing Issues #20035

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [andrewstucki commented: run tests]

• Start Time: 2020-07-23T01:58:19.180+0000

• Duration: 57 min 2 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4191
Skipped	562
Total	4753

[leehinman]

Don't think we need this. In the convert it should be "fail_on_error: false", not "ignore_failure: true", then we don't need the dropIfEmpty

[andrewstucki]

Made the change, but are we ok with passing off an empty string in an ip field? You can take a look at the difference in the approach here: a6d288c

cc: @tonymeehan

[tonymeehan]

First, I heart you. I had started working on this while I was on PTO but haven't had time to get to it since I've been back. I'll provide some comments

[tonymeehan]

These were my changes:

diff --git a/x-pack/filebeat/module/crowdstrike/falcon/config/pipeline.js b/x-pack/filebeat/module/crowdstrike/falcon/config/pipeline.js
index 6ef773761..002bb1d33 100644
--- a/x-pack/filebeat/module/crowdstrike/falcon/config/pipeline.js
+++ b/x-pack/filebeat/module/crowdstrike/falcon/config/pipeline.js
@@ -34,8 +34,7 @@ var crowdstrikeFalcon = (function() {
             { from: "crowdstrike.event.UserIp", to: "source.ip", type: "ip" },
         ],
         mode: "copy",
-        ignore_missing: true,
-        ignore_failure: true
+        fail_on_error: false,
     });
 
     var parseTimestamp = new processor.Timestamp({
@@ -46,6 +45,14 @@ var crowdstrikeFalcon = (function() {
         ignore_missing: false,
     });
 
+    var parseEventTimestamp = new processor.Timestamp({
+        field: "crowdstrike.event.UTCTimestamp",
+        target_field: "crowdstrike.event.UTCTimestamp",
+        timezone: "UTC",
+        layouts: ["UNIX"],
+        ignore_missing: true,
+    });
+
     var processEvent = function(evt) {
         var eventType = evt.Get("crowdstrike.metadata.eventType")
         var outcome = evt.Get("crowdstrike.event.Success")
@@ -172,6 +179,7 @@ var crowdstrikeFalcon = (function() {
     var pipeline = new processor.Chain()
         .Add(decodeJson)
         .Add(parseTimestamp)
+        .Add(parseEventTimestamp)
         .Add(dropFields)
         .Add(convertFields)
         .Add(processEvent)

It's basically two things.

First, use fail_on_error to ignore missing fields.

Second, convert crowdstrike.event.UTCTimestamp so that it's rendered in the row-rendered view in Elastic Security (it's converted for some reason today in the JSON view but not the row-rendered view).

It might also be good to add event.ingested while we are here given #20073

[andrewstucki]

@tonymeehan for the UTCTimestamp field there's a bit of a caveat -- sometimes it's in unix ms format, sometimes it's in unix second format. For example, if we do the parseEventTimestamp version on everything, we fail to parse AuthActivityAuditEvent events because they're in UNIX_MS, not UNIX--it's lame.

Rather than conditionally figuring out which events have unix v. unix ms and potentially missing some, I guess my approach was, "let's just shove all timestamps through this, check to see if they look like UNIX, convert to UNIX_MS and let Elasticsearch translate them into the properly encoded underlying date fields"--thoughts?

Edit:

Here's an example of UTCTimestamp in seconds from the epoch:

beats/x-pack/filebeat/module/crowdstrike/falcon/test/falcon-audit-events.log

Line 127 in 787144f

"UTCTimestamp": 1581546248

Here's an example of UTCTimestamp in milliseconds from the epoch in the same file:

beats/x-pack/filebeat/module/crowdstrike/falcon/test/falcon-audit-events.log

Line 144 in 787144f

"UTCTimestamp": 1581601312140,

[tonymeehan]

@andrewstucki gotcha, thanks for double checking. Like we discussed in Slack, I generally don't like guessing the timestamp format and would prefer to make it conditional based on event type (assuming it's at least consistent at the event level), but perhaps in this case for this data and because we don't have direct access to the product to validate whether the format is consistent for each event type, this is the best option we have to render the time correctly in the product.

[andrewstucki]

So, from the sounds of it, the row renderers better support timestamps that are in string format in their source documents, so I just went ahead and ran all of the timestamps we set through the normalization function and made it also stringify the fields.

I can look into adding an event.ingested timestamp to the ingest pipeline as well.

[leehinman]

Needs a Changelog entry, but code changes LGTM

[vi-or-die]

@tonymeehan & @andrewstucki I have access to the product and am happy to provide anything you may need. Attached is a scrubbed set of sample data hope this helps!

edit by @gtback: removed the file

[andrewstucki]

@vi-or-die thanks a bunch, I'll see about adding those into the test harness and it'll help us identify additional field mappings for SIEM compatibility

[tonymeehan]

❤️

[leehinman]

LGTM