New input for Crowdstrike Falcon events

[tonymeehan]

What does this PR do?

This adds a new input for Crowdstrike Falcon events forwarded by Crowdstrike's SIEM forwarder. This input uses the default JSON output format from the SIEM forwarder.

Why is it important?

We've had several users asks us for adding support for Crowdstrike Falcon in ECS. We strive to be data agnostic to help enable our SOC users aggregate all of their data in our SIEM. Using some log data provided by our users, I've put this new module together to ingest the data and convert many fields into ECS (including categorization).

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works

How to test this PR locally

mage update build integTest

Screenshots

Proof this works

[success] 0.85% test_xpack_modules.XPackTest.test_fileset_file_008_crowdstrike: 2.7708s
----------------------------------------------------------------------
Ran 135 tests in 326.929s

OK

I also ran make docs to validate the documentation looked good.

Other comments

I'll squash before merging.

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[tonymeehan]

Looking into CI errors. Need to run make check and mage fmt update

[leehinman]

Looking good :-)

couple general questions/suggestions:

• any value in geoip on ip addrs?
• if event has ips, users or hashes probably a good idea to add those to the related fields

specific questions in-line.

[webmat]

This is a great idea!

This is just a preliminary review, here's a bunch of comments and suggestions :-)

Broader discussion point: as I note in one of my comments, this seems to be a log that mixes their service's audit logs with actual endpoint detection events. Both are worth capturing, of course. But I wonder if we should find a way to separate them out in two different event.dataset, as their meaning & semantics are pretty different.

Doing so would help clarify different handling of various things like host.name, how to fill event.action, etc; that have been raised by me and @leehinman.

[webmat]

Indeed. @leehinman, this is how to make it into an array, correct?

Suggested change

	evt.Put("event.type", "info")
	evt.Append("event.type", "info")

[tsg]

I wonder if it's worth also supporting the syslog input here? https://www.crowdstrike.com/blog/tech-center/integrate-with-your-siem/ seems to suggest that it's possible for the Connector to send date directly to a syslog server?

[tonymeehan]

It would be easy to add the config option for syslog and maybe not make it the default choice. But my concern is we can't test it to validate that the output is identical to the file output. I'm on the fence but can be convinced to add it if it's not the default choice.

[tsg]

I see, makes sense. I think we can leave it out and add it if it's being requested. Then we have a way to validate it.

[andrewkroh]

It looks like CI is much more happy now. The failures are from flakey tests and appear unrelated to your changes.

[tonymeehan]

It looks like CI is much more happy now. The failures are from flakey tests and appear unrelated to your changes.

Yep, thanks @andrewkroh. I was just rebuilding to see if that was the case. I'll start working on the recommendations tonight.

[tonymeehan]

Thanks for the feedback.

PR Feedback Addressed

• Split up the filesets into two components, falcon_endpoint and falcon_audit.
• Added additional ECS 1.5 fields
• Fixed some bugs (e.g. event.type and event.category are array fields)

PR Feedback Not Addressed

• Geo IP -- two reasons -- most endpoint IPs will be private, geo data won't make sense here. And the the other reason was I wanted to avoid installing an ingest pipeline for our users that have Logstash setup between their Beats and the stack.
• Setting enabled to false in config.yml (rationale explained above)

[tonymeehan]

jenkins retest this please

[tonymeehan]

I've been able to address most of the PR feedback (thanks again, everyone). I think we're ready for some LGTMs unless anyone has any additional feedback/changes.

[andrewkroh]

This looks pretty good! Two small requests:

• The docs need changed to reflect there's only one fileset.
• Please add an entry to the CHANGELOG.next.asciidoc file in the root of the repo under the Filebeat/Added section.

[andrewkroh]

Suggested change

This module segments events forwarded by the Falcon SIEM connector into two datasets for endpoint data and Falcon platform audit data.

[tonymeehan]

I was going to keep this since we're still doing this (two datasets), just from the same fileset. That okay?

[andrewkroh]

Oh, that's interesting. I didn't notice that the module was setting the event.dataset values. We've never had a case AFAIK where a fileset produced an event.dataset value other than the default of {module}.{fileset}. I don't think this will cause any issues. I think it would be good to callout the event.dataset values that it produces in this paragraph.

[leehinman]

Changes look really good.

Looks like you need to run "mage fmt update" and push changes though.

[tonymeehan]

Thanks for all the help @leehinman @andrewkroh @webmat and @tsg!