[Filebeat][New Module] Threat intel module for filebeat #21795

P1llus · 2020-10-14T13:10:08Z

What does this PR do?

This PR adds a new module for threat intel, using each TI source as a fileset.

Why is it important?

Adds possibility to ingest Threat Intel data to be used for security usecases

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Resolves #23406

elasticmachine · 2020-10-14T13:10:10Z

Pinging @elastic/siem (Team:SIEM)

elasticmachine · 2020-10-14T14:15:00Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #21795 updated
Start Time: 2021-02-15T11:04:23.117+0000
Duration: 47 min 4 sec
Commit: 500b4ac

Test stats 🧪

Test	Results
Failed	0
Passed	13014
Skipped	2033
Total	15047

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	13014
Skipped	2033
Total	15047

filebeat/docs/fields.asciidoc

x-pack/filebeat/module/threatintel/abuseurl/_meta/fields.yml

x-pack/filebeat/module/threatintel/abuseurl/test/abusechurl.ndjson.log-expected.json

x-pack/filebeat/module/threatintel/abusemalware/test/abusechmalware.ndjson.log-expected.json

x-pack/filebeat/module/threatintel/abuseurl/_meta/fields.yml

P1llus · 2020-10-19T07:52:46Z

@FrankHassanabad and @leehinman Added in your comments, feel free to let me know if its anything else we should change as well.

leehinman · 2020-10-19T14:50:06Z

Looks like larted is still getting stored as a string:

"threatintel.larted": "true",

Might be a good idea to use a convert to store it as a boolean so it doesn't have to be converted when used.

shimonmodi · 2020-10-19T15:46:42Z

@P1llus - I have a threat intel ECS RFC in progress right now, and want to apply those ECS fields to this module. the mapping is available here https://github.com/elastic/security-team/issues/177 . Whats the best way to reconcile the mappings you have used with the one's in the RFC?

P1llus · 2020-10-19T16:18:38Z

@shimonmodi If we want to support the ECS RFC for threat intel whenever we release the module then I don't mind converting it.

I was just unsure if this RFC was complete enough yet to use, if it is then I can convert this module before we proceed :)

P1llus · 2020-10-19T22:03:52Z

Sorry about that @leehinman . I only updated the field mapping and didn't actually convert it. Added the update now. Hopefully everything else is alright? :)

P1llus · 2020-10-19T22:04:37Z

Let's not merge it yet though, but good to know if its "LGTM" or not!

leehinman · 2020-10-20T02:01:23Z

Sorry about that @leehinman . I only updated the field mapping and didn't actually convert it. Added the update now. Hopefully everything else is alright? :)

LGTM

shimonmodi · 2020-10-20T03:51:23Z

@P1llus - perfect, lets chat about it on Wednesday when we meet. My goal is to validate that the proposed ECS fields work for this feed and identify gaps.

P1llus · 2020-10-30T15:05:28Z

I think I will leave this PR as draft for now, that way we can modify it a bit more while testing and discussing with different teams. There might be some breaking changes and temp content in here, so I think for any new content we can hold off new reviews for now.

elasticmachine · 2020-10-30T16:00:29Z

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	4488
Skipped	561
Total	5049

elasticmachine · 2021-02-08T11:07:59Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

P1llus · 2021-02-08T11:08:16Z

The module is now ready for review!

…wing for special characters

… document field

P1llus · 2021-02-08T16:50:57Z

jenkins run tests please

…allbacks

P1llus · 2021-02-08T22:10:17Z

Added some fixes above for general feedback received outside of the PR. All planned changes has been applied.

andrewkroh

I reviewed the module, but not the data mappings. It looks like it's in good shape with the exception of a few really minor comments below. Before merging please create a ticket to track the work necessary to add automated integration tests for the httpjson side of the filesets.

filebeat/docs/modules/threatintel.asciidoc

x-pack/filebeat/module/threatintel/abusemalware/manifest.yml

x-pack/filebeat/module/threatintel/_meta/fields.yml

adriansr

LGTM

x-pack/filebeat/module/threatintel/_meta/docs.asciidoc

Co-authored-by: Adrian Serrano <[email protected]>

* inital MVP commit for threat intel module for filebeat * Fixing typos based on PR comments * adding converter for larted field * adding concept for a new MISP module * setting correct field type for misp group sharing id * setting timestamp to age of attribute timestamp * stashing initial support for AlienVault OTX * add threatintel.otx to timestamp bypass for testing * stashing upcoming changes for new httpjson format * updating settings with pagination * big overwrite of the whole module to fit the new TI ECS fields, add new test data, and make the ingest pipelines more sturdy * update default url for anomali * updating field names based on feedback * final commit fixing certain bugs, adding the missing field mapping etc * updating field defintion, it had a duplicate field * updating the anomali config to access the header in a safer way, allowing for special characters * added stripping of null values and made sure it looked at the correct document field * updating field mapping to default_field false * updating default config descriptions * disable default_field for top group * updating changelog * adding fallback for uri_parts when using older ES version * updating test_modules to ignore timestamps * adding support for uri_parts for all relevant ingest pipelines, and fallbacks * updating default config templates and docs based on PR comments * Update x-pack/filebeat/module/threatintel/_meta/docs.asciidoc Co-authored-by: Adrian Serrano <[email protected]> * mage update Co-authored-by: Adrian Serrano <[email protected]> (cherry picked from commit 70d00b9)

… for filebeat (#24039) * [Filebeat][New Module] Threat intel module for filebeat (#21795) * inital MVP commit for threat intel module for filebeat * Fixing typos based on PR comments * adding converter for larted field * adding concept for a new MISP module * setting correct field type for misp group sharing id * setting timestamp to age of attribute timestamp * stashing initial support for AlienVault OTX * add threatintel.otx to timestamp bypass for testing * stashing upcoming changes for new httpjson format * updating settings with pagination * big overwrite of the whole module to fit the new TI ECS fields, add new test data, and make the ingest pipelines more sturdy * update default url for anomali * updating field names based on feedback * final commit fixing certain bugs, adding the missing field mapping etc * updating field defintion, it had a duplicate field * updating the anomali config to access the header in a safer way, allowing for special characters * added stripping of null values and made sure it looked at the correct document field * updating field mapping to default_field false * updating default config descriptions * disable default_field for top group * updating changelog * adding fallback for uri_parts when using older ES version * updating test_modules to ignore timestamps * adding support for uri_parts for all relevant ingest pipelines, and fallbacks * updating default config templates and docs based on PR comments * Update x-pack/filebeat/module/threatintel/_meta/docs.asciidoc Co-authored-by: Adrian Serrano <[email protected]> * mage update Co-authored-by: Adrian Serrano <[email protected]> (cherry picked from commit 70d00b9) * mage fmt update

inital MVP commit for threat intel module for filebeat

129c1fd

P1llus added in progress Pull request is currently in progress. Filebeat Filebeat Team:SIEM labels Oct 14, 2020

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 14, 2020

P1llus marked this pull request as draft October 14, 2020 13:10

FrankHassanabad reviewed Oct 15, 2020

View reviewed changes

filebeat/docs/fields.asciidoc Outdated Show resolved Hide resolved

FrankHassanabad reviewed Oct 15, 2020

View reviewed changes

x-pack/filebeat/module/threatintel/abuseurl/_meta/fields.yml Outdated Show resolved Hide resolved

FrankHassanabad reviewed Oct 15, 2020

View reviewed changes

x-pack/filebeat/module/threatintel/abuseurl/test/abusechurl.ndjson.log-expected.json Outdated Show resolved Hide resolved

leehinman requested changes Oct 16, 2020

View reviewed changes

x-pack/filebeat/module/threatintel/abusemalware/test/abusechmalware.ndjson.log-expected.json Outdated Show resolved Hide resolved

x-pack/filebeat/module/threatintel/abuseurl/_meta/fields.yml Outdated Show resolved Hide resolved

Fixing typos based on PR comments

9a31392

adding converter for larted field

89f1d46

leehinman approved these changes Oct 20, 2020

View reviewed changes

adding concept for a new MISP module

6ac66cc

P1llus added 3 commits November 2, 2020 13:40

Merge branch 'master' into filebeat_threat_intel

dba0276

setting correct field type for misp group sharing id

6586905

setting timestamp to age of attribute timestamp

c0225a0

final commit fixing certain bugs, adding the missing field mapping etc

636df2c

P1llus marked this pull request as ready for review February 8, 2021 11:07

P1llus added 4 commits February 8, 2021 13:02

updating field defintion, it had a duplicate field

c46f186

updating the anomali config to access the header in a safer way, allo…

a3ee27b

…wing for special characters

added stripping of null values and made sure it looked at the correct…

99c7dc5

… document field

updating field mapping to default_field false

e8357cd

P1llus added 7 commits February 8, 2021 18:06

updating default config descriptions

73b0d15

disable default_field for top group

1546bad

updating changelog

70ca689

Merge branch 'master' into filebeat_threat_intel

140501c

adding fallback for uri_parts when using older ES version

68e10ac

updating test_modules to ignore timestamps

c264183

adding support for uri_parts for all relevant ingest pipelines, and f…

c6be6a2

…allbacks

rylnd mentioned this pull request Feb 12, 2021

[Security Solution][Detections] Adds Indicator path config for indicator match rules elastic/kibana#91260

Merged

7 tasks

andrewkroh reviewed Feb 12, 2021

View reviewed changes

filebeat/docs/modules/threatintel.asciidoc Outdated Show resolved Hide resolved

x-pack/filebeat/module/threatintel/abusemalware/manifest.yml Show resolved Hide resolved

x-pack/filebeat/module/threatintel/_meta/fields.yml Show resolved Hide resolved

updating default config templates and docs based on PR comments

5a18974

adriansr approved these changes Feb 15, 2021

View reviewed changes

x-pack/filebeat/module/threatintel/_meta/docs.asciidoc Outdated Show resolved Hide resolved

P1llus and others added 2 commits February 15, 2021 11:40

Update x-pack/filebeat/module/threatintel/_meta/docs.asciidoc

0be158f

Co-authored-by: Adrian Serrano <[email protected]>

mage update

500b4ac

P1llus added enhancement and removed in progress Pull request is currently in progress. labels Feb 15, 2021

P1llus merged commit 70d00b9 into elastic:master Feb 15, 2021

P1llus mentioned this pull request Feb 15, 2021

Cherry-pick #21795 to 7.x: [Filebeat][New Module] Threat intel module for filebeat #24039

Merged

6 tasks

P1llus added the v7.12.0 label Feb 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat][New Module] Threat intel module for filebeat #21795

[Filebeat][New Module] Threat intel module for filebeat #21795

P1llus commented Oct 14, 2020 •

edited

Loading

elasticmachine commented Oct 14, 2020

elasticmachine commented Oct 14, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

P1llus commented Oct 19, 2020

leehinman commented Oct 19, 2020

shimonmodi commented Oct 19, 2020

P1llus commented Oct 19, 2020

P1llus commented Oct 19, 2020

P1llus commented Oct 19, 2020

leehinman commented Oct 20, 2020

shimonmodi commented Oct 20, 2020

P1llus commented Oct 30, 2020

elasticmachine commented Oct 30, 2020 •

edited by jenkins-beats-ci bot

Loading

Test stats 🧪

elasticmachine commented Feb 8, 2021

P1llus commented Feb 8, 2021

P1llus commented Feb 8, 2021

P1llus commented Feb 8, 2021

andrewkroh left a comment

adriansr left a comment

[Filebeat][New Module] Threat intel module for filebeat #21795

[Filebeat][New Module] Threat intel module for filebeat #21795

Conversation

P1llus commented Oct 14, 2020 • edited Loading

What does this PR do?

Why is it important?

Checklist

elasticmachine commented Oct 14, 2020

elasticmachine commented Oct 14, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

P1llus commented Oct 19, 2020

leehinman commented Oct 19, 2020

shimonmodi commented Oct 19, 2020

P1llus commented Oct 19, 2020

P1llus commented Oct 19, 2020

P1llus commented Oct 19, 2020

leehinman commented Oct 20, 2020

shimonmodi commented Oct 20, 2020

P1llus commented Oct 30, 2020

elasticmachine commented Oct 30, 2020 • edited by jenkins-beats-ci bot Loading

💚 Flaky test report

Test stats 🧪

elasticmachine commented Feb 8, 2021

P1llus commented Feb 8, 2021

P1llus commented Feb 8, 2021

P1llus commented Feb 8, 2021

andrewkroh left a comment

Choose a reason for hiding this comment

adriansr left a comment

Choose a reason for hiding this comment

P1llus commented Oct 14, 2020 •

edited

Loading

elasticmachine commented Oct 14, 2020 •

edited by jenkins-beats-ci bot

Loading

elasticmachine commented Oct 30, 2020 •

edited by jenkins-beats-ci bot

Loading