Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Migrate enrich.py to modular system #117

Merged
merged 53 commits into from
Apr 8, 2021
Merged

Migrate enrich.py to modular system #117

merged 53 commits into from
Apr 8, 2021

Conversation

fastlorenzo
Copy link
Collaborator

@fastlorenzo fastlorenzo commented Nov 17, 2020 •
edited
Loading

  • Enrich CS beacon data
  • Enrich redirector traffic with greynoise
  • Enrich redirector traffic with tor exit nodes
  • Enrich redirector traffic with IP list (iplist_unknown.conf)
  • Enrich redirector traffic with IP list (iplist_redteam.conf)
  • Enrich redirector traffic with IP list (iplist_customer.conf)
  • Enrich redirector traffic with IP list (iplist_alarmed.conf)
  • Enrichment to replace roguedomains.conf => To be done in another PR
  • Test enrich CS beacon data
  • Test enrich redirector traffic with greynoise
  • Test enrich redirector traffic with tor exit nodes
  • Test enrich redirector traffic with IP list (iplist_unknown.conf)
  • Test enrich redirector traffic with IP list (iplist_redteam.conf)
  • Test enrich redirector traffic with IP list (iplist_customer.conf)
  • Test enrich redirector traffic with IP list (iplist_alarmed.conf)
  • redteamdomains.conf => still to decide how to use it => To be done in another PR
  • Remove enrichment for known_testsystems.conf and known_sandboxes.conf
  • Add possibility to enable/disable enrichment modules (via config file)
  • Add possibility to configure run interval for enrichment and alarm modules (via config file)
  • Fix small errors in installer for dev run (don't run certbot in dev)

Fixes #108

@github-actions github-actions bot added docker Related to docker container builds elkserver Related to RedELK server components installer Related to RedELK installers labels Nov 17, 2020
@fastlorenzo
Copy link
Collaborator Author

@MarcOverIP @xychix could you already have a look at the proposed approach?

Regarding the iplist, I would get rid of the iplist config files from /etc/redek and move that to a specific index (iplist).
The different IPs can be then modified via python scripts (alarms, enrichment, Jupyter) or via the Kibana plugin (interface to be created)

@fastlorenzo
Copy link
Collaborator Author

@MarcOverIP @xychix could you already have a look at the proposed approach?

Regarding the iplist, I would get rid of the iplist config files from /etc/redek and move that to a specific index (iplist).
The different IPs can be then modified via python scripts (alarms, enrichment, Jupyter) or via the Kibana plugin (interface to be created)

As discussed, we'll keep the following iplist config files in sync with the related ES index:

  • rogue_useragents.conf
  • iplist_redteam.conf
  • redteamdomains.conf
  • iplist_customer.conf
  • iplist_alarmed.conf

@github-actions github-actions bot added the helpers Related to RedELK helper scripts label Mar 8, 2021
@fastlorenzo fastlorenzo mentioned this pull request Mar 8, 2021
Closed
fastlorenzo and others added 18 commits March 30, 2021 21:55
@fastlorenzo
Added CS Beacon enrich module
4cc2d35
@fastlorenzo
Added enrich greynoise module
8310441
@fastlorenzo
Deleted old reference to greynoise
61c3832
@fastlorenzo
Fixed installer for dev (cert)
78c0815
@fastlorenzo
Added option to disable enrich modules in config
f52e218
@fastlorenzo
Added interval options for modules
96b2ff2
@fastlorenzo
Added check if alarm fails
e275109
@fastlorenzo
Added enrich_tor module
27bc8e3
@fastlorenzo
Removed abuse.ch ssl and botnet lists (to be created in new alarms la…
e4a1b6d 
…ter)
@fastlorenzo
added iplist in ES + small fixes
d969ffc
@fastlorenzo
small fix
4ed43fe
@fastlorenzo
Added iplist config
4c1a9c8
@fastlorenzo
Updated helper script
2ecfe65
@fastlorenzo
Added missing templates
c9e27b3
@fastlorenzo
Cleaned code
e41400a
@fastlorenzo
Fetch LOGLEVEL from config.json for alarms.py
2030561 
Fetch LOGLEVEL from config.json for alarms.py
@fastlorenzo
Added blueteam IP list
51768e0 
Signed-off-by: fastlorenzo <[email protected]>
@fastlorenzo
Minor updates to alarms
a31c5d6 
@lorenzo please do check if alarms.py matches with ones in your open PR. Further these are minor additions
fastlorenzo and others added 17 commits March 30, 2021 23:01
@fastlorenzo
Added iplist config
14a3694
@fastlorenzo
Updated helper script
35bf3d4
@fastlorenzo
Added missing templates
b454847
@fastlorenzo
Cleaned code
5ffacf6
@fastlorenzo
Fetch LOGLEVEL from config.json for alarms.py
b557b67 
Fetch LOGLEVEL from config.json for alarms.py
@fastlorenzo
Added blueteam IP list
0d07a47 
Signed-off-by: fastlorenzo <[email protected]>
@fastlorenzo
Minor updates to alarms
355c049 
@lorenzo please do check if alarms.py matches with ones in your open PR. Further these are minor additions
@fastlorenzo
removed logs + fixed line endings
e82f525
@fastlorenzo
Updated enrichment and alarm scripts
d37ab39
@fastlorenzo
fixed merge
bde4020 
Signed-off-by: fastlorenzo <[email protected]>
@fastlorenzo
Cleaned python scripts
d66d937 
Signed-off-by: fastlorenzo <[email protected]>
@fastlorenzo
Split the alarm file into different functions to reduce complexity
06ac482 
Signed-off-by: fastlorenzo <[email protected]>
@fastlorenzo
fixed iplist pattern + sample data ingestor
fad5a01 
Signed-off-by: fastlorenzo <[email protected]>
@fastlorenzo
renamed alarm script to daemon + minor py fixes
e4183f9 
Signed-off-by: fastlorenzo <[email protected]>
@fastlorenzo
fixed lint
939a65c 
Signed-off-by: fastlorenzo <[email protected]>
@fastlorenzo
renamed iplist
14a0997 
Signed-off-by: fastlorenzo <[email protected]>
@fastlorenzo
Added RedELK internal health dashboard (IP Lists + modules run results)
e008078 
Signed-off-by: fastlorenzo <[email protected]>
@fastlorenzo fastlorenzo marked this pull request as ready for review March 31, 2021 22:11
@github-actions github-actions bot added the documentation Related to RedELK documentation label Mar 31, 2021
@fastlorenzo
Copy link
Collaborator Author

@MarcOverIP @xychix should be ready to be merged in master \o/

@fastlorenzo
Fixed typo
666530d 
Signed-off-by: fastlorenzo <[email protected]>
@fastlorenzo
Fix in case iplist file does not exists
c46f4e1 
Signed-off-by: fastlorenzo <[email protected]>
@fastlorenzo
Fixed notifications
57d9964 
Signed-off-by: fastlorenzo <[email protected]>
This was referenced Apr 5, 2021
Closed
Closed
@MarcOverIP MarcOverIP merged commit 9b3899d into outflanknl:master Apr 8, 2021
@fastlorenzo fastlorenzo deleted the enrich-modules branch April 8, 2021 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
docker Related to docker container builds documentation Related to RedELK documentation elkserver Related to RedELK server components helpers Related to RedELK helper scripts installer Related to RedELK installers
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Migrate enrich script to new modular system
2 participants
@fastlorenzo @MarcOverIP