GreyNoise Fluent Bit Lua Filter

This is a prototype Fluent Bit container using a filter plugin which calls the GreyNoise API to drop, re-route, or enrich records. This specific filter leverages the Fluent Bit Lua script filter.

Getting Started

These instructions will cover usage information and for the docker container

Prerequisities

In order to run this container you'll need docker installed.

• Windows
• OS X
• Linux

In order to run make stats you will need jq installed

• JQ

GreyNoise Sign-Up

1. Sign-Up for GreyNoise
2. Copy .env_example to .env
3. Copy your GreyNoise API key from the Account section in the top right corner
4. Replace the <REPLACE_ME> in .env with your API key

Docker

Usage

Container Parameters

docker run --env-file .env -it -p 2020:2020 -v $(PWD):/app greynoise/greynoise-fluentbit-lua:latest -c /app/conf/myconfig.conf

Environment Variables (required)

• GREYNOISE_API_KEY - GreyNoise API key to use for HTTP requests.
• GREYNOISE_IP_FIELD - Named field from the Fluent Bit parser to use for IP lookups.
• GREYNOISE_LUA_LOG_LEVEL - Lua logging level (info/error/warning/debug)
• GREYNOISE_LUA_CACHE_SIZE - The number of IP records to cache in-memory before overwriting.

Volumes

• /app - Core working directory (mounted from the base repo folder)

Sample Data Testing

The sample data tests are meant to be run from the repo base folder.

Example 1 - Dummy Data

conf/dummy.conf

This example just generates the same JSON line over and over.

1. Copy .env_example to .env
2. Copy your GreyNoise API key from the Account section in the top right corner
3. Replace the <REPLACE_ME> in .env with your API key
4. Run make build
5. Run make run

Example 2 - Auth.log

conf/tail.conf

This example watches reads a log file in and watches for new lines.

1. Run make run-tail
2. Run make stats in another terminal to see metrics

Example 3 - Auth.log With RewriteTag Rules

conf/rewrite.conf

This is the same as #2 except this leverages rewrite_tag filter to drop records. This config drops invalid IPv4 records, bogon address space, GreyNoise RIOT records, and GreyNoise Noise records.

1. Run make run-rewrite
2. Run make stats in another terminal to see metrics (note the drop rates)

Running in your environment

1. Create a directory for your configs mkdir conf
2. Add your Fluent Bit configs and parsers to conf/
3. Create a directory for your outputs mkdir output
4. You should now have a directory tree that looks something like the following:

   conf/
       parser.conf
       myconfig.conf
   output/
   .env

5. Run the docker command

docker run --env-file .env -it -p 2020:2020 -v $(PWD):/app greynoise/greynoise-fluentbit-lua:latest -c /app/conf/myconfig.conf

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

Authors

• Matt Lehman - Initial work - Obsecurus

See also the list of contributors who participated in this project.

License

This project is licensed under the MIT License - see the LICENSE.md file for details.

Acknowledgments

• Eduardo Silva (https://github.com/edsiper) - guidance on Fluent Bit Lua optimizations
• leite (https://github.com/leite) - greynoise/src/iputil.lua module
• rxi (https://github.com/rxi) - greynoise/src/log.lua module

Links

• GreyNoise.io
• GreyNoise Terms
• GreyNoise Developer Portal

Contact Us

Have any questions or comments about GreyNoise? Contact us at [email protected]