New Feature - Accumulator (Multiline logs with consistent IDs) #198
Conversation
Accumulator allows the collection of data across multiple log entries by
decoding and grouping on an id field. The cache for the accumulator
lasts:
Default expire to 2 minutes (on lookup of exact key)
Default purge count to 200 lookups (on lookup of any key)
Default purge interval to 5 minutes (on lookup of any key)
== hash_op.(c|h)
* Change to design of the API, OSHash owns the key's memory, caller owns
the data elements memory.
* OSHash_Add() strdup's the key
* OSHash_Free() now free()'s the key element
* OSHash_Delete() added so hashes can be dynamic
== accumulator.(c|h)
* This allows log lines to accumulate details based on the previous log
entries which share the same id element. Lines must contain, therefor
the decoder must extract, the id element, or the accumulator simply
returns without performing any action. It is important to note, this
is not a multi-line reader, each log line is treated as it's own
event. As more data is learned about each ID, the events are able
to use that data.
* Accumulate_Init() sets up the accumulator constants and initilizations
variables
* Accumulate() performs the accumulation of event data. The default
expiry for an event ID is 5 minutes of inactivity.
* Accumulate_CleanUp() is called once every 100 lookups or once every 10
minutes, whichever happens first.
== etc/decoder.xml
* Enabled accumulator plugin for the OpenLDAP Decoder. This allows for
rules to be created with <same_source_ip/>, like with SSHD multiple
login failures.
Bug fixes included:
* Accumulator working with in-memory structs
* Testrule.c now is very flagrant about the accumulator usage
* Fixed bug in Accumulator_CleanUp() which caused crashes
syscheck. The Eventinfo struct only included file data for builds with
Prelude integration. This prevented the AR from handing filename off
anyways.
* Eventinfo now contains file data always
* Added *expect* option for 'filename'
* Added FILENAME to ar.h
* Filename validation is tricky, so shell escape instead
* Added os_shell_escape() to string_op.h, allocates memory for an
escaped string and passes the pointer back to caller. Caller must
cleanup that memory.
* Call os_shell_escape() before passing to execd.
* Added string_test.c to test the os_shell_escape() function.
This quick fix will add setenv function using the avaiable putenv call. This should be in a win32 shared section of the code, but I will take care of that as I clean more of the win32 build process up.
This should allow the user to specify a debug level for the remoted daemon using the remoted.debug option in the internal_options.conf. The debug level specified on the command line takes precedence.
…aming conflict to struct config in zlib
… global remoted struct logr
Conflicts: .gitignore src/Makeall src/win32/make.sh
Accumulator allows the collection of data across multiple log entries by
decoding and grouping on an id field. The cache for the accumulator
lasts:
Default expire to 2 minutes (on lookup of exact key)
Default purge count to 200 lookups (on lookup of any key)
Default purge interval to 5 minutes (on lookup of any key)
== hash_op.(c|h)
* Change to design of the API, OSHash owns the key's memory, caller owns
the data elements memory.
* OSHash_Add() strdup's the key
* OSHash_Free() now free()'s the key element
* OSHash_Delete() added so hashes can be dynamic
== accumulator.(c|h)
* This allows log lines to accumulate details based on the previous log
entries which share the same id element. Lines must contain, therefor
the decoder must extract, the id element, or the accumulator simply
returns without performing any action. It is important to note, this
is not a multi-line reader, each log line is treated as it's own
event. As more data is learned about each ID, the events are able
to use that data.
* Accumulate_Init() sets up the accumulator constants and initilizations
variables
* Accumulate() performs the accumulation of event data. The default
expiry for an event ID is 5 minutes of inactivity.
* Accumulate_CleanUp() is called once every 100 lookups or once every 10
minutes, whichever happens first.
== etc/decoder.xml
* Enabled accumulator plugin for the OpenLDAP Decoder. This allows for
rules to be created with <same_source_ip/>, like with SSHD multiple
login failures.
Bug fixes included:
* Accumulator working with in-memory structs
* Testrule.c now is very flagrant about the accumulator usage
* Fixed bug in Accumulator_CleanUp() which caused crashes
…nto feature/accumulator Conflicts: etc/decoder.xml src/analysisd/Makefile
|
This is a huge patch and some I want to pull in and start get others test and playing with. I was wondering if you could get a document ion pull request started? Also how do we test this I was wondering if you integrated it into logtest. I don't see it in my first go around of the code but I could easy have missed that. Thank you again this is something ossec has needed for a while now and will solve a lot of problems for people. |
|
Is this something that can wait until after 2.8? On Fri, May 2, 2014 at 8:59 AM, Jeremy Rossi [email protected]:
|
|
We would not pull this into the 2.8 code base. This would pull into master for the next release 2.9
|
|
Yes, but can that all wait until after 2.8 is released? Spending time on On Fri, May 2, 2014 at 1:46 PM, Jeremy Rossi [email protected]:
|
|
I'll start on the docs.. and yes, it's incorporated into logtest, because I would have lost my mind without an easy way to test. It also fixes issues with some of the core library's hash functions and includes tests for those changes as well. 2.9 is fine, meant to get this out the door sooner for 2.8, but failed to do so. |
|
Been running this for a while.....have not written and real rules for it yet, but thing it make sense to get into master soon. Anyone else running this and testing it yet? @ossec |
|
@reyjrar how are you coming with the ossec/ossec-docs pull request? I will pull this in once we docs ready to go. |
|
I can't upload them now, but I have added the documentation for this. I'll try to make a pull request tonight. |
New Feature - Accumulator (Multiline logs with consistent IDs)
From Wazuh pull request ossec#198 by @jesusfidalgo
As detailed in this blog post OSSEC-HIDS Accumulator
Running in production for >12 months, maybe closer to two years.