Trimming PCAPs

Introduction

PCAPs (as a data type) typically take up the most disk space on a Security Onion sensor, and usually aren't able to be kept for extended periods of time. We can lessen the space consumed by PCAPs by trimming them, using a special tool called TrimPCAP from NETRESEC. Using this tool, we can trim the flows within PCAPs to a desired size.

Please be aware and cognizant of the fact that it may take a while to process a large amount of PCAPs. With this in mind, you'll want to consider running TrimPCAP at non-peak times (without high PCAP write volume, etc.).

One retention schedule that could be used is as follows:
(http://www.netresec.com/?page=Blog&month=2017-12&post=Don%27t-Delete-PCAP-Files---Trim-Them)

Age	Size (per flow)
Older than 3 days	1MB
Older than 6 days	102KB
Older than 30 days	10KB

Trimming

We can install TrimPCAP using the following commands:

sudo apt-get install python-pip
sudo pip install dpkt
sudo pip install repoze.lru
sudo wget -O /opt/trimpcap.py https://www.netresec.com/?download=trimpcap

Then we can run TrimPCAP, as follows (specifying a size of 102KB per flow, iterating over all PCAPs of all ages, in all <sensor_interface> directories):

sudo python /opt/trimpcap.py 102400 /nsm/sensor_data/*/dailylogs/*/*

If we want to this for PCAPs older than 3 days, we can do something like the following:

sudo find /nsm/sensor_data/*/dailylogs/*/* -mmin +$((60*72)) -type f -exec /opt/trimpcap.py 102400 {} \;

We can then automate this using a cron job, so our PCAPs are checked daily.

#/etc/cron.d/trimpcap
#
#crontab entry for TrimPCAP

TRIMPCAP="/opt/trimpcap.py"
LOG="/var/log/trimpcap.log"
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# Trim after 3 days
0 1 * * * root echo $(date) >> $LOG; /usr/bin/find /nsm/sensor_data/*/dailylogs/*/* -mmin +$((60*72)) -type f -exec 
$TRIMPCAP 1000000 {} \; >> $LOG 2>&1;

To automatically configure PCAPs to be trimmed at the above recommended intervals, we can do the following:

sudo wget https://raw.githubusercontent.com/weslambert/misc/master/trimpcap_install && sudo chmod +x ./trimpcap_install && sudo ./trimpcap_install