-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathsave_json_stream.txt
88 lines (65 loc) · 3.56 KB
/
save_json_stream.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
Quickstart:
Imports Bricata or Corelight sensor json lines, such as ones
exported over a network, to standard Zeek json log format.
To listen to incoming json records arriving at a port:
python3 ./save_json_stream.py -p 4567 -o $HOME/testimport
For more detail about the import:
python3 ./save_json_stream.py -p 4567 -o $HOME/testimport --debug
Now send records from a Bricata or Corelight sensor, or simulate
this by sending lines from a saved log:
cat samples/raw_export_sample | ncat 127.0.0.1 4567
or
ncat 127.0.0.1 4567 <samples/raw_export_sample
Note, standard netcat and nc will not do - they don't close the
connection when the input lines are finished, so use ncat (part of the
nmap package).
To import directly from a local file, try:
cat samples/raw_export_sample | python3 ./save_json_stream.py -o $HOME/testimport
or
python3 ./save_json_stream.py -o $HOME/testimport -f samples/raw_export_sample
Underneath the top level directory you request on the command
line, the records will be saved to
YY-MM-DD/file_type.HH:00:00-HH+1:00:00.log
If you also add the "--by_sensor" command line option, that
directory tree will also include the unique name (UUID) for that sensor,
like:
sensor_uuid/YY-MM-DD/file_type.HH:00:00-HH+1:00:00.log
To have AC-Hunter import these logs automatically, run
save_json_stream.py on the same system as AC-Hunter. It will need to run
as a user with privileges to write to /opt/zeek/remotelogs/ .
python3 ./save_json_stream.py -o /opt/zeek/remotelogs/ --limit_writes --by_sensor
The --limit_writes option only writes the 6 zeek logs that
AC-Hunter needs, saving ~10% disk space.
The databases that AC-Hunter shows will be named
sensor_uuid-rolling .
A single copy of this program can 1) listen on a port, 2) accept
lines on stdin, or 3) read lines from a file. If you want to do more
than one of these (or listen to more than one port) you'll need to start
multiple copies, one for each task, each writing to their own output
directory.
A test run with version 0.4.5 found it will process ~4300
records/second from a network socket on a laptop. Since each copy can
only use a single processor core, the upper bound on how many records can
be processed in a second is approximately 4300 * number of cores if
running that many copies (though this will also be limited by the speed
of the network connection and the available disk bandwidth).
To use a TLS connection, generate a test key once with:
echo 'testphrase' >passphrase.txt
openssl req -x509 -newkey rsa:2048 -keyout testkey.pem -out testcert.pem -days 365 -passin pass:`head -1 passphrase.txt` -passout pass:`head -1 passphrase.txt`
(Use a different passphrase than testphrase, obviously.)
When you run the server, include the key and certificate
filenames on the command line:
python3 ./save_json_stream.py -p 9999 -c testcert.pem -k testkey.pem -o $HOME/testimport
If the certificate is protected by a passphrase, please use either
--passphrase_file file_name
, which reads the passphrase from the first line of file_name, or
--passphrase_ask
, which asks the user for the passphrase at startup.
The corresponding test tool for sending lines is:
cat samples/raw_export_sample | openssl s_client -quiet -servername localhost -connect localhost:9999
To limit the IPs that are allowed to connect, use the
"--sensorips" option. Place one or more lowercase ipv4 and/or ipv6 ips
after this option. Any IPs other than these will not be allowed to
connect. Remember to include "::1" and "127.0.0.1" if you want localhost
to be able to connect as well. If you do not use this option all IPs can
connect.