Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slips v1.0.7 #397

Merged
merged 135 commits into from
Sep 15, 2023
Merged
Changes from 1 commit
Commits
Show all changes
135 commits
Select commit Hold shift + click to select a range
8b6dd26
basic cprofile
danieltherealyang Jun 4, 2023
2ca83f6
viztracer added
danieltherealyang Jun 4, 2023
9a2ae68
Changed docker requirements.txt
danieltherealyang Jun 6, 2023
a76aba3
added profiler interface
danieltherealyang Jun 8, 2023
8add78a
added OOP inheritance and factory method
danieltherealyang Jun 8, 2023
ef9b29c
fix develop merge conflict
danieltherealyang Jun 15, 2023
4e49dc7
fixed sh<defunct> from redis_manager's os.popen
danieltherealyang Jun 17, 2023
35cf34b
added redis channel for live mode. gave up on multiprocessing takes t…
danieltherealyang Jun 18, 2023
9c683df
added cpu profiling to the changelog
danieltherealyang Jun 18, 2023
e8dd06d
slips.conf: more clear description of CPU profiler modes
AlyaGomaa Jun 19, 2023
e5acdb5
cpu_profiler.py: fix the path of result.json
AlyaGomaa Jun 19, 2023
68711c6
viztracer test: only main process enabled
danieltherealyang Jun 19, 2023
91258d8
update: issue is definitely inputProcess
danieltherealyang Jun 19, 2023
ebbdc6e
implemented multiprocess profiling option for dev mode
danieltherealyang Jun 21, 2023
94ae688
fixed inputProcess zeek subprocess detach child, fixed output file pa…
danieltherealyang Jun 21, 2023
6410b44
something in modules is making viztracer fail, just disable modules w…
danieltherealyang Jun 25, 2023
e32d9c1
reverted zeek thread and enabled load_modules
danieltherealyang Jun 28, 2023
db4afb7
changed zeek_run() to zeek_thread.start()
danieltherealyang Jun 28, 2023
6b89614
Merge branch 'develop' into cpu-profile
danieltherealyang Jul 3, 2023
b1c7b41
Merge branch 'develop' into cpu-profile
danieltherealyang Jul 3, 2023
1f7023f
single process profiling works
danieltherealyang Jul 3, 2023
328d772
viz is working now, limited stack depth to 10 otherwise too much ram …
danieltherealyang Jul 3, 2023
051b54f
disabled update manager in slips.conf
danieltherealyang Jul 4, 2023
dd3a5fc
corrected dev mode multiprocess option selection
danieltherealyang Jul 4, 2023
ee8d3c8
added cpu profiling documentation to features.md
danieltherealyang Jul 5, 2023
abeb2b8
update the profiling docs
AlyaGomaa Jul 7, 2023
2ad16d6
update the dev mode profiling docs
AlyaGomaa Jul 7, 2023
e25243f
don't use sys.exit(0) if the profiler is enabled
AlyaGomaa Jul 7, 2023
1bbe106
slips.conf: disable the cpu profiler by default
AlyaGomaa Jul 7, 2023
8389d77
whitelist: check the queries of each dns flow for whitelisted domains
AlyaGomaa Jul 7, 2023
a061ca1
whitelist: check the answers of each dns flow for whitelisted ips
AlyaGomaa Jul 7, 2023
fa5a36b
fixed clear redis cache test error
danieltherealyang Jul 7, 2023
b2658a6
Merge pull request #381 from stratosphereips/alya-fix-whitelist2
AlyaGomaa Jul 8, 2023
f9e256e
utils: use communityid python library to calculate the cid of each co…
AlyaGomaa Jul 10, 2023
aed83be
zeek.py: add community id field to each Conn flow
AlyaGomaa Jul 10, 2023
fc233b9
db: store the community_id in the flows table in case of zeek flows
AlyaGomaa Jul 10, 2023
73980ad
add communityid to requirements.txt
AlyaGomaa Jul 10, 2023
e03e24a
removing updatemanager from disabled modules. Tested and works with c…
danieltherealyang Jul 10, 2023
884894b
added cpu_profiler_dev_mode_entries to slips.conf
danieltherealyang Jul 10, 2023
010ad07
changed conf file dev mode entries comment
danieltherealyang Jul 11, 2023
1b5d4a8
Update README.md
verovaleros Jul 11, 2023
3a01226
Merge pull request #382 from stratosphereips/alya-add-community-id
AlyaGomaa Jul 12, 2023
13e2a55
Merge remote-tracking branch 'origin/develop' into cpu-profile
AlyaGomaa Jul 12, 2023
2dbe836
add "profiling" section to all config files used in integration tests
AlyaGomaa Jul 12, 2023
14b54f4
Merge pull request #362 from danieltherealyang/cpu-profile
AlyaGomaa Jul 12, 2023
5408718
Squashed commit of the following:
danieltherealyang Jul 13, 2023
77451b2
done with dev mode
danieltherealyang Jul 13, 2023
38eb0e2
singleprocess done, need to add autospawned TUI so doesn't block process
danieltherealyang Jul 20, 2023
352a2e9
prototype multiprocess profiling structure (untested)
danieltherealyang Jul 24, 2023
2f0ae56
changed Dict Type
danieltherealyang Jul 24, 2023
1ab81ac
stdout to pipe file fixed
danieltherealyang Jul 25, 2023
36263ac
non blocking single process
danieltherealyang Jul 20, 2023
701e422
signal handling and global pid->process mapping
danieltherealyang Jul 25, 2023
1b8b6e2
proc mapping finished
danieltherealyang Jul 26, 2023
f968977
moved mutex locking to start and end tracker
danieltherealyang Jul 26, 2023
3a0de45
communication works, need to find out why tracer isn't starting on si…
danieltherealyang Jul 26, 2023
e37be2c
memory profiler done
danieltherealyang Jul 27, 2023
1e8e976
Added notes to memory_profiler_example
danieltherealyang Jul 27, 2023
7d2fe86
testing print statements
danieltherealyang Aug 2, 2023
eda609c
removed custom join and terminate
danieltherealyang Aug 2, 2023
0429dad
wrote documentation
danieltherealyang Aug 2, 2023
80adab2
Added notes on multiprocess live profiling
danieltherealyang Aug 12, 2023
401945f
memory profiler works correctly with multiprocessing but modules gett…
danieltherealyang Aug 13, 2023
653c398
Summary: fixed inheritance issues preventing modules from getting pro…
danieltherealyang Aug 17, 2023
4cd6468
commented out multiproc_test in slips.py, text from multiprocessPatch…
danieltherealyang Aug 18, 2023
e76bd3d
made changes
danieltherealyang Aug 18, 2023
b29cfbd
removed print
danieltherealyang Aug 18, 2023
15fa16c
made small change
danieltherealyang Aug 22, 2023
e5c3c6d
add logic to calc aid
AlyaGomaa Aug 25, 2023
d696fd6
get aid instead of cid in sqlite db
AlyaGomaa Aug 25, 2023
eb944f5
Merge pull request #384 from stratosphereips/verovaleros-patch-README
AlyaGomaa Aug 28, 2023
e9664aa
Merge pull request #390 from stratosphereips/alya/calc-aid
AlyaGomaa Aug 28, 2023
05e7754
use transactions and try max 2 times to re-execute failed queries bef…
AlyaGomaa Aug 29, 2023
857736e
Merge pull request #391 from stratosphereips/alya-fix-malformed-sqlite
AlyaGomaa Aug 29, 2023
31167a5
evidence: pass all alert details to sqlite
AlyaGomaa Aug 30, 2023
2db32a7
sqlite: store all alerts details in the alerts table
AlyaGomaa Aug 30, 2023
9eb6bbe
sqlite: store twid start and end date in the alerts table
AlyaGomaa Aug 30, 2023
c5fc002
sqlite: init the trial var to keep track of failed sql queries [skip ci]
AlyaGomaa Aug 30, 2023
26ec1eb
sqlite: print sqlite db errors to errors.log
AlyaGomaa Aug 30, 2023
d74f07a
sqlite: store alert time in unix format in alerts table
AlyaGomaa Aug 30, 2023
7dd87ef
Merge pull request #392 from stratosphereips/alya-add-sqlite-alerts-t…
AlyaGomaa Aug 30, 2023
f36318c
fix unable to get daddr from conn.log flow in flowalerts (quick-fix) …
AlyaGomaa Aug 31, 2023
b3b68c4
sqlite: make it possible to have multiple instances of the sqlite cl…
AlyaGomaa Sep 4, 2023
8038295
start a new instance of sqlite for each module in slips
AlyaGomaa Sep 4, 2023
f207f65
start a new instance of sqlite for each Core file in slips
AlyaGomaa Sep 4, 2023
1c9045b
sqlite: create the db if it doesn't exist, or connect to it if it does
AlyaGomaa Sep 4, 2023
16986a1
make sure the ts used in calculating aid has microseconds
AlyaGomaa Sep 5, 2023
3f5526f
Merge pull request #394 from stratosphereips/alya/assert_microseconds…
AlyaGomaa Sep 5, 2023
cdfb448
change how we instantiate dbmanager to be able to mock the sqlite db
AlyaGomaa Sep 6, 2023
99fdeb3
change the creation of all modules in the unit tests to mock sqlite a…
AlyaGomaa Sep 6, 2023
10d634b
use mock_rdb instead of generic moc_db in all unit tests
AlyaGomaa Sep 6, 2023
0f8e88d
don't connect manually to redis in the test_add_flow_to_profile test
AlyaGomaa Sep 6, 2023
b44b585
Merge pull request #393 from stratosphereips/alya/create_many_sqlite_…
AlyaGomaa Sep 6, 2023
e34a186
don't treat dbmanager as a singelton
AlyaGomaa Sep 7, 2023
af07694
close the db in the destructor of each class and core file
AlyaGomaa Sep 7, 2023
791638d
make sure the aid stored in the flows.sqlite db is string not bytes
AlyaGomaa Sep 8, 2023
ce9bf55
Merge pull request #395 from stratosphereips/alya/fix_sharing_sqlite_…
AlyaGomaa Sep 11, 2023
57f1386
use a lock even when creating tables
AlyaGomaa Sep 12, 2023
6a42ee3
wait 5 seconds before next trial when db is locked
AlyaGomaa Sep 12, 2023
2d5a138
commit even on SELECT
AlyaGomaa Sep 12, 2023
d1a9132
db: wait for the db to beunlocked 20s instead of the default 5s
AlyaGomaa Sep 12, 2023
d963514
Merge pull request #396 from stratosphereips/alya/use_lock_even_when_…
AlyaGomaa Sep 13, 2023
4da65a9
Squashed commit of the following:
danieltherealyang Jul 13, 2023
230ccd9
done with dev mode
danieltherealyang Jul 13, 2023
d4a485c
singleprocess done, need to add autospawned TUI so doesn't block process
danieltherealyang Jul 20, 2023
4727834
prototype multiprocess profiling structure (untested)
danieltherealyang Jul 24, 2023
eb59aa6
changed Dict Type
danieltherealyang Jul 24, 2023
3eba22f
stdout to pipe file fixed
danieltherealyang Jul 25, 2023
fb96463
non blocking single process
danieltherealyang Jul 20, 2023
af5d04d
signal handling and global pid->process mapping
danieltherealyang Jul 25, 2023
8b47542
proc mapping finished
danieltherealyang Jul 26, 2023
b3ade23
moved mutex locking to start and end tracker
danieltherealyang Jul 26, 2023
dd4cd25
communication works, need to find out why tracer isn't starting on si…
danieltherealyang Jul 26, 2023
8e423cd
memory profiler done
danieltherealyang Jul 27, 2023
6dcb63e
Added notes to memory_profiler_example
danieltherealyang Jul 27, 2023
3140a1d
testing print statements
danieltherealyang Aug 2, 2023
4995eed
removed custom join and terminate
danieltherealyang Aug 2, 2023
b9da555
wrote documentation
danieltherealyang Aug 2, 2023
13435b5
Added notes on multiprocess live profiling
danieltherealyang Aug 12, 2023
ec0ba70
memory profiler works correctly with multiprocessing but modules gett…
danieltherealyang Aug 13, 2023
723acc6
Summary: fixed inheritance issues preventing modules from getting pro…
danieltherealyang Aug 17, 2023
42303cb
commented out multiproc_test in slips.py, text from multiprocessPatch…
danieltherealyang Aug 18, 2023
9ce6349
made changes
danieltherealyang Aug 18, 2023
3c339b5
removed print
danieltherealyang Aug 18, 2023
1ff3aa7
made small change
danieltherealyang Aug 22, 2023
2bb153c
Merge remote-tracking branch 'danieltherealyang/memory-profile' into …
AlyaGomaa Sep 15, 2023
947be4f
add docs for memory and CPU profiler [skip ci]
AlyaGomaa Sep 15, 2023
42a2111
update changelog, slips version and requirements.txt
AlyaGomaa Sep 15, 2023
cd73bc7
disable memory profiler by default
AlyaGomaa Sep 15, 2023
7151cd0
update slips.gif to use v1.0.7
AlyaGomaa Sep 15, 2023
97b9175
Merge pull request #388 from danieltherealyang/memory-profile
AlyaGomaa Sep 15, 2023
3150273
Merge remote-tracking branch 'origin/develop' into develop
AlyaGomaa Sep 15, 2023
979eb70
fix error stopping daemon [skip ci]
AlyaGomaa Sep 15, 2023
dd0e005
to trigger ci
AlyaGomaa Sep 15, 2023
fe19f88
comment out faling unit test
AlyaGomaa Sep 15, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix error stopping daemon [skip ci]
  • Loading branch information
AlyaGomaa committed Sep 15, 2023
commit 979eb7091c261f9959d3855face7cf06423b6038
7 changes: 4 additions & 3 deletions slips.py
Original file line number Diff line number Diff line change
Expand Up @@ -122,9 +122,10 @@ def cpu_profiler_init(self):
self.cpuProfilerEnabled = False

def cpu_profiler_release(self):
if self.cpuProfilerEnabled and not self.cpuProfilerMultiprocess:
self.cpuProfiler.stop()
self.cpuProfiler.print()
if hasattr(self, 'cpuProfilerEnabled' ):
if self.cpuProfilerEnabled and not self.cpuProfilerMultiprocess:
self.cpuProfiler.stop()
self.cpuProfiler.print()

def memory_profiler_init(self):
self.memoryProfilerEnabled = slips.conf.get_memory_profiler_enable() == "yes"
Expand Down