Unable to reproduce the RAT06-Saefko results as described at https://stratospherelinuxips.readthedocs.io/en/develop/slips_in_action.html

[srini38]

Describe the bug

Tried using the RAT06-Saefko test as described in https://stratospherelinuxips.readthedocs.io/en/develop/slips_in_action.html both using Slips 1.0.7 docker and Slips 1.07 normal install. Basically the Evidence thread does not report any malicious/infection. Also see
"Killing modules that took more than 15.0 mins to finish." message both in docker and in host/local mode runs.

To Reproduce
Steps to reproduce the behavior:
Shared in screenshots section

Expected behavior
Expected output https://raw.githubusercontent.com/stratosphereips/StratosphereLinuxIPS/develop/docs/images/slips.gif

Screenshots

root@user-virtual-machine:/home/user/source/StratosphereLinuxIPS# sudo ./slips.py -e 1 -f RAT06_Saefko.pcap
[Main] Storing Slips logs in output/RAT06_Saefko.pcap_2023-11-12_00:31:49/
Slips. Version 1.0.7 (822db6d7)
https://stratosphereips.org
---------------------------
[Main] Using redis server on port: 6379
Started Main process [PID 4918]
Started Output Process [PID 4929]
Starting modules
                Starting the module Risk IQ (Module to get passive DNS info about IPs from RiskIQ) [PID 4944]
                Starting the module ARP (Detect arp attacks) [PID 4945]
                Starting the module Flow Alerts (Alerts about flows: long connection, successful ssh, password guessing, self-signed certificate, data exfiltration, etc.) [PID 4947]
                Starting the module Flow ML Detection (Train or test a Machine Learning model to detect malicious flows) [PID 4949]
                Starting the module HTTP Analyzer (Analyze HTTP flows) [PID 4951]
                Starting the module IP Info (Get different info about an IP/MAC address) [PID 4952]
                Starting the module Leak Detector (Detect leaks of data in the traffic) [PID 4955]
                Starting the module Network Discovery (Detect Horizonal, Vertical Port scans, ICMP, and DHCP scans) [PID 4958]
                Starting the module Threat Intelligence (Check if the source IP or destination IP are in a malicious list of IPs) [PID 4959]
                Starting the module Timeline (Creates kalipso timeline of what happened in the network based on flows and available data) [PID 4962]
                Starting the module Update Manager (Update Threat Intelligence files) [PID 4963]
                Starting the module Virustotal (IP, domain and file hash lookup on Virustotal) [PID 4965]
---------------------------
[Main] Disabled Modules: ['template', 'ensembling', 'rnnccdetection', 'Exporting Alerts', 'p2ptrust', 'CESNET', 'blocking', 'CYST']
[Evidence] Storing Slips logs in output/RAT06_Saefko.pcap_2023-11-12_00:31:49/
Started Evidence Process [PID 4967]
Started Profiler Process [PID 4968]
[Main] Metadata added to output/RAT06_Saefko.pcap_2023-11-12_00:31:49/metadata
Started Input Process [PID 4969]
[Main] Warning: Slips may generate a large amount of traffic by querying TI sites.
[Input] Storing zeek log files in output/RAT06_Saefko.pcap_2023-11-12_00:31:49/zeek_files
[Update Manager] Connection error while downloading the file https://check.torproject.org/torbulkexitlist. Aborting.
[Input] We read everything. No more input. Stopping input process. Sent 1048 lines hr: 3. (2023/11/12 00:32:28)
[Update Manager] Error while reading the TI file modules/threat_intelligence/remote_data_files/AIP_historical_blacklist_prioritized_by_newest_attackers.csv. Could not find a column with an IP or domain
[Update Manager] Error parsing feed https://mcfp.felk.cvut.cz/publicDatasets/CTU-AIPP-BlackList/Todays-Blacklists/AIP_historical_blacklist_prioritized_by_newest_attackers.csv. Updating was aborted.
Total analyzed IPs so far: 15. Evidence added: 234. IPs sending traffic in the last 1 hr: 0. (2023/11/12 00:34:38)
---------------------------
Stopping Slips

[Main] Analysis of RAT06_Saefko.pcap finished in 2.81 minutes
        Network Discovery       Stopped. 13 left.
        Leak Detector           Stopped. 12 left.
        Flow ML Detection       Stopped. 11 left.
        ARP                     Stopped. 10 left.
        IP Info                 Stopped. 9 left.
        Threat Intelligence     Stopped. 8 left.
        HTTP Analyzer           Stopped. 7 left.
        Timeline                Stopped. 6 left.

[Main] The following modules are busy working on your data.

['Update Manager', 'Flow Alerts', 'Output', 'Input', 'Evidence', 'Profiler']

You can wait for them to finish, or you can press CTRL-C again to force-kill.

[Main] Update Manager may take several minutes to finish updating 45+ TI files.
        Flow Alerts             Stopped. 5 left.
Killing modules that took more than 15.0 mins to finish.
        Output                  Stopped. 4 left.
        Update Manager          Stopped. 3 left.
        Input                   Stopped. 2 left.
        Evidence                Stopped. 1 left.
        Profiler                Stopped. 0 left.
root@user-virtual-machine:/home/user/source/StratosphereLinuxIPS# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.5 LTS
Release:        20.04
Codename:       focal
root@user-virtual-machine:/home/user/source/StratosphereLinuxIPS# cd output/RAT06_Saefko.pcap_2023-11-12_00:31:49/
root@user-virtual-machine:/home/user/source/StratosphereLinuxIPS/output/RAT06_Saefko.pcap_2023-11-12_00:31:49# grep -r Evidence
slips.log:2023/11/12 00:31:58.251274 [Evidence] Storing Slips logs in output/RAT06_Saefko.pcap_2023-11-12_00:31:49/
root@user-virtual-machine:/home/user/source/StratosphereLinuxIPS/output/RAT06_Saefko.pcap_2023-11-12_00:31:49# grep -r infection
root@user-virtual-machine:/home/user/source/StratosphereLinuxIPS/output/RAT06_Saefko.pcap_2023-11-12_00:31:49#

Branch
root@user-virtual-machine:/home/user/source/StratosphereLinuxIPS# git log | more
commit 822db6d
Merge: f938969 fe19f88
Author: Alya Gomaa [email protected]
Date: Fri Sep 15 16:46:59 2023 +0300

Merge pull request #397 from stratosphereips/develop

Slips v1.0.7

Environment (please complete the following information):

• OS: [e.g. iOS] Ubuntu

• Version [e.g. 22] Ubuntu 20.04.5 LTS

• Python version [e.g. 3.8] python3 --version
  Python 3.8.10

• Are you running slips in docker or locally? [yes/no] tried both

• Docker version (if running slips in docker) [e.g. 20.10.22] Docker version 24.0.2, build cb74dfc

• Slips docker image used (if running slips in docker) [e.g. macosm1-image, macosm1-P2P-image, ubuntu-image, dependency-image] 1.0.7 ubuntu based image

Additional context
Add any other context about the problem here.

[AlyaGomaa]

Hello @srini38
So, this tutorial is using an older version of slips, now in v1.0.7 we changed the ensembling of alerts
which means that, evidence that used to trigger an alert before, won't necessary trigger one now.

the colored alerts you're seeing in the expected behaviour is an alert. and now in v1.0.7 slips doesn't think that the generated evidence are enough to trigger an alert

However, you can still see the generated evidence by checking the output directory, which is output/RAT06_Saefko.pcap_2023-11-12_00:31:49/alerts.log in your case above. but it changes every run.

let me know if you managed to do so annd I'll update the docs with the used slips version in the tutorial to avoid any confusion.

[srini38]

Hello @AlyaGomaa

Thank you for your response. Post checking out git commit 42a2111, I was able to reproduce the output of https://raw.githubusercontent.com/stratosphereips/StratosphereLinuxIPS/develop/docs/images/slips.gif using https://mcfp.felk.cvut.cz/publicDatasets/CTU-Malware-Capture-Botnet-25-1/2013-11-06_capture-win6.pcap

I am trying to test the flowmldetection module using the pre-trained model model.bin and scaler.bin. Please let me know if there are any pcaps that I could use to get flowmldetection module to detect and log alerts? Tried using RAT06_Saefko.pcap and slips in -e 3 mode, but could not see any ML based alerts.

[AlyaGomaa]

hey good to hear!

unfortunately, i don't know of any pcap that triggers this evidence at the moment. but will check with @eldraco and get back to you, maybe we can create a PCAP to trigger this evidence specifically.