Detect Outdated Application or Services Using Network Scanning with Take action

this Repository contains Docker-compose for Airflow, Postgresql, pgAdmin, JupyterLab and MongoDB. also have Airflow dags, notebooks for Detect Outdated Application or Services Using Network Scanning

Prerequisites

• Install Python 3.6 or above
• Install Docker
• Install Docker Compose

Test Environment

Metasploitable is virtual machine based on Linux that contains several intentional vulnerabilities for you to exploit. It is essentially a penetration testing lab in a box, available as a VMware virtual machine (VMX). (The Metasploitable login is “msfadmin”; the password is also “msfadmin”.)

Exploit Database

ExploitDB is an archive of public exploits and corresponding vulnerable software, developed for use by penetration testers and vulnerability researchers. This Database used to check the result of network scan against it.

Files

• The Dags located in ./notebooks/src directory
• The Jupyter notebooks located in ./notebooks directory

Installation

Clone the Repository and navigate to docker-airflow directory
Then run the docker-compose file

docker-compose up -d

Usage

• Airflow => localhsot:8080
• JupyterLab => localhsot:8888
• PGAdmin => localhsot:8888

Extract and Load ExploitDB

• Using PGAdmin create ExploitDB database
• Trigger the DAG manually

Detect Outdated Application or Services Using Network Scanning

• Change the IP address in nmap_scan DAG with Metasploitable OS IP address
• Change the email and password in nmap_scan DAG with your credentials
• Trigger the DAG manually

Information

The repo contains two main airflow workflows dags the first one is load_exploits_data and the second one is nmap_scan. the following is detailed description for each dag:

load_exploits_data

the exploitDB extracted from The Github repo mentioned above and some clean operation applied on it then the data loaded on PostgresDB. the operations inside the dag as following:

• exrtract_data operation is a Python Operation used to extract the data from The Github repo and stored it on raw shape in PostgresDB
• transform_data operation is a Python Operation used to clean the data, the cleaning process contains split the description into three fields description, product and version

Workflow Design


Result screenshot

nmap_scan

the namp python package used to scan the netwrok and return the open ports with the binded application(Name and version) on each port. after that check process aginst it ExploitDB in Postgress Applied to get the outdated application and service. finally the outdated application and services killed and email sent to the admin.

• scan_host operation is a Python Operation used to scan the host using nmap package to get the open ports and binded application then pass the result to the next operation using xCom in airflow
• scan_db operation is a Python Operation recive information from the pervious step then check if the applications with specific version is outdated or not is yes the information of outdated applicated and related ports sent to the next operation using xCom in airflow
• kill_process operation is a Python Operation used to kill outdated application on the host using paramiko package which is python ssh client.
• send_email operation is a Python Operation used to send email the admin about the killed processes, reasons and exploit details.

Workflow Design


Result screenshot

Roadmap

• Apply Better way to extract Product version
• Apply fuzzy search on Product Name and Version
• Kill process in Windows server