MIDAS implementation in Julia

Anomaly Detection on Dynamic (time-evolving) Graphs in Real-time and Streaming manner. Detecting intrusions (DoS and DDoS attacks), frauds, fake rating anomalies.

Installation

using Pkg
Pkg.add("https://github.com/ashryaagr/MIDAS.jl")

Table of Contents

• Features
• Example
• Online Articles
• MIDAS in other Languages
• Citation

Features

• Finds Anomalies in Dynamic/Time-Evolving Graph: (Intrusion Detection, Fake Ratings, Financial Fraud)
• Detects Microcluster Anomalies (suddenly arriving groups of suspiciously similar edges e.g. DoS attack)
• Theoretical Guarantees on False Positive Probability
• Constant Memory (independent of graph size)
• Constant Update Time (real-time anomaly detection to minimize harm)

Example

using MIDAS
using ROC

# Load data and ground truth labels
data, labels = @load_darpa

# scores using midas algorithm
anomaly_score = midas(
    data,
    num_rows=2,
    num_buckets=769
    )

# ROC analysis of scores of midas. This will take some time to run
roc_midas = roc(anomaly_score, labels, 1.0)

# AUC value for midas
println(AUC(roc_midas))


# scores using midasR algorithm.
anomaly_score_R = midasR(
    data,
    num_rows=2,
    num_buckets=769,
    factor=0.4
    )

# ROC analysis for midasR. This will take some time to run
roc_midasR = roc(anomaly_score_R, labels, 1.0)

# AUC value for midasR
println(AUC(roc_midas_R))

Online Articles

1. KDnuggets: Introducing MIDAS: A New Baseline for Anomaly Detection in Graphs
2. Towards Data Science: Controlling Fake News using Graphs and Statistics
3. Towards Data Science: Anomaly detection in dynamic graphs using MIDAS
4. Towards AI: Anomaly Detection with MIDAS
5. AIhub Interview

MIDAS in Other Languages

1. Golang by Steve Tan
2. Ruby by Andrew Kane
3. Rust by Scott Steele
4. R by Tobias Heidler
5. Python by Ritesh Kumar
6. Java by Joshua Tokle
7. C++ by Siddharth Bhatia

Note: This julia implementation is based on the research paper of authors of c++ implementation and the Julia implementation adopts the design from python implementation.

Citation

If you use this code for your research, please consider citing the paper.

@inproceedings{bhatia2020midas,
    title="MIDAS: Microcluster-Based Detector of Anomalies in Edge Streams",
    author="Siddharth {Bhatia} and Bryan {Hooi} and Minji {Yoon} and Kijung {Shin} and Christos {Faloutsos}",
    booktitle="AAAI 2020 : The Thirty-Fourth AAAI Conference on Artificial Intelligence",
    year="2020"
}