HSTforU: Anomaly Detection in Aerial and Ground-based Videos with Hierarchical Spatio-Temporal Transformer for U-net
This is the code for HSTforU: Anomaly Detection in Aerial and Ground-based Videos with Hierarchical Spatio-Temporal Transformer for U-net 
MoGuP: See MoGuP:Motion-guided Prediction for Video Anomaly Detection.
ASTNet: See Attention-based Residual Autoencoder for Video Anomaly Detection.
- [8/19/2023] Code of HSTforU is released (with password). We will remove the password after this paper is accepted.
The code can be run under any environment with Python 3.7 and above. (It may run with lower versions, but we have not tested it).
Install the required packages:
pip install -r requirements.txt
Clone this repo:
git clone https://github.com/vt-le/HSTforU.git
cd HSTforU/
We evaluate HSTforU on:
| Dataset | Link |
|---|---|
| UCSD Ped2 | |
| CUHK Avenue | |
| ShanghaiTech | |
| Drone-Anomaly |
A dataset is a directory with the following structure:
$ tree data
ped2/avenue
├── training
│ └── frames
│ ├── ${video_1}$
│ │ ├── 000.jpg
│ │ ├── 001.jpg
│ │ └── ...
│ ├── ${video_2}$
│ │ ├── 00.jpg
│ │ └── ...
│ └── ...
├── testing
│ └── frames
│ ├── ${video_1}$
│ │ ├── 000.jpg
│ │ ├── 001.jpg
│ │ └── ...
│ ├── ${video_2}$
│ │ ├── 000.jpg
│ │ └── ...
│ └── ...
└── ped2/avenue.mat
shanghaitech
├── training
│ └── frames
│ ├── ${video_1}$
│ │ ├── 000.jpg
│ │ ├── 001.jpg
│ │ └── ...
│ ├── ${video_2}$
│ │ ├── 00.jpg
│ │ └── ...
│ └── ...
├── testing
│ └── frames
│ ├── ${video_1}$
│ │ ├── 000.jpg
│ │ ├── 001.jpg
│ │ └── ...
│ ├── ${video_2}$
│ │ ├── 000.jpg
│ │ └── ...
│ └── ...
└── test_frame_mask
├── 01_0014.npy
├── 01_0015.npy
└── ...
drone
├──bike
│ ├──training
│ │ └── frames
│ │ ├── ${video_1}$
│ │ │ ├── 0.jpg
│ │ │ ├── 1.jpg
│ │ │ └── ...
│ │ ├── ${video_2}$
│ │ │ ├── 00.jpg
│ │ │ └── ...
│ │ └── ...
│ ├── testing
│ │ └── frames
│ │ ├── ${video_1}$
│ │ │ ├── 000.jpg
│ │ │ ├── 001.jpg
│ │ │ └── ...
│ │ ├── ${video_2}$
│ │ │ ├── 000.jpg
│ │ │ └── ...
│ │ └── ...
│ └── annotation
│ ├── 01.npy
│ ├── 0.npy
│ └── ...
├── highway
│ ├── ...
└── ...
Please first download the pre-trained model
| Dataset | Pretrained Model |
|---|---|
| UCSD Ped2 | |
| CUHK Avenue | |
| ShanghaiTech | |
| Drone-Anomaly |
To evaluate a pretrained HSTforU on a dataset, run:
python test.py \
--cfg <path/to/config/file> \
--pretrained </path/to/pre-trained/model> \
[--batch-size <batch-size> --tag <job-tag>]For example, to evaluate HSTforU on Ped2:
python test.py \
--cfg config/scripts/ped2/ped2_pvt2_hst.yaml \
--model-file output/HSTforU/ped2_pvt2_hst/ckpt_ped2.pth- We use YAML for configuration.
- We provide a couple preset configurations.
- Please refer to
config.pyfor documentation on what each configuration does.
If you find our work useful, please consider citing:
@article{hstforu,
title = {HSTforU: Anomaly Detection in Aerial and Ground-based Videos with Hierarchical Spatio-Temporal Transformer for U-net},
author = {Le, Viet-Tuan and Jin, Hulin and Kim, Yong-Guk},
}For any question, please file an issue or contact:
Viet-Tuan Le: [email protected]