Skeleton-based GAR via Spatial Temporal Panoramic Graph

This is the official implementation of ECCV'2024 paper "Skeleton-based Group Activity Recognition via Spatial-Temporal Panoramic Graph". [paper]

Installation

Install Pytorch >= 1.11.0 and the following packages:

pyyaml
tqdm
tensorboardX
fvcore

Dataset Preparation

Volleyball Fully Supervised

We use skeletal data provided by COMPOSER, you can download the data via the original link or a compressed version provided by us. The directory looks like

volleyball/
|--videos/ # we only use labels for data preparation
|--joints/ # human pose extracted by COMPOSER using HRNet
|--tracks_normalized_with_person_action_label.pkl # individual labels
|--volleyball_ball_annotation/ # ball annotations from Perez et.al.

Unzip it, put it under ./data/raw and generate human pose and ball data using

python main.py -c config/volleyball_gendata.yaml -gd

Note

You may need to modify the following path parameters in config/volleyball_gendata.yaml

• dataset_root_folder: path to the volleyball folder
• out_folder: path to the prepared npy data

Volleyball Weakly Supervised

We provide skeletal data extracted by yolov8. Download and unzip it, the directory looks like

volleyball-weak/
|--videos/ 
|--joints/ # human pose extractd by us
|--volleyball_ball_annotation/ # ball annotations from Perez et.al.

Unzip, and generate human pose and ball data using

python main.py -c config/volleyball_weak_gendata.yaml -gd

NBA dataset

Download nba.zip. We keep only activity annotations in videos from the original dataset and provide both human pose and object keypoints extracted by yolov8. Refer to the original dataset for RGB video data.

nba/
|--videos/
|--joints/ # human pose extractd using yolov8
|--objects/ # ball and basketball net detected by yolov8
|--train_video_ids
|--test_video_ids

Generate human pose and ball data using the following command. It may take several minutes to complete.

python main.py -c config/volleyball_weak_gendata.yaml -gd

Kinetics

We use the skeletal data from pyskl, please follow this instruction to download.

Unzip and put it under ./data/k400_hrnet, then generate train/eval label from the original dataset.

python main.py -c config/kinetics_gendata.yaml -gd

Train and Evaluation

Train

python main.py --config config/volleyball/mpgcn.yaml --gpus 0

Evaluate

Before evaluating, you should ensure that the trained model corresponding the config is already existed in the <--pretrained_path> or '<--work_dir>' folder. Then run

python main.py -c <path-to-config> --evaluate

Pretrained Models

• TODO

Citation

@inproceedings{li2024mpgcn,
  title={Skeleton-based Group Activity Recognition via Spatial-Temporal Panoramic Graph}, 
  author={Zhengcen Li and Xinle Chang and Yueran Li and Jingyong Su},
  booktitle={ECCV}
  year={2024},
}