ACL

This is an official PyTorch implementation of **"Joint Adversarial and Collaborative Learning for Self-Supervised Action Recognition" **.

Requirements

Data Preparation

• Download the raw data of NTU RGB+D and PKU-MMD.
• For NTU RGB+D dataset, preprocess data with tools/ntu_gendata.py. For PKU-MMD dataset, preprocess data with tools/pku_part1_gendata.py.
• Then downsample the data to 50 frames with feeder/preprocess_ntu.py and feeder/preprocess_pku.py.
• If you don't want to process the original data, download the file folder in Google Drive action_dataset or BaiduYun link action_dataset, code: 0211. NTU-120 is also provided: NTU-120-frame50.

Installation

# Install torchlight
$ cd torchlight
$ python setup.py install
$ cd ..

# Install other python libraries
$ pip install -r requirements.txt

Unsupervised Pre-Training and Linear Evaluation

Example for unsupervised pre-training and linear evaluation of 3s-CACA. You can change some settings of .yaml files in config/three_streams/dataset folder. More examples refer to run.sh.

# train on NTU RGB+D xsub (three-stream)
$ python main.py pretrain_skeleton_3views --config config/three_streams/ntu60_cs/pretext_caca_3views_xsub_cross_2_10.yaml
$ python main.py linear_evaluation_3views --config config/three_streams/ntu60_cs/linear_eval_caca_3views_xsub_cross_2_10.yaml

# train on NTU RGB+D xsub (joint-stream)
$ python main.py pretrain_skeleton --config config/single_stream/stgcn/ntu60_cs/pretext/pretext_caca_512_2048_512_2048_0.996_joint.yaml
$ python main.py linear_evaluation --config config/single_stream/stgcn/ntu60_cs/linear_eval/linear_eval_caca_512_2048_512_2048_0.996_joint.yaml

Trained models

We release several trained models in released_model. You can download them, put them in model and test them with linear evaluation by changing weights in .yaml files.

dataset	Linear Evaluation (%)
NTU-60 xsub	78.57
NTU-60 xsub	84.50
NTU-120 xsub	68.54
NTU-120 xset	71.10
PKU-MMD part I	88.05
PKU-MMD part II	53.48

Visualization

The t-SNE visualization of the embeddings after pre-training on NTU60-xview.

Citation

Please cite our paper if you find this repository useful in your resesarch:

@inproceedings{guo2022aimclr,
  Title= {Joint Adversarial and Collaborative Learning for Self-Supervised Action Recognition},
  Author= {Tianyu, Guo and Mengyuan, Liu and Hong, Liu and Wehao, Li and Jingwen, Guo and Tao, Wang and Yidi, Li},
  Booktitle= {-},
  Year= {2023}
}

Acknowledgement

The framework of our code is extended from the following repositories. We sincerely thank the authors for releasing the codes.

• The framework of our code is based on CrosSCLR and AimCLR.
• The encoder is based on ST-GCN.

Licence

This project is licensed under the terms of the MIT license.