HandFormer

PyTorch implementation of "On the Utility of 3D Hand Poses for Action Recognition", ECCV 2024.

Requirements

Data Sources

Assembly101

Poses

Download AssemblyPoses.zip from here. For access, follow the instructions from Assembly101 official repo.

$ unzip AssemblyPoses.zip -d /<path_to_your_data_dir>/assembly101_poses/

RGB

For experiments with RGB, we provide DINOv2 features (at 30fps) for view 4 (Camera ID: C10119_rgb) of Assembly101 here.

Alternatively, TSM features for any of the views can be downloaded from the same directory.

Frame-wise TSM or DINOv2 features are stored as lmdb files.

Annotations

Download the CSV files from fine-grained annotations and put them into <path_to_your_data_dir>/Assembly101_FG_annotations/.

Details about the annotations can be found here.

H2O $${\color{green}\text{Coming soon...}}$$

Environment

Create a conda environment with the necessary packages.

$ conda env create -f package_list.yml -n handformer_env

Preprocessing

The parameters and data directories for preprocessing are placed in Preprocess/config.py. Follow the instructions there to modify and run the following:

$ python Preprocess/1_get_clip_data.py
$ python Preprocess/2_get_final_data.py

Training

All the parameters and data paths for training are defined and explained in train_config.yaml. The parameters are populated with default values. Modify paths (and parameters, if necessary) and run-

$ python HandFormer/train_model.py

Both Pose-only and Pose+RGB variants can be found in the configuration file. If RGB is used, rgb_feature_source needs to be specified. Additionally, update data paths in HandFormer/feeders/feeder.py (within lines 80-115) to refer to the downloaded lmdb files of TSM or DINOv2 features.

Evaluation

To obtain test scores, simply put additional placeholder columns in test.csv to match train.csv. Set parameters in test_config.yaml and run-

$ python HandFormer/test_model.py

Prepare appropriate output file from the saved scores to submit to the evaluation platforms (e.g., [Assembly101][H2O].)

Acknowledgements

This repository is based on

• Assembly101
• MS-G3D
• ISTA-Net
• HTT

Thanks for their great work!

Citation

Please cite this work if you find it useful:

@inproceedings{shamil2025utility,
  title={On the Utility of 3D Hand Poses for Action Recognition},
  author={Shamil, Md Salman and Chatterjee, Dibyadip and Sener, Fadime and Ma, Shugao and Yao, Angela},
  booktitle={European Conference on Computer Vision},
  pages={436--454},
  year={2025},
  organization={Springer}
}