X as Supervision: Contending with Depth Ambiguity in Unsupervised Monocular 3D Pose Estimation

Yuchen Yang  ·  Xuanyi Liu  ·  Xing Gao  ·  Zhihang Zhong  ·  Xiao Sun

arXiv

Installation

Refer to INSTALL.md

Data Preparation

Refer to DATA_PREPARATION.md

Get Started

We provide scripts for training and testing on SLURM. Tensorboard records are saved in log directory.

You can manually launch the task using torchrun in the script.

Train

cd scripts
./launch_train.sh <partition> <gpu_num> ../config/<config_name>S1.yaml <extra_tag(optional)>

Finetune

./launch_finetune.sh <partition> <gpu_num> ../config/<config_name>S2.yaml ../log/<checkpoint_name>/<checkpoint>.pth.tar <extra_tag(optional)>

Eval

eval_mode: ['best', 'confident']

./launch_eval.sh <partition> <gpu_num> ../config/<config_name>S2.yaml ../log/<checkpoint_name>/<checkpoint>.pth <eval_mode>

Note

We conduct experiments in two stages: training with *S1.yaml and finetuning with *S2.yaml. One-stage training is also works, but the performance is not as good as two-stage training.

Configs are named in <dataset>_<detector_type>_<distribution_type><stage>.yaml.

Further experiment code and configs, including 3D-2D mix training, single hypothesis, ... can be found in this url.

Model Zoo

We provide the pretrained models in this url.

Citation

If you find this work useful in your research, please consider citing:

@misc{yang2024X,
      title={X as Supervision: Contending with Depth Ambiguity in Unsupervised Monocular 3D Pose Estimation}, 
      author={Yuchen, Yang and Xuanyi, Liu and Xing, Gao and Zhihang, Zhong and Xiao, Sun},
      year={2024},
      eprint={2411.13026},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgement

We thank the authors of 3D Pose Baseline, MMHuman3D, IntegralPose, SMPLPytorch, FLAME, Surreal, SAM for their great works. We partially refer to their codebases for this project.