VioPose

This repository provides code and pretrained weights for the paper. The code is provided without any warranty. If you benefited from the work, or partially or directly use our code, please cite our work as shown below. For more information please visit our project website

@inproceedings{yooshrestha2024viopose,
	title     = {{VioPose}: Violin Performance 4D Pose Estimation by Hierarchical Audiovisual Inference},
	author    = {Yoo, Seong Jong and Shrestha, Snehesh and Muresanu, Irina and Fermüller, Cornelia},
	journal   = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
	year      = {2025},
}

Installation

Installation using Conda

conda create -n viopose python==3.9
conda activate viopose
conda install pip
pip install -r requirements.txt
./download.sh

• Install tensorflow $\approx$ 2.13 and cuda accordingly
• Download dataset (will release soon) and locate it at data/FullData

Run Demo

VioDat Test Demo (currently unavailable)

1. Run test.py script. This code will generate and save result.npz at ./Logs/VioPose

python3 test.py --folder Logs/VioPose --data violin --data_path data/FullData/mmViolin_v1.0.npz

2. Run generate_video.py script

In-the-wild Demo

1. Prepare video (mp4 format) and audio (wav format). For demo we prepared video at ./data/demo
2. Run inference.py script

python3 inference.py --folder Logs/VioPose --video_path data/demo/demo.mp4 --audio_path data/demo/demo.wav --vis True

• If you don't provide audio file then VioPose_wo_audio model will be used for inference
• The output video is saved at ./output/output_w_audio.mp4 with audio and ./output/output.mp4 without audio

Train from Scratch

1. Prepare VioDat Dataset at ./data/FullData
2. Run main.py script

python3 main.py --cfg ./config/viopose.yaml > ./out_log/viopose.out
python3 test.py --folder /Logs/viopose --data violin --data_path data/FullData/mmViolin_v1.0.npz

Bug Report

Please raise an issue on Github for issues related to this code. If you have any questions related about the code feel free to send an email to here ([email protected]).

License

VioPose and VioDat is available for non-commercial and research use only and may not be redistributed and should follow requirements under the conditions detailed on the license page. For commercial licensing or if you have any questions, please get in touch with us at [email protected].