video-hand-coordinates

This github contains two main scripts:

• video_to_hands.py:

  outputs hand coordinates from videos

• show_hands_coords.py:

  shows video with coordinates and arrows plotted for the speed and acceleration of each keypoint

/!\ check the requirements

Getting hand coordinates from videos

Usage

python video_to_hands.py\
<path to mmpose git folder>/demo/mmdetection_cfg/cascade_rcnn_x101_64x4d_fpn_1class.py cascade_rcnn_x101_64x4d_fpn_20e_onehand10k-dac19597_20201030.pth\
<path to mmpose git folder>/configs/hand/2d_kpt_sview_rgb_img/deeppose/onehand10k/res50_onehand10k_256x256.py deeppose_res50_onehand10k_256x256-cbddf43a_20210330.pth\
--video-root <folder path of input videos> --out-video-root [folder path of outputed annotated videos]\
--savepath <folder path of the outputed coordinates> [--device <cpu/gpu>] [--show] 

notes:

• --show will open an OpenCV window showing the model's inferences in real-time
• if --device is not specified, it will run on gpu

Input format

--video-root argument, give the folder in which are your .mp4 / .MP4 videos

Output format

--savepath argument, give the folder in which to save coordinates files.

This python script outputs a pickle file containing all relevant hand coordinates. This is its structure:

The pickle file is a list, each element of the list correspond to one frame of the video.

For each frame, the list element is a python dictionnary containing 0, 1 or 2 keys : none, left and/or right.
The dictionnary values is another dictionnary with 2 keys: 'keypoints' and 'bbox'.
The values for 'keypoints' is a list of the hand keypoints, and the value for 'bbox' contains the detection bounding box for the hand and its inference confidence.

i.e. pickle_list[2649]['left']['keypoints'][12] is the 12th keypoint of the left hand visible on the 2649th frame

Showing coordinates on videos

Usage

python show_hands_coords.py <path of the video> <path of the coordinates file> [--out-video <path of the outputed video file>]

Requirements

In order to use this github, you need to clone and install the following github repositories in a conda environment:

• install instructions for mmpose
• install instructions for mmdetection

You will also need to download two weight files to be loaded in the machine learning models (you can put them in a 'weights' folder):

• deeppose_res50_onehand10k_256x256-cbddf43a_20210330.pth for the hand pose model
• cascade_rcnn_x101_64x4d_fpn_20e_onehand10k-dac19597_20201030.pth for the hand detection model

optional: create these folders weights video output, to store respectively the weights downloaded above, your videos, and your outputted pickle files.