Generating Talking Face Landmarks

The code for the paper "Generating Talking Face Landmarks from Speech."

You can find the project page here.

An improved version of this project can be found here.

Installation

The project depends on the following Python packages:

• Keras --- 2.2.4
• Tensorflow --- 1.9.0
• Librosa --- 0.6.0
• opencv-python --- 3.3.0.10
• dlib --- 19.7.0
• tqdm
• subprocess

It also depends on the following packages:

• ffmpeg --- 3.4.1
• OpenCV --- 3.3.0

The code has been tested on Ubuntu 16.04 and OS X Sierra and High Sierra.

Code Example

The generation code has the following arguments:

• -i --- Input speech file
  • See this link for supported audio formats.
• -m --- Input talking face landmarks model
• -d --- Delay in terms of frames, where one frame is 40 ms
• -c --- Number of context frames
• -o --- Output path

You can run the following code to test the system:

python generate.py -i test_samples/test1.flac -m models/D40_C3.h5 -d 1 -c 3 -o results/D40_C3_test1

Feature Extraction

You can run featureExtractor.py to extract features from videos directly. The arguments are as follows:

• -vp --- Input folder containing video files (if your video file types are different from .mpg or .mp4, please modify the script accordingly)
• -sp --- Path to shape_predictor_68_face_landmarks.dat. You can download this file here.
• -o --- Output file name

Usage:

python featureExtractor.py -vp path-to-video-files/ -sp path-to-shape-predictor-68-face-landmarks-dat -o output-file-name.hdf5

Training

The training code has the following arguments:

• -i --- Input hdf5 file containing training data
• -u --- Number of hidden units
• -d --- Delay in terms of frames, where one frame is 40 ms
• -c --- Number of context frames
• -o --- Output folder path to save the model

Usage:

python train.py -i path-to-hdf5-train-file/ -u number-of-hidden-units -d number-of-delay-frames -c number-of-context-frames -o output-folder-to-save-model-file

Citation

@inproceedings{eskimez2018generating,
  title={Generating talking face landmarks from speech},
  author={Eskimez, Sefik Emre and Maddox, Ross K and Xu, Chenliang and Duan, Zhiyao},
  booktitle={International Conference on Latent Variable Analysis and Signal Separation},
  pages={372--381},
  year={2018},
  organization={Springer}
}