Neural-Artist, Feed-Forward Styling in Pytorch

A fully featured implementation of Neural Style that includes videos, webcam and web stylization(Coming Soon™).

Candy x Obrien Centre, University College Dublin

It took 2.5 hours to train the feedforward algorithm on a gtx 1080, and less than 300ms to test on a 1920x1080 photo of O'Brien Centre for Science

Table of contents

• Home
• Image Styization
• Training
  • Setup
  • Running Training
  • Customization
• Video Stylization
  • How to
  • Adding Audio
• Webcam Stylization
  • How to
• Export To Onnx
• Acknowledgments
• Cool Stylized Examples

Image Stylization

Models are located in the models folder

python style.py eval --content {content_image} --model {path to model} --out {output image}

Example:

python style.py eval --content content.jpg --model model.pth --out output.jpg

Training

Setup

Install Prerequisites

Python is a must, specifically python 3.

You'll need a package manager. Pip or Conda will do, I personally use miniconda. A plus for conda is that you don't need to install python before hand.

Finally if you use conda, specifically miniconda, you need to run conda install pip, preferably in an environment.

Make sure to

• Install Pytorch and Torchvision
  • Visit the Pytorch website to install for your relevant OS and package manager. I used conda, hence the following command:
    • conda install pytorch torchvision cudatoolkit=10.2 -c pytorch
• Install OpenCV
  • pip install opencv-python
• Install tqdm
  • pip install tqdm

Download Training Data (Only if you intend on training a model)

We need to download the COCO dataset.

Simply run ./setup_train.sh to download and setup the dataset.

Running Training

For simple Training run:

python style.py train --style images/style/wave_2.jpg --dataset data --checkpoint-dir models

Customization

• Model Customization, checkout src/train.py
  • You can also choose between VGG16 and VGG19
  • You can also choose specific style layers and content layers
• Choose weights for Style, Content & TV Loss
• Choose Loss Network input range (0,255) or (0,1)
• Find out about Flags for hyperparameters here

There are a multitude of variations you can do for training that will result in major differences.

For Example using the 256x256 Udnie results in smaller feature transformers than the larger 512x512 Udnie

Small Feature Udnie

Large Feature Udnie

Video Stylization

K-pop is in, right?

It took 22 minutes on a GTX 1080 to style the full (1920x1080) video by Black Pink. Full video here.

How to Video:

You can produce a simple mp4 video using the following command:

python style.py video --model {path to model} --content {path to content video} --output-dir {output path} --show-frame {True/False, shows current Frame being processed}

Example:

python style.py video --model models/style_cubist.pth --content images/content/blackpink.mp4 --output-dir images/results

Audio

Audio isn't processed with opencv, so you'll have to use ffmpeg if you want audio added to your video.

ffmpeg -i {input mp4} -i {input audio mp4 or mp3} -c copy output.mp4

Example:

ffmpeg -i images/results/blackpink.mp4 -i audio.mp3 -c copy output.mp4

Webcam Stylization

Udnie Wave

Candy Cubist

How to Webcam:

You can produce live webcam styling using the following:

python style.py cam --model {path to model} --width {Your chosen camera width} --height {Your chosen camera height}

Example

python style.py cam --model models/udnie.pth

Export to Onnx

For wide platform compatibility I've added the option to export to Onnx. I have plans to try using ONNX.js for native on-browser stylization.

python style.py export --model {model path} --output-dir {output path}

Example:

python style.py export --model models/udnie.pth

Acknowledgements

• Original Algorithm by Leon A. Gatys: A Neural Algorithm of Artistic Style
• Feedforward method developed by Justin Johnson: Perceptual Losses for Real-Time Style Transfer and Super-Resolution
• Justin Johnson's Original Repo that I used for hyperparameter optimization
• Demystifying Neural Style Transfer
• Improved Fast Stylization using Instance Normalization by Dmitry Ulyanov
• Pytorch example library I used for implementing the auto encoder and debugging(Normalizing input completely flew over my head 🙃)
• I got the cubist painting from hzy46, I don't know where it's from unfortunately.
• README styling influenced by Lengstrom and Justin Johnson

Example Image Transformations

London