Learning Linear Transformations for Fast Image and Video Style Transfer

[Paper] [Project Page]

Prerequisites

• Pytorch
• torchvision
• opencv for video generation

All code tested on Ubuntu 16.04, pytorch 0.4.1, and opencv 3.4.2

Style Transfer

• Clone from github: git clone https://github.com/sunshineatnoon/LinearStyleTransfer
• Download pre-trained models from google drive.
• Uncompress to root folder :

cd LinearStyleTransfer
unzip models.zip
rm models.zip

Artistic style transfer

python TestArtistic.py

or conduct style transfer on relu_31 features

python TestArtistic.py --vgg_dir models/vgg_r31.pth --decoder_dir models/dec_r31.pth --matrixPath models/r31.pth --layer r31

Photo-realistic style transfer

For photo-realistic style transfer, we need first compile the pytorch_spn repository.

cd libs/pytorch_spn
sh make.sh
cd ../..

Then:

python TestPhotoReal.py

Note: images with _filtered.png as postfix are images filtered by the SPN after style transfer, images with _smooth.png as postfix are images post process by a smooth filter.

Video style transfer

python TestVideo.py

Real-time video demo

python real-time-demo.py --vgg_dir models/vgg_r31.pth --decoder_dir models/dec_r31.pth --matrixPath models/r31.pth --layer r31

Model Training

Data Preparation

• MSCOCO

wget http://msvocds.blob.core.windows.net/coco2014/train2014.zip

• WikiArt
  • Either manually download from kaggle.
  • Or install kaggle-cli and download by running:

  kg download -u <username> -p <password> -c painter-by-numbers -f train.zip
  

Training

Train a style transfer model

To train a model that transfers relu4_1 features, run:

python Train.py --vgg_dir models/vgg_r41.pth --decoder_dir models/dec_r41.pth --layer r41 --contentPath PATH_TO_MSCOCO --stylePath PATH_TO_WikiArt --outf OUTPUT_DIR

or train a model that transfers relu3_1 features:

python Train.py --vgg_dir models/vgg_r31.pth --decoder_dir models/dec_r31.pth --layer r31 --contentPath PATH_TO_MSCOCO --stylePath PATH_TO_WikiArt --outf OUTPUT_DIR

Key hyper-parameters:

• style_layers: which features to compute style loss.
• style_weight: larger style weight leads to heavier style in transferred images.

Intermediate results and weight will be stored in OUTPUT_DIR

Train a SPN model to cancel distortions for photo-realistic style transfer

Run:

python TrainSPN.py --contentPath PATH_TO_MSCOCO

Acknowledgement

• We use the smooth filter by LouieYang in the photo-realistic style transfer.

Citation

@inproceedings{li2018learning,
    author = {Li, Xueting and Liu, Sifei and Kautz, Jan and Yang, Ming-Hsuan},
    title = {Learning Linear Transformations for Fast Arbitrary Style Transfer},
    booktitle = {IEEE Conference on Computer Vision and Pattern Recognition},
    year = {2019}
}