gan-deeplab-spade

Code for ‘Drawing’ the inner world of a story using GauGAN in a real environment.

Installation

• Make sure you are using an SSH key to connect to GitHub.

Clone the repository including its submodules

git clone --recurse-submodules [email protected]:Marrow-AI/gan-deeplab-spade.git

You can safely ignore the error Failed to recurse into submodule path 'SPADE'.

Create and activate a conda environment from the suppplied file

cd gan-deeplab-spade
conda env create --file environment.yml
conda activate deeplab-spade

Note: The supplied environment requires a CUDA enabled GPU device. It may be possible to create your own environment without GPU acceleration.

Download the DeepLabV2 CocoStuff checkpoint:

mkdir -p checkpoints/deeplab
gdown --id 18kR928yl9Hz4xxuxnYgg7Hpi36hM8J2d -O checkpoints/deeplab/deeplabv2_resnet101_msc-cocostuff164k-100000.pth

Download and extract the SPADE checkpoint:

mkdir -p checkpoints/spade
gdown --id 12gvlTbMvUcJewQlSEaZdeb2CdOB-b8kQ -O checkpoints/spade/checkpoints.tar.gz
tar xvf checkpoints/spade/checkpoints.tar.gz -C checkpoints/spade/

Running

Run the demo to generate images from your webcam:

./demo_spade.sh

Label manipulation

The demo transforms bottles (label ID 43) into flowers (label ID 118) in this line:

labelmap[labelmap == 43] = 118

Mappings between IDs and labels are found here.

Attribution

• Feel free to use the code for any purpose, but we request that you attribute the work to shirin anlen and Avner Peled of the Marrow project.
• Deeplab pytorch implementation by Kazuto Nakashima.
• GauGAN SPADE implementation by NVIDIA Corporation.

Thank you and enjoy!