Neural_Style_Transfer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Style Transfer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import scipy.io\n",
    "import scipy.misc\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.pyplot import imshow\n",
    "from PIL import Image\n",
    "from nst_utils import *\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import pprint\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load parameters from the VGG model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pp = pprint.PrettyPrinter(indent=4)\n",
    "model = load_vgg_model(\"pretrained-model/imagenet-vgg-verydeep-19.mat\")\n",
    "pp.pprint(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing the content cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the content image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "content_image = scipy.misc.imread(\"images/louvre.jpg\")\n",
    "imshow(content_image);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Content Cost Function $J_{content}(C,G)$\n",
    "We will define the content cost function as:\n",
    "\n",
    "$$J_{content}(C,G) =  \\frac{1}{4 \\times n_H \\times n_W \\times n_C}\\sum _{ \\text{all entries}} (a^{(C)} - a^{(G)})^2\\tag{1} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_content_cost(a_C, a_G):\n",
    "    \"\"\"\n",
    "    Computes the content cost\n",
    "    \n",
    "    Arguments:\n",
    "    a_C -- tensor of dimension (1, n_H, n_W, n_C), hidden layer activations representing content of the image C \n",
    "    a_G -- tensor of dimension (1, n_H, n_W, n_C), hidden layer activations representing content of the image G\n",
    "    \n",
    "    Returns: \n",
    "    J_content -- scalar that you compute using equation 1 above.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Retrieve dimensions from a_G\n",
    "    m, n_H, n_W, n_C = a_G.get_shape().as_list()\n",
    "    \n",
    "    # Reshape a_C and a_G\n",
    "    a_C_unrolled = tf.reshape(a_C, shape=[m,-1,n_C])\n",
    "    a_G_unrolled = tf.reshape(a_G, shape=[m,-1,n_C])\n",
    "    \n",
    "    # compute the cost with tensorflow\n",
    "    J_content = 1/(4*n_H*n_W*n_C)*tf.reduce_sum(tf.square(tf.subtract(a_C_unrolled, a_G_unrolled)))\n",
    "    \n",
    "    return J_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "with tf.Session() as test:\n",
    "    tf.set_random_seed(1)\n",
    "    a_C = tf.random_normal([1, 4, 4, 3], mean=1, stddev=4)\n",
    "    a_G = tf.random_normal([1, 4, 4, 3], mean=1, stddev=4)\n",
    "    J_content = compute_content_cost(a_C, a_G)\n",
    "    print(\"J_content = \" + str(J_content.eval()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing the Style Cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the style image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "style_image = scipy.misc.imread(\"images/monet_800600.jpg\")\n",
    "imshow(style_image);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compute $G_{gram}$\n",
    "In Neural Style Transfer (NST), you can compute the Style matrix by multiplying the \"unrolled\" filter matrix with its transpose:\n",
    "\n",
    "<img src=\"images/NST_GM.png\" style=\"width:900px;height:300px;\">\n",
    "\n",
    "$$\\mathbf{G}_{gram} = \\mathbf{A}_{unrolled} \\mathbf{A}_{unrolled}^T$$\n",
    "\n",
    "#### $G_{(gram)i,j}$: correlation\n",
    "The result is a matrix of dimension $(n_C,n_C)$ where $n_C$ is the number of filters (channels). The value $G_{(gram)i,j}$ measures how similar the activations of filter $i$ are to the activations of filter $j$. \n",
    "\n",
    "#### $G_{(gram),i,i}$: prevalence of patterns or textures\n",
    "* The diagonal elements $G_{(gram)ii}$ measure how \"active\" a filter $i$ is. \n",
    "* For example, suppose filter $i$ is detecting vertical textures in the image. Then $G_{(gram)ii}$ measures how common  vertical textures are in the image as a whole.\n",
    "* If $G_{(gram)ii}$ is large, this means that the image has a lot of vertical texture. \n",
    "\n",
    "\n",
    "By capturing the prevalence of different types of features ($G_{(gram)ii}$), as well as how much different features occur together ($G_{(gram)ij}$), the Style matrix $G_{gram}$ measures the style of an image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gram_matrix(A):\n",
    "    \"\"\"\n",
    "    Argument:\n",
    "    A -- matrix of shape (n_C, n_H*n_W)\n",
    "    \n",
    "    Returns:\n",
    "    GA -- Gram matrix of A, of shape (n_C, n_C)\n",
    "    \"\"\"\n",
    "    \n",
    "    GA = tf.matmul(A,A,transpose_b=True)\n",
    "    \n",
    "    return GA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "with tf.Session() as test:\n",
    "    tf.set_random_seed(1)\n",
    "    A = tf.random_normal([3, 2*1], mean=1, stddev=4)\n",
    "    GA = gram_matrix(A)\n",
    "    \n",
    "    print(\"GA = \\n\" + str(GA.eval()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Style cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The corresponding style cost for this layer is defined as: \n",
    "\n",
    "$$J_{style}^{[l]}(S,G) = \\frac{1}{4 \\times {n_C}^2 \\times (n_H \\times n_W)^2} \\sum _{i=1}^{n_C}\\sum_{j=1}^{n_C}(G^{(S)}_{(gram)i,j} - G^{(G)}_{(gram)i,j})^2\\tag{2} $$\n",
    "\n",
    "* $G_{gram}^{(S)}$ Gram matrix of the \"style\" image.\n",
    "* $G_{gram}^{(G)}$ Gram matrix of the \"generated\" image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_layer_style_cost(a_S, a_G):\n",
    "    \"\"\"\n",
    "    Arguments:\n",
    "    a_S -- tensor of dimension (1, n_H, n_W, n_C), hidden layer activations representing style of the image S \n",
    "    a_G -- tensor of dimension (1, n_H, n_W, n_C), hidden layer activations representing style of the image G\n",
    "    \n",
    "    Returns: \n",
    "    J_style_layer -- tensor representing a scalar value, style cost defined above by equation (2)\n",
    "    \"\"\"\n",
    "    \n",
    "    # Retrieve dimensions from a_G\n",
    "    m, n_H, n_W, n_C = a_G.get_shape().as_list()\n",
    "    \n",
    "    # Reshape the images to have them of shape (n_C, n_H*n_W)\n",
    "    a_S = tf.transpose(tf.reshape(a_S, shape=[n_H*n_W, n_C]))\n",
    "    a_G = tf.transpose(tf.reshape(a_G, shape=[n_H*n_W, n_C]))\n",
    "\n",
    "    # Computing gram_matrices for both images S and G\n",
    "    GS = gram_matrix(a_S)\n",
    "    GG = gram_matrix(a_G)\n",
    "\n",
    "    # Computing the loss\n",
    "    J_style_layer = 1/(4*n_C**2*(n_H*n_W)**2)*tf.reduce_sum(tf.square(tf.subtract(GS,GG)))    \n",
    "    \n",
    "    return J_style_layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "with tf.Session() as test:\n",
    "    tf.set_random_seed(1)\n",
    "    a_S = tf.random_normal([1, 4, 4, 3], mean=1, stddev=4)\n",
    "    a_G = tf.random_normal([1, 4, 4, 3], mean=1, stddev=4)\n",
    "    J_style_layer = compute_layer_style_cost(a_S, a_G)\n",
    "    \n",
    "    print(\"J_style_layer = \" + str(J_style_layer.eval()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Style Weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "STYLE_LAYERS = [\n",
    "    ('conv1_1', 0.2),\n",
    "    ('conv2_1', 0.2),\n",
    "    ('conv3_1', 0.2),\n",
    "    ('conv4_1', 0.2),\n",
    "    ('conv5_1', 0.2)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can combine the style costs for different layers as follows:\n",
    "\n",
    "$$J_{style}(S,G) = \\sum_{l} \\lambda^{[l]} J^{[l]}_{style}(S,G)$$\n",
    "\n",
    "where the values for $\\lambda^{[l]}$ are given in `STYLE_LAYERS`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_style_cost(model, STYLE_LAYERS):\n",
    "    \"\"\"\n",
    "    Computes the overall style cost from several chosen layers\n",
    "    \n",
    "    Arguments:\n",
    "    model -- our tensorflow model\n",
    "    STYLE_LAYERS -- A python list containing:\n",
    "                        - the names of the layers we would like to extract style from\n",
    "                        - a coefficient for each of them\n",
    "    \n",
    "    Returns: \n",
    "    J_style -- tensor representing a scalar value, style cost defined above by equation (2)\n",
    "    \"\"\"\n",
    "    \n",
    "    # initialize the overall style cost\n",
    "    J_style = 0\n",
    "\n",
    "    for layer_name, coeff in STYLE_LAYERS:\n",
    "\n",
    "        # Select the output tensor of the currently selected layer\n",
    "        out = model[layer_name]\n",
    "\n",
    "        # Set a_S to be the hidden layer activation from the layer we have selected, by running the session on out\n",
    "        a_S = sess.run(out)\n",
    "\n",
    "        # Set a_G to be the hidden layer activation from same layer. Here, a_G references model[layer_name] \n",
    "        # and isn't evaluated yet. Later in the code, we'll assign the image G as the model input, so that\n",
    "        # when we run the session, this will be the activations drawn from the appropriate layer, with G as input.\n",
    "        a_G = out\n",
    "        \n",
    "        # Compute style_cost for the current layer\n",
    "        J_style_layer = compute_layer_style_cost(a_S, a_G)\n",
    "\n",
    "        # Add coeff * J_style_layer of this layer to overall style cost\n",
    "        J_style += coeff * J_style_layer\n",
    "\n",
    "    return J_style"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining the total cost\n",
    "\n",
    "$$J(G) = \\alpha J_{content}(C,G) + \\beta J_{style}(S,G)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def total_cost(J_content, J_style, alpha = 10, beta = 40):\n",
    "    \"\"\"\n",
    "    Computes the total cost function\n",
    "    \n",
    "    Arguments:\n",
    "    J_content -- content cost coded above\n",
    "    J_style -- style cost coded above\n",
    "    alpha -- hyperparameter weighting the importance of the content cost\n",
    "    beta -- hyperparameter weighting the importance of the style cost\n",
    "    \n",
    "    Returns:\n",
    "    J -- total cost as defined by the formula above.\n",
    "    \"\"\"\n",
    "    \n",
    "    J = alpha*J_content + beta*J_style\n",
    "    \n",
    "    return J"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "with tf.Session() as test:\n",
    "    np.random.seed(3)\n",
    "    J_content = np.random.randn()    \n",
    "    J_style = np.random.randn()\n",
    "    J = total_cost(J_content, J_style)\n",
    "    print(\"J = \" + str(J))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Solving the optimization problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start the interactive session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reset the graph\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# Start interactive session\n",
    "sess = tf.InteractiveSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the content and style image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "content_image = scipy.misc.imread(\"images/louvre_small.jpg\")\n",
    "content_image = reshape_and_normalize_image(content_image)\n",
    "\n",
    "style_image = scipy.misc.imread(\"images/monet.jpg\")\n",
    "style_image = reshape_and_normalize_image(style_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we initialize the \"generated\" image as a noisy image created from the content_image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "generated_image = generate_noise_image(content_image)\n",
    "imshow(generated_image[0]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load pre-trained VGG19 model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = load_vgg_model(\"pretrained-model/imagenet-vgg-verydeep-19.mat\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Content Cost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assign the content image to be the input of the VGG model.  \n",
    "sess.run(model['input'].assign(content_image))\n",
    "\n",
    "# Select the output tensor of layer conv4_2\n",
    "out = model['conv4_2']\n",
    "\n",
    "# Set a_C to be the hidden layer activation from the layer we have selected\n",
    "a_C = sess.run(out)\n",
    "\n",
    "# Set a_G to be the hidden layer activation from same layer. Here, a_G references model['conv4_2'] \n",
    "# and isn't evaluated yet. Later in the code, we'll assign the image G as the model input, so that\n",
    "# when we run the session, this will be the activations drawn from the appropriate layer, with G as input.\n",
    "a_G = out\n",
    "\n",
    "# Compute the content cost\n",
    "J_content = compute_content_cost(a_C, a_G)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Style Cost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assign the input of the model to be the \"style\" image \n",
    "sess.run(model['input'].assign(style_image))\n",
    "\n",
    "# Compute the style cost\n",
    "J_style = compute_style_cost(model, STYLE_LAYERS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Total Cost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "J = total_cost(J_content, J_style, 10, 40)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimizer\n",
    "\n",
    "* Use the Adam optimizer to minimize the total cost `J`.\n",
    "* Use a learning rate of 2.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define optimizer\n",
    "optimizer = tf.train.AdamOptimizer(2.0)\n",
    "\n",
    "# define train_step\n",
    "train_step = optimizer.minimize(J)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We now implement the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def model_nn(sess, input_image, num_iterations = 200):\n",
    "    \n",
    "    # Initialize global variables (you need to run the session on the initializer)\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    # Run the noisy input image (initial generated image) through the model. Use assign().\n",
    "    sess.run(model[\"input\"].assign(input_image))\n",
    "    \n",
    "    for i in range(num_iterations):\n",
    "    \n",
    "        # Run the session on the train_step to minimize the total cost\n",
    "        sess.run(train_step)\n",
    "        \n",
    "        # Compute the generated image by running the session on the current model['input']\n",
    "        generated_image = sess.run(model[\"input\"])\n",
    "\n",
    "        # Print every 20 iteration.\n",
    "        if i%20 == 0:\n",
    "            Jt, Jc, Js = sess.run([J, J_content, J_style])\n",
    "            print(\"Iteration \" + str(i) + \" :\")\n",
    "            print(\"total cost = \" + str(Jt))\n",
    "            print(\"content cost = \" + str(Jc))\n",
    "            print(\"style cost = \" + str(Js))\n",
    "            \n",
    "            # save current generated image in the \"/output\" directory\n",
    "            save_image(\"output/\" + str(i) + \".png\", generated_image)\n",
    "    \n",
    "    # save last generated image\n",
    "    save_image('output/generated_image.jpg', generated_image)\n",
    "    \n",
    "    return generated_image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now run to generate an artistic image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_nn(sess, generated_image)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}