ch01-tensorflow-introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](img/neural_network.png)\n",
    "\n",
    "### Problem\n",
    "\n",
    "##### Given an input signal (image, sound, text etc), predict some class .  (supervised learning)\n",
    "-  There are lots of algorithms that can help you with that.\n",
    "-  Neural networks are state of the art in many interesting areas\n",
    "-  __[Click here](https://playground.tensorflow.org/)__\n",
    "-  An accessible \"mathy\" introduction (really really cool) https://www.youtube.com/watch?v=aircAruvnKk\n",
    "***\n",
    "\n",
    "### How does it work\n",
    "\n",
    "-  ##### Neural Network Inference\n",
    ">-  Propagate input signal (i.e. images with cats) through layers of neurons to output (usually) some class.\n",
    ">-  Each neuron computes a weighted sum of the neurons in the previous layer\n",
    ">-  A neuron is \"activated\" if the sum passes a thresholdand, if the sum passes a threshold, it is activated.\n",
    "\n",
    "-  ##### Neural Network Training:\n",
    ">-  Compute \"error\" function (i.e. the difference between what is predicted and the truth).\n",
    ">-  Evaluate the influence of each parameter on the error function\n",
    ">-  Make a small update on the weights so as to minimize the error function.\n",
    "***\n",
    "-  ##### The math under the hood:\n",
    ">-  Computing the input in each neuron amounts to a matrix multiplication\n",
    ">-  \"Activating\" the neuron amounts to applying a non-linear function on the neuron's input\n",
    ">-  Evaluating the influence of each parameter on the error amounts to computing the derivative of the error function with respect to each parameter\n",
    ">-  ##### Tensorflow helps with all these and more.\n",
    "***\n",
    "### Steps:\n",
    "\n",
    "    1. Create a data-flow graph.\n",
    "    2. Create a session.\n",
    "    3. Access the graph through the session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#The usual imports\n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28) (60000, 10) (10000, 28, 28) (10000, 10)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAADgpJREFUeJzt3X+MVfWZx/HPs1j+kKI4aQRCYSnEYJW4082IjSWrxkzVDQZHrekkJjQapn8wiU02ZA3/VNNgyCrslmiamaZYSFpKE3VB0iw0otLGZuKIWC0srTFsO3IDNTjywx9kmGf/mEMzxbnfe+fec++5zPN+JeT+eM6558kNnznn3O+592vuLgDx/EPRDQAoBuEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxDUZc3cmJlxOSHQYO5u1SxX157fzO40syNm9q6ZPVrPawFoLqv12n4zmybpj5I6JQ1Jel1St7sfSqzDnh9osGbs+ZdJetfd33P3c5J+IWllHa8HoInqCf88SX8Z93goe+7vmFmPmQ2a2WAd2wKQs3o+8Jvo0OJzh/Xu3i+pX+KwH2gl9ez5hyTNH/f4y5KO1dcOgGapJ/yvS7rGzL5iZtMlfVvSrnzaAtBoNR/2u/uImfVK2iNpmqQt7v6H3DoD0FA1D/XVtDHO+YGGa8pFPgAuXYQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8EVfMU3ZJkZkclnZZ0XtKIu3fk0RTyM23atGT9yiuvbOj2e3t7y9Yuv/zy5LpLlixJ1tesWZOsP/XUU2Vr3d3dyXU//fTTZH3Dhg3J+uOPP56st4K6wp+5zd0/yOF1ADQRh/1AUPWG3yXtNbM3zKwnj4YANEe9h/3fcPdjZna1pF+b2f+6+/7xC2R/FPjDALSYuvb87n4suz0h6QVJyyZYpt/dO/gwEGgtNYffzGaY2cwL9yV9U9I7eTUGoLHqOeyfLekFM7vwOj939//JpSsADVdz+N39PUn/lGMvU9aCBQuS9enTpyfrN998c7K+fPnysrVZs2Yl173vvvuS9SINDQ0l65s3b07Wu7q6ytZOnz6dXPett95K1l999dVk/VLAUB8QFOEHgiL8QFCEHwiK8ANBEX4gKHP35m3MrHkba6L29vZkfd++fcl6o79W26pGR0eT9YceeihZP3PmTM3bLpVKyfqHH36YrB85cqTmbTeau1s1y7HnB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgGOfPQVtbW7I+MDCQrC9atCjPdnJVqffh4eFk/bbbbitbO3fuXHLdqNc/1ItxfgBJhB8IivADQRF+ICjCDwRF+IGgCD8QVB6z9IZ38uTJZH3t2rXJ+ooVK5L1N998M1mv9BPWKQcPHkzWOzs7k/WzZ88m69dff33Z2iOPPJJcF43Fnh8IivADQRF+ICjCDwRF+IGgCD8QFOEHgqr4fX4z2yJphaQT7r40e65N0g5JCyUdlfSAu6d/6FxT9/v89briiiuS9UrTSff19ZWtPfzww8l1H3zwwWR9+/btyTpaT57f5/+ppDsveu5RSS+5+zWSXsoeA7iEVAy/u++XdPElbCslbc3ub5V0T859AWiwWs/5Z7t7SZKy26vzawlAMzT82n4z65HU0+jtAJicWvf8x81sriRltyfKLeju/e7e4e4dNW4LQAPUGv5dklZl91dJ2plPOwCapWL4zWy7pN9JWmJmQ2b2sKQNkjrN7E+SOrPHAC4hFc/53b27TOn2nHsJ69SpU3Wt/9FHH9W87urVq5P1HTt2JOujo6M1bxvF4go/ICjCDwRF+IGgCD8QFOEHgiL8QFBM0T0FzJgxo2ztxRdfTK57yy23JOt33XVXsr53795kHc3HFN0Akgg/EBThB4Ii/EBQhB8IivADQRF+ICjG+ae4xYsXJ+sHDhxI1oeHh5P1l19+OVkfHBwsW3vmmWeS6zbz/+ZUwjg/gCTCDwRF+IGgCD8QFOEHgiL8QFCEHwiKcf7gurq6kvVnn302WZ85c2bN2163bl2yvm3btmS9VCrVvO2pjHF+AEmEHwiK8ANBEX4gKMIPBEX4gaAIPxBUxXF+M9siaYWkE+6+NHvuMUmrJf01W2ydu/+q4sYY57/kLF26NFnftGlTsn777bXP5N7X15esr1+/Pll///33a972pSzPcf6fSrpzguf/093bs38Vgw+gtVQMv7vvl3SyCb0AaKJ6zvl7zez3ZrbFzK7KrSMATVFr+H8kabGkdkklSRvLLWhmPWY2aGblf8wNQNPVFH53P+7u5919VNKPJS1LLNvv7h3u3lFrkwDyV1P4zWzuuIddkt7Jpx0AzXJZpQXMbLukWyV9ycyGJH1f0q1m1i7JJR2V9N0G9gigAfg+P+oya9asZP3uu+8uW6v0WwFm6eHqffv2JeudnZ3J+lTF9/kBJBF+ICjCDwRF+IGgCD8QFOEHgmKoD4X57LPPkvXLLktfhjIyMpKs33HHHWVrr7zySnLdSxlDfQCSCD8QFOEHgiL8QFCEHwiK8ANBEX4gqIrf50dsN9xwQ7J+//33J+s33nhj2VqlcfxKDh06lKzv37+/rtef6tjzA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQjPNPcUuWLEnWe3t7k/V77703WZ8zZ86ke6rW+fPnk/VSqZSsj46O5tnOlMOeHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCqjjOb2bzJW2TNEfSqKR+d/+hmbVJ2iFpoaSjkh5w9w8b12pclcbSu7u7y9YqjeMvXLiwlpZyMTg4mKyvX78+Wd+1a1ee7YRTzZ5/RNK/uftXJX1d0hozu07So5JecvdrJL2UPQZwiagYfncvufuB7P5pSYclzZO0UtLWbLGtku5pVJMA8jepc34zWyjpa5IGJM1295I09gdC0tV5Nwegcaq+tt/MvijpOUnfc/dTZlVNByYz65HUU1t7ABqlqj2/mX1BY8H/mbs/nz193MzmZvW5kk5MtK6797t7h7t35NEwgHxUDL+N7eJ/Iumwu28aV9olaVV2f5Wknfm3B6BRKk7RbWbLJf1G0tsaG+qTpHUaO+//paQFkv4s6VvufrLCa4Wconv27NnJ+nXXXZesP/3008n6tddeO+me8jIwMJCsP/nkk2VrO3em9xd8Jbc21U7RXfGc391/K6nci90+maYAtA6u8AOCIvxAUIQfCIrwA0ERfiAowg8ExU93V6mtra1sra+vL7lue3t7sr5o0aKaesrDa6+9lqxv3LgxWd+zZ0+y/sknn0y6JzQHe34gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCCrMOP9NN92UrK9duzZZX7ZsWdnavHnzauopLx9//HHZ2ubNm5PrPvHEE8n62bNna+oJrY89PxAU4QeCIvxAUIQfCIrwA0ERfiAowg8EFWacv6urq656PQ4dOpSs7969O1kfGRlJ1lPfuR8eHk6ui7jY8wNBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUObu6QXM5kvaJmmOpFFJ/e7+QzN7TNJqSX/NFl3n7r+q8FrpjQGom7tbNctVE/65kua6+wEzmynpDUn3SHpA0hl3f6rapgg/0HjVhr/iFX7uXpJUyu6fNrPDkor96RoAdZvUOb+ZLZT0NUkD2VO9ZvZ7M9tiZleVWafHzAbNbLCuTgHkquJh/98WNPuipFclrXf3581stqQPJLmkH2js1OChCq/BYT/QYLmd80uSmX1B0m5Je9x90wT1hZJ2u/vSCq9D+IEGqzb8FQ/7zcwk/UTS4fHBzz4IvKBL0juTbRJAcar5tH+5pN9IeltjQ32StE5St6R2jR32H5X03ezDwdRrsecHGizXw/68EH6g8XI77AcwNRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCavYU3R9I+r9xj7+UPdeKWrW3Vu1Lorda5dnbP1a7YFO/z/+5jZsNuntHYQ0ktGpvrdqXRG+1Kqo3DvuBoAg/EFTR4e8vePsprdpbq/Yl0VutCumt0HN+AMUpes8PoCCFhN/M7jSzI2b2rpk9WkQP5ZjZUTN728wOFj3FWDYN2gkze2fcc21m9msz+1N2O+E0aQX19piZvZ+9dwfN7F8L6m2+mb1sZofN7A9m9kj2fKHvXaKvQt63ph/2m9k0SX+U1ClpSNLrkrrd/VBTGynDzI5K6nD3wseEzexfJJ2RtO3CbEhm9h+STrr7huwP51Xu/u8t0ttjmuTMzQ3qrdzM0t9Rge9dnjNe56GIPf8ySe+6+3vufk7SLyStLKCPlufu+yWdvOjplZK2Zve3auw/T9OV6a0luHvJ3Q9k909LujCzdKHvXaKvQhQR/nmS/jLu8ZBaa8pvl7TXzN4ws56im5nA7AszI2W3Vxfcz8UqztzcTBfNLN0y710tM17nrYjwTzSbSCsNOXzD3f9Z0l2S1mSHt6jOjyQt1tg0biVJG4tsJptZ+jlJ33P3U0X2Mt4EfRXyvhUR/iFJ88c9/rKkYwX0MSF3P5bdnpD0gsZOU1rJ8QuTpGa3Jwru52/c/bi7n3f3UUk/VoHvXTaz9HOSfubuz2dPF/7eTdRXUe9bEeF/XdI1ZvYVM5su6duSdhXQx+eY2YzsgxiZ2QxJ31TrzT68S9Kq7P4qSTsL7OXvtMrMzeVmllbB712rzXhdyEU+2VDGf0maJmmLu69vehMTMLNFGtvbS2PfePx5kb2Z2XZJt2rsW1/HJX1f0n9L+qWkBZL+LOlb7t70D97K9HarJjlzc4N6Kzez9IAKfO/ynPE6l364wg+IiSv8gKAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8E9f/Ex0YKZYOZcwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def transform_labels(numerical_labels):\n",
    "    #use one hot encoding of labels instead of numeric\n",
    "    #e.g. 2 will become [0,0,1,0,0,0,...]\n",
    "    categorical = np.zeros((len(numerical_labels),10))\n",
    "    for idx,label in enumerate(numerical_labels):\n",
    "        categorical[idx][label] = 1\n",
    "    return categorical\n",
    "\n",
    "#Load mnist (the \"hello world\" of machine learning)\n",
    "\n",
    "(train_img,train_label),(test_img,test_label) = tf.keras.datasets.mnist.load_data()\n",
    "plt.imshow(train_img[0],cmap='gray')\n",
    "\n",
    "train_label = transform_labels(train_label)\n",
    "test_label  = transform_labels(test_label)\n",
    "print (train_img.shape,train_label.shape,test_img.shape,test_label.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the dataflow "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "#Inputs in the dataflow graph\n",
    "image_input = tf.placeholder(tf.float32,shape=(None,28,28))\n",
    "label_input = tf.placeholder(tf.int16,shape=(None,10))\n",
    "\n",
    "#Define parameters; initially they are random; they will be learned\n",
    "weights = tf.get_variable(\"weights\",\n",
    "                          shape=[28*28,10],\n",
    "                          dtype=tf.float32,\n",
    "                          initializer=tf.random_normal_initializer(),\n",
    "                         )\n",
    "\n",
    "bias    = tf.get_variable(\"bias\",\n",
    "                          shape=[1],\n",
    "                          dtype=tf.float32,\n",
    "                          initializer=tf.constant_initializer(0))\n",
    "\n",
    "#Transform the input matrix from a 2d array into a 1d array\n",
    "#Multiply the reshaped image with the weights and add the bias\n",
    "\n",
    "predicted_logits = tf.matmul(tf.reshape(image_input,shape=(-1,28*28)),weights)+bias\n",
    "\n",
    "#Compute the error\n",
    "#Transform the output into probabilities and compare the obtained distribution with the desired distribution\n",
    "#https://deepnotes.io/softmax-crossentropy for the curious\n",
    "\n",
    "error = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits=predicted_logits,labels=label_input))\n",
    "\n",
    "#Define the optimizer; this is the thing that actually minimizez the error;\n",
    "#It looks at all the variables that contribute to the error and adjusts them \n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=1e-3).minimize(error)\n",
    "\n",
    "#Record the error for tensorboard\n",
    "#During training we run \"tensorboard --logdir=/path/to/logs\"\"\n",
    "\n",
    "error_summary = tf.summary.scalar('average_error', error) \n",
    "#We need to ru this in order to initialize the variables\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "#At this point the blueprint is created. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Perform the training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial number of errors: 8768\n",
      "Number  of errors after epoch 0 is: 1359\n",
      "Number  of errors after epoch 1 is: 1262\n",
      "Number  of errors after epoch 2 is: 1163\n",
      "Number  of errors after epoch 3 is: 1166\n",
      "Number  of errors after epoch 4 is: 1209\n",
      "Number  of errors after epoch 5 is: 1134\n",
      "Number  of errors after epoch 6 is: 1062\n",
      "Number  of errors after epoch 7 is: 1164\n",
      "Number  of errors after epoch 8 is: 1185\n",
      "Number  of errors after epoch 9 is: 1066\n"
     ]
    }
   ],
   "source": [
    "#How many passes through the data should we do\n",
    "num_epochs = 10\n",
    "#How many images at once\n",
    "batch_size  = 32\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    #Initialize the weights and bias\n",
    "    sess.run(init)\n",
    "    #Define the model saver. We will subsequently load the model in order to use it.\n",
    "    saver = tf.train.Saver()\n",
    "    \n",
    "    #Define the actual tensorboard logger\n",
    "    summary_writter = tf.summary.FileWriter('./mnist_training_summary',sess.graph)\n",
    "\n",
    "    #At each passing you want to feed the data in a different order\n",
    "    read_order = np.arange(len(train_img))\n",
    "    batch_idx = 0\n",
    "    \n",
    "    #Let's see what we get when we just use the random weights\n",
    "    logits,test_loss = sess.run([predicted_logits,error],feed_dict={\n",
    "                image_input:test_img,\n",
    "                label_input:test_label\n",
    "            })\n",
    "    print(\"Initial number of errors: {}\".format(np.count_nonzero(np.argmax(logits,axis=1)-np.argmax(test_label,axis=1))))\n",
    "    \n",
    "    for epoch in range(num_epochs):\n",
    "        \n",
    "        #Feed images, get predictions, apply gradients - all just by running \"optimizer\" operation\n",
    "        while batch_idx+batch_size<len(train_img): \n",
    "            _,summary = sess.run([optimizer,error_summary],feed_dict={\n",
    "                image_input:train_img[read_order[batch_idx:batch_idx+batch_size]],\n",
    "                label_input:train_label[read_order[batch_idx:batch_idx+batch_size]]\n",
    "            })\n",
    "            batch_idx+=batch_size\n",
    "            \n",
    "            #Log the error at the current training step in order to be visualised in tensorboard\n",
    "            training_step = len(train_img)/batch_size*epoch+batch_idx/batch_size\n",
    "            summary_writter.add_summary(summary,training_step) \n",
    "        \n",
    "        #Test the model after each epoch\n",
    "        logits,test_loss = sess.run([predicted_logits,error],feed_dict={\n",
    "                image_input:test_img,\n",
    "                label_input:test_label\n",
    "            })\n",
    "        \n",
    "        print(\"Number  of errors after epoch {} is: {}\".format(epoch,np.count_nonzero(np.argmax(logits,axis=1)-np.argmax(test_label,axis=1))))\n",
    "        \n",
    "        #Save the model after each epoch\n",
    "        saver.save(sess, './mnist_model',global_step=epoch)\n",
    "        \n",
    "        #Prepare for the next epoch; need to shuffle the data and reset the index\n",
    "        np.random.shuffle(read_order)\n",
    "        batch_idx=0\n",
    "        #For cooler ways to handle the data google \"tf.data\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Restore the saved model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./mnist_model-9\n",
      "Final number of errors is: 1066\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    saver = tf.train.import_meta_graph(\"./mnist_model-9.meta\")\n",
    "    saver.restore(sess,tf.train.latest_checkpoint('./'))\n",
    "    logits,test_loss = sess.run([predicted_logits,error],feed_dict={\n",
    "                image_input:test_img,\n",
    "                label_input:test_label\n",
    "            })\n",
    "        \n",
    "    print(\"Final number of errors is: {}\".format(np.count_nonzero(np.argmax(logits,axis=1)-np.argmax(test_label,axis=1))))\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises\n",
    "\n",
    "##### Modify the above code to:\n",
    "1. Actually create a neural network: add 1 hidden layer with 512 neurons. Use sigmoid activation.\n",
    "2. Change the activation function to relu\n",
    "3. Add l2 regularization.\n",
    "4. Trace each component of the error and the test accuracy (in %) in tensorboard. \n",
    "5. Add dropout\n",
    "6. Go deeper with the layers\n",
    "7. And deeper\n",
    "8. ... you get the idea\n",
    "\n",
    "##### Make sure NOT to accidentaly create Skynet!\n",
    "For the next lab:\n",
    "\n",
    "9. Shuffle the pixels of the images (in the same way). Train again.\n",
    "10. Train on cifar10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:pytorch]",
   "language": "python",
   "name": "conda-env-pytorch-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}