unit06_svm/lab06_ext_mnist_partial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 6:  SVMs on an Extended  MNIST \n",
    "\n",
    "In addition to the concepts in the [MNIST demo](mnist_svm.ipynb), you will learn:\n",
    "\n",
    "* Use the `skimage` module for some basic pre-processing of images in machine learning\n",
    "* Run and test an SVM classifier on a dataset you have created\n",
    "* Perform error handling in python\n",
    "\n",
    "In the [MNIST demo](mnist_svm.ipynb), we saw how SVMs can be used for the classic MNIST problem of digit recognition.  In this lab, we are going to extend the MNIST dataset by adding a number of non-digit letters and see if the classifier can distinguish the digits from the non-digits.  All non-digits will be lumped as a single 11-th class.  In image processing, this is called a 'detection' as opposed to 'classification' problem.  Detection is vital in OCR and related problems since the non useful characters must be rejected.  For this lab we will create a very simple version of this problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the MNIST data\n",
    "\n",
    "We first import the standard modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "from sklearn import linear_model, preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, fetch the digits with `fetch_mldata` command as shown in the demo.  Save the digits data matrix and labels to variables `Xdig` and `ydig`.  Also, recall that the pixel values in `Xdig` are between 0 and 255.  Create a scaled version of `Xdig` called `Xdigs` where the components are between -1 and 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO\n",
    "# Xdig = ...  Load MNIST data\n",
    "# ydig = ...\n",
    "# Xdigs = ...  Rescale MNIST data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a function `plt_digit` that plots the digits.  You can use the code from the demo.  Test the function by plotting four random digits.  Use the `plt.title` command to print the numeric label in `ydig` above each digit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Select four random digits and plot them using the subplot command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exception Handling\n",
    "\n",
    "In the routines we will develop below, we will need to handle error conditions, called exceptions. A very nice description of how to perform exception handling in python is given in\n",
    "\n",
    "https://docs.python.org/3/tutorial/errors.html\n",
    "\n",
    "As described there, errors are described by a class that derives from a base class Exception. When the error occurs, the program raises the exception with the raise command. The calling function can catch the exception with the try ... except control flow. We will define our exception as follows which has an optional string argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class ImgException(Exception):\n",
    "    def __init__(self, msg='No msg'):\n",
    "        self.msg = msg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exceptions are used as follows:  First, when there is an error in some function, you `raise` the exception as follows:\n",
    "\n",
    "    foo():\n",
    "        ...\n",
    "        if (error):\n",
    "           raise ImgException(\"File not found\")\n",
    "           \n",
    "        # Code that will not execute if the error condition occured\n",
    "        \n",
    "     \n",
    "The function that calls `foo()` can catch the error using the following syntax:\n",
    "\n",
    "    try: \n",
    "        foo()\n",
    "        \n",
    "        # Continue processing in case when there was no exception\n",
    "        ....\n",
    "        \n",
    "    except ImgException as e:\n",
    "        print(\"foo() didn't work\")\n",
    "        print(\"Error msg = %s\" % e.msg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Non-Digit Characters\n",
    "\n",
    "We will now build a set of non-digit characters.  As a simple source, we will get hand-written lowercase letters 'a' to 'z' and process them with the `skimage` package.  The `skimage` module is a very powerful package that has a similar interface as OpenCV.  We first import the relevant modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.image as mpimg\n",
    "import skimage.io\n",
    "from skimage.filters import threshold_otsu\n",
    "from skimage.segmentation import clear_border\n",
    "from skimage.measure import label, regionprops\n",
    "from skimage.morphology import closing, square\n",
    "from skimage.color import label2rgb\n",
    "from skimage.transform import resize\n",
    "import matplotlib.patches as mpatches\n",
    "from skimage import data\n",
    "import skimage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get a set of character images from a very nice website\n",
    "\n",
    "http://www.ee.surrey.ac.uk/CVSSP/demos/chars74k/\n",
    "\n",
    "Go to this website, and download the file `EnglishHnd.tgz`.  After you untar this file, there are a large number of `.png` files in the directory:\n",
    "\n",
    "    EnglishHnd\\English\\Hnd\\Img\n",
    "    \n",
    "Each directory has about 55 samples of hand-written letters and numbers.  After you have downloaded this file, complete the function `load_img` to load an image from a character and sample index.\n",
    "\n",
    "Alternatively, the files are available on Google Drive:\n",
    "\n",
    "https://drive.google.com/file/d/0BxOz-SM9a1h4UksxSXBjQ0dabUk/view?usp=sharing \n",
    "\n",
    "You can download and unzip the file.\n",
    "\n",
    "The code at the end will test the function to see if it working correctly.  For one sample, it should print the image and a second it should say the file was not found."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os.path\n",
    "\n",
    "def load_img(char_ind, samp_ind):\n",
    "    \"\"\"\n",
    "    Returns the image from the dataset given a character and sample index.\n",
    "    \n",
    "        \n",
    "    If the file doesn't exist, it raises an Exception with the filename.   \n",
    "    \"\"\" \n",
    "    \n",
    "    # TODO:  Set the file name based on char_ind and samp_ind\n",
    "    # fname = ...\n",
    "    \n",
    "    # TODO:  Use the os.path.isfile command to check if the file exists.  \n",
    "    # If not raise an ImgException with the message \"[fname] not found\"\n",
    "\n",
    "    # TODO:  Use the skimage.io.imread() command to read the png file and return the image.\n",
    "    # img = ...\n",
    "    return img\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test the `load_img` function.  This should:\n",
    "* Plot the image in `Sample047\\img047-006.png`\n",
    "* Say that the `Sample047\\img047-070.png` is not found."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "char_ind = 47\n",
    "samp_inds = [6,70]\n",
    "for samp_ind in samp_inds:\n",
    "    try:\n",
    "        img = load_img(char_ind=char_ind, samp_ind=samp_ind)\n",
    "        print(\"Char = %d samp=%d\" % (char_ind, samp_ind))\n",
    "        plt.imshow(img)\n",
    "    except ImgException as e:\n",
    "        print(e.msg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The images in the sample directory have very high resolution.  Complete the following method to find the image and place it in a 28 x 28 box.  You can look at this very nice demo of the `skimage` methods here:\n",
    "\n",
    "http://scikit-image.org/docs/dev/auto_examples/segmentation/plot_label.html\n",
    "\n",
    "The code is somewhat complex, so I have provided some of the steps, esp. for the thresholding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def mnist_resize(img):\n",
    "    \"\"\"\n",
    "    Extracts a character from the image, and places in a 28x28 image to match the MNIST format.\n",
    "    \n",
    "    Returns:\n",
    "    img1:  MNIST formatted 28 x 28 size image with the character from img\n",
    "    box:   A bounding box indicating the locations where the character was found in img.    \n",
    "    \"\"\"\n",
    "    # Image sizes (fixed for now).  To match the MNIST data, the image \n",
    "    # will be first resized to 20 x 20.  Then, the image will be placed in center of 28 x 28 box\n",
    "    # offet by 4 on each side.\n",
    "    nx_img = 20   \n",
    "    ny_img = 20\n",
    "    nx_box = 28   \n",
    "    ny_box = 28\n",
    "    offx = 4\n",
    "    offy = 4\n",
    "    \n",
    "    # TODO:  Convert the image to gray scale using the skimage.color.rgb2gray method.\n",
    "    # bw = ...\n",
    "    \n",
    "    # Threshold the image using OTSU threshold\n",
    "    thresh = threshold_otsu(bw)\n",
    "    bw = closing(bw < thresh, square(3)).astype(int)\n",
    "    \n",
    "    # Get the regions in the image.\n",
    "    # This creates a list of regions in the image where the digit possibly is.\n",
    "    regions = regionprops(bw)\n",
    "\n",
    "    # TODO:  Find region with the largest area.  You can get the region area from region.area.\n",
    "    # region_max = ...\n",
    "     \n",
    "    # Raise an ImgException if no region with area >= 100 was found\n",
    "    if (area_max < 100):\n",
    "        raise ImgException(\"No image found\")    \n",
    "                \n",
    "    # Get the bounding box of the character from region_max.bbox\n",
    "    minr, minc, maxr, maxc = region_max.bbox\n",
    "    box = [minr,minc,maxr,maxc]\n",
    "    \n",
    "    # TODO:  Crop the image in bw to the bounding box\n",
    "    # bw_crop = bw[...]\n",
    "        \n",
    "    # TODO:  Resize the cropped image to a 20x20 using the resize command.\n",
    "    # You will need to use the mode = 'constant' option\n",
    "    # bw_resize = ...\n",
    "    \n",
    "    # TODO:  Threshold back to a 0-1 image by comparing the pixels to their mean value\n",
    "    \n",
    "    # TODO:  Place extracted 20 x 20 image in larger image 28 x 28\n",
    "    # img1 = ...\n",
    "    return img1, box"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now test the `mnist_resize` program by completing the following code.  Create two subplots:\n",
    "* subplot(1,2,1):  The original image with the bounding box for the character that was found in the image.\n",
    "* subplot(1,2,2):  The MNIST resized image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load an image\n",
    "img = load_img(13,9)\n",
    "\n",
    "try:\n",
    "    # Resize the image\n",
    "    # img1, box = mnist_resize(img)\n",
    "    \n",
    "    # TODO:  Plot the original image, img, along with a red box around the captured character.\n",
    "    # Use the mpatches.Rectangle and ax.add_patch methods to construct the rectangle.\n",
    "   \n",
    "    # TODO:  Plot the resized 28 x 28 image, img1.  You can use the plt_digit(img1) command \n",
    "       \n",
    "except ImgException as e:\n",
    "    print(e.msg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, run the command `nlet=1000` times to get 1000 letter images.  In each iteration, select a random image from a lowercase letter and add it to a matrix `Xlet`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "images captured = 50\n",
      "images captured = 100\n",
      "images captured = 150\n",
      "images captured = 200\n",
      "images captured = 250\n",
      "images captured = 300\n",
      "images captured = 350\n",
      "images captured = 400\n",
      "images captured = 450\n",
      "images captured = 500\n",
      "images captured = 550\n",
      "images captured = 600\n",
      "images captured = 650\n",
      "images captured = 700\n",
      "images captured = 750\n",
      "images captured = 800\n",
      "images captured = 850\n",
      "images captured = 900\n",
      "images captured = 950\n",
      "images captured = 1000\n"
     ]
    }
   ],
   "source": [
    "# Dimensions\n",
    "nlet = 1000\n",
    "nrow = 28\n",
    "ncol = 28\n",
    "npix = nrow*ncol\n",
    "Xlet = np.zeros((nlet, npix))\n",
    "\n",
    "i = 0\n",
    "while i < nlet:\n",
    "    # TODO:  Generate a random character and sample    \n",
    "    # char_ind = random number corresponding to a lowercase letter except 'O' and 'I'\n",
    "    # samp_ind = random number from 0 to 49\n",
    "  \n",
    "        \n",
    "    try:\n",
    "        # TODO:  Load the image with load_img function\n",
    "        # img = ...\n",
    "        \n",
    "        # TODO:  Reize the image with mnist_resize function\n",
    "        # img1, box = ...\n",
    "        \n",
    "        # TODO:  Store the image in a row of Xlet[i,:] and increment i\n",
    "        i += 1\n",
    "        \n",
    "        # Print progress\n",
    "        if (i % 50 == 0):\n",
    "            print ('images captured = {0:d}'.format(i))\n",
    "    except ImgException:\n",
    "        # Skip if image loading or resizing failed\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since this takes a long time to generate, save the matrix `Xlet` to a file `Xlet.p` using the `pickle.dump` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "# TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reload the data `Xlet` from the file `Xlet.p`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Extended Training Data\n",
    "\n",
    "Now, create an extended data set by combining `ndig=5000` randomly selected digit samples and `nlet=1000` letters. \n",
    "* Select `ndig=5000` random samples from `Xdigs` and their labels in `ydig`.\n",
    "* Rescale the letters `Xlet` to a new matrix `Xlets = 2*Xlet-1` to make the pixel values go from -1 to 1.\n",
    "* Use the `np.vstack` command to create a 6000 element alpha-numeric data set `X`\n",
    "* Create a corresponding label vector `y` where all the non-digit characters are labeled with a non-digit label, `letter_lbl=10`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO\n",
    "# X = ...  Array with 6000 characters (5000 digits + 1000 letters)\n",
    "# y = ...  Array with 6000 labels (0-9 for the digits, 10 = non-digit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run the SVM classifier\n",
    "\n",
    "First create the SVM classifer.  Use an \"rbf\" classifier with `C=2.8` and `gamma=.0073`.  Not sure if these are the best parameters, you could try to search for better ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import svm\n",
    "\n",
    "# TODO:  Create a classifier: a support vector classifier\n",
    "# svc = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get 5000 training samples `Xtr,ytr` and 1000 test samples `Xts,yts`.  Remember to randomly select them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO\n",
    "# Xtr = ...\n",
    "# ytr = ...\n",
    "# Xts = ...\n",
    "# yts = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `svc.fit` command to fit on the training data.  This may take a few minutes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Measure the accuracy on the test samples.  You should get about 96% accuracy.  You can get better by using more training samples, but it will just take longer to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Print the normalized confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO\n",
    "from sklearn.metrics import confusion_matrix\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting some error samples\n",
    "\n",
    "We now plot some errors.  Plot up to four images where yhat == 10 but yts != 10.  That is, the true image was a digit, but the classifier classified it as a non-digit.  Note there may be less than four such errors (when I ran it I got only three such errors).  In that case, just plot only the errors you got.  If there are no errors, print \"No such error found\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now plot up to four images where yhat != 10, but yts == 10.  That is, the image was a non-digit, but the classifier thought it was an image.  I happened to get no such images.  If you find no such examples, print \"No such error found\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, plot up to four images where yts != yhat and both yts < 10 and yhat < 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}