Spark_Spam_Message.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Employing pySpark to classify emails and detect spam"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Alexandros Dimitrios Nalmpantis; Georgios Kyriakopoulos (2017)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "_**Abstract:**_ _This notebook presents pySpark code that processes and clasifies text data from a corpus of emails. The code implements  data wrangling and classification techniques (i.e. logistic regression analysis) to build a process that recognises whether a certain email is spam or not. Alternative specifications of the classification technique used are explored to assess the performance and efficiency of the process._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "* We start with importing the  modules that the analysis will use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import re\n",
    "from operator import add\n",
    "from pyspark.mllib.regression import LabeledPoint\n",
    "from pyspark.mllib.classification import LogisticRegressionWithLBFGS\n",
    "from pyspark.mllib.util import MLUtils\n",
    "from math import log\n",
    "import time\n",
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "* We define directory paths, which include the corpus of text files that the analysis will use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "prefix = '/data/'\n",
    "dirPath = prefix + 'spam/bare/part1'\n",
    "dirPath_2 = prefix + 'spam/bare/part10'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* We create two functions. Function *__splitFileWords()__* creates (file, word) tuples (words being in lowercase). Using this function, a second function *__read_file_word_RDD()__* uploads the text files and generates information, nemaley, the count and the location of the files used as well as descriptive statistics (histogram) of word counts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part1/3-1msg1.txt',\n",
      "  'subject'),\n",
      " ('hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part1/3-1msg1.txt', 're')]\n"
     ]
    }
   ],
   "source": [
    "def splitFileWords(file_text): # function (a) builds (file, word) tuples from (file, text) tuples\n",
    "    f,t = file_text # define the input to the function\n",
    "    file_word_List = [] # create an empty (file,word) list\n",
    "    word_List = re.split('\\W+',t) # split texts into words using regular expression\n",
    "    for w in word_List: \n",
    "        file_word_List.append((f,w.lower())) # append words in lowercase to their corresponding file\n",
    "    return file_word_List\n",
    "\n",
    "def read_file_word_RDD(argDir): # function (b) builds (file, word) tuples using function (a) (which builds (file, word) tuples from (file, text) tuples \n",
    "    file_text_RDD = sc.wholeTextFiles(argDir)# read the files and build (file, text) tuples\n",
    "    file_word_RDD = file_text_RDD.flatMap(splitFileWords) #use function (a)to build (file, word) tuples\n",
    "    #print('Read {} files from directory {}'.format(file_text_RDD.count(), argDir)) # print count and location of files used\n",
    "    #print('file word count histogram')\n",
    "    #print(file_word_RDD.map(lambda fwL: (len(fwL[1]))).histogram([0,10,100,500, 1000, 5000, 10000])) # print word-count histogram \n",
    "    return file_word_RDD \n",
    "\n",
    "file_word_RDD = read_file_word_RDD(dirPath) # apply function (b) on the text corpus for the analysis \n",
    "pprint(file_word_RDD.take(2)) # print (file, word) tuples indicatively\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* We create the function *__file_word_RDD_map_reduce()__* that yields ((file, word), count) tuples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(('hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part1/5-1230msg1.txt',\n",
      "   'quite'),\n",
      "  1),\n",
      " (('hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part1/spmsga140.txt',\n",
      "   'that'),\n",
      "  31)]\n"
     ]
    }
   ],
   "source": [
    "def file_word_RDD_map_reduce(file_word): # function (c) uses map and reduce to build ((file, word), count) tuples\n",
    "    file_word_1_RDD = file_word.map(lambda x: (x,1)) # map (file, word) tuples against 1\n",
    "    fileWord_count_RDD = file_word_1_RDD.reduceByKey(add) # aggregate the (file, word) tuples\n",
    "    return fileWord_count_RDD\n",
    "\n",
    "fileWord_count_RDD = file_word_RDD_map_reduce(file_word_RDD) # map (file, word) tuples \n",
    "pprint(fileWord_count_RDD.take(2)) # print ((file, word), 1) tuples indicatively"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* We create the function *__reorganise_tuples()__* that reorganises the reduced tuples from *((file, word), count)* to *(file, (word, count))*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part1/spmsga141.txt',\n",
      "  [('your', 5)]),\n",
      " ('hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part1/spmsga141.txt',\n",
      "  [('you', 9)])]\n"
     ]
    }
   ],
   "source": [
    "def reorganise_tuples(fw_c): # function (d) reorganises tuples from ((file, word), count) to (file, (word, count))\n",
    "    fw,c = fw_c # unpack the ((file, word), count) tuple into its elements\n",
    "    f,w = fw # unpack the nested (filename,word) tuple into its elements\n",
    "    return (f,[(w,c)]) # reorganise the elements into the structure (file, (word, count))\n",
    "\n",
    "file_wordCount_RDD = fileWord_count_RDD.map(reorganise_tuples) \n",
    "pprint(file_wordCount_RDD.top(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* We create the function *__make_file_termFreq_norm_RDD()__* that yields normalised frequency vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part1/3-550msg1.txt', [('sikillian', 0.045454545454545456), ('or', 0.09090909090909091), ('does', 0.045454545454545456), ('lists', 0.09090909090909091), ('latin', 0.045454545454545456), ('bitnet', 0.045454545454545456), ('any', 0.045454545454545456), ('to', 0.045454545454545456), ('query', 0.045454545454545456), ('anyone', 0.045454545454545456), ('annotext', 0.045454545454545456), ('', 0.045454545454545456), ('thanks', 0.045454545454545456), ('greek', 0.045454545454545456), ('subject', 0.045454545454545456), ('know', 0.045454545454545456), ('classical', 0.045454545454545456), ('michael', 0.045454545454545456), ('internet', 0.045454545454545456), ('dedicated', 0.045454545454545456)])]\n",
      "0.9999999999999999\n"
     ]
    }
   ],
   "source": [
    "def make_file_termFreq_norm_RDD(argDir): # function (e) produces normalised frequency vectors\n",
    "    file_word_RDD = read_file_word_RDD(argDir) # use function (b) \n",
    "    fileWord_count_RDD = file_word_RDD_map_reduce(file_word_RDD) # use function (c)\n",
    "    file_wordCount_RDD = fileWord_count_RDD.map(reorganise_tuples) # use function (d)\n",
    "    file_wordCount2_RDD = file_wordCount_RDD.reduceByKey(add)\n",
    "    file_wordCount_norm_RDD = file_wordCount2_RDD.map(lambda f_wcL:(f_wcL[0],[(w,c/sum([c for (w, c) in f_wcL[1]]))for (w,c) in f_wcL[1]])) # normalise\n",
    "    return file_wordCount_norm_RDD                                                \n",
    "\n",
    "file_wordCount_norm_RDD = make_file_termFreq_norm_RDD(dirPath) # test\n",
    "print(file_wordCount_norm_RDD.take(1))\n",
    "\n",
    "word_count_norm = file_wordCount_norm_RDD.take(1)[0][1] # get the first normalised word count list\n",
    "pprint(sum([c for (w,c) in word_count_norm])) # check that sum of normalised sum approximates 1 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* We create the function *__hashing_vectorizer()__* that creates fixed-sized frequency vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def hashing_vectorizer(word_count_list, N): # function (f) applies the hashing approach to creating a vector\n",
    "     v = [0] * N  # create fixed size vector of 0s\n",
    "     for word_count in word_count_list: \n",
    "         word,count = word_count# unpack tuple\n",
    "         h = hash(word)# get hash value\n",
    "         v[h % N] = v[h % N] + count # add count\n",
    "     return v# return hashed word vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* We create the function *__make_file_wordHashVector_norm_RDD()__* that creates fixed-sized frequency vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part1/3-550msg1.txt', [0.09090909090909091, 0.09090909090909091, 0, 0.09090909090909091, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.045454545454545456, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.045454545454545456, 0, 0, 0, 0, 0, 0, 0, 0, 0.045454545454545456, 0, 0, 0.045454545454545456, 0, 0.045454545454545456, 0, 0, 0.045454545454545456, 0, 0, 0, 0, 0, 0.045454545454545456, 0.045454545454545456, 0, 0, 0, 0.045454545454545456, 0, 0, 0, 0, 0, 0, 0, 0.045454545454545456, 0, 0, 0, 0, 0, 0, 0, 0.045454545454545456, 0, 0, 0, 0.045454545454545456, 0, 0, 0, 0, 0.09090909090909091, 0.045454545454545456, 0, 0, 0.045454545454545456]), ('hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part1/3-416msg2.txt', [0.006915629322268327, 0.013831258644536654, 0.009681881051175657, 0.02351313969571231, 0, 0.012448132780082988, 0.017980636237897647, 0.013831258644536652, 0.0027662517289073307, 0.019363762102351315, 0.030428769017980632, 0.0013831258644536654, 0.005532503457814661, 0.0013831258644536654, 0.004149377593360996, 0.006915629322268326, 0.006915629322268327, 0.05532503457814661, 0.006915629322268327, 0.011065006915629323, 0.008298755186721992, 0.016597510373443983, 0.006915629322268327, 0.004149377593360996, 0.012448132780082988, 0.0027662517289073307, 0.005532503457814661, 0.008298755186721992, 0, 0.011065006915629323, 0.017980636237897647, 0.008298755186721992, 0.006915629322268327, 0.004149377593360996, 0, 0.004149377593360996, 0.011065006915629323, 0.005532503457814661, 0.0013831258644536654, 0, 0.009681881051175657, 0.006915629322268327, 0.017980636237897647, 0.0027662517289073307, 0.012448132780082988, 0.0027662517289073307, 0.0013831258644536654, 0.006915629322268326, 0.006915629322268326, 0.02074688796680498, 0, 0.004149377593360996, 0.03319502074688797, 0.01521438450899032, 0.004149377593360996, 0.004149377593360996, 0.009681881051175657, 0.0027662517289073307, 0.008298755186721992, 0.004149377593360996, 0.0027662517289073307, 0, 0.0027662517289073307, 0.004149377593360996, 0.011065006915629323, 0.03734439834024896, 0.008298755186721992, 0.004149377593360996, 0.009681881051175657, 0.012448132780082988, 0.04011065006915629, 0.005532503457814661, 0, 0.011065006915629323, 0.011065006915629323, 0.009681881051175657, 0.008298755186721992, 0.0276625172890733, 0.009681881051175657, 0.004149377593360996, 0.03319502074688797, 0.0027662517289073307, 0.0027662517289073307, 0.02904564315352697, 0.005532503457814661, 0.005532503457814661, 0.005532503457814661, 0.013831258644536654, 0.0013831258644536654, 0.023513139695712303, 0.0013831258644536654, 0.004149377593360996, 0.0027662517289073307, 0.006915629322268327, 0.004149377593360996, 0.019363762102351315, 0.006915629322268327, 0.01521438450899032, 0.02074688796680498, 0.015214384508990318])]\n",
      "0.9999999999999997\n"
     ]
    }
   ],
   "source": [
    "def make_file_wordHashVector_norm_RDD(file_wordCount_norm, argN): # function (g) applies the hashing vectoriser\n",
    "    file_wordHashVector_norm_RDD = file_wordCount_norm.map(lambda f_wc: (f_wc[0],hashing_vectorizer(f_wc[1],argN))) \n",
    "    return file_wordHashVector_norm_RDD\n",
    "\n",
    "N=100\n",
    "file_wordHashVector_norm_RDD = make_file_wordHashVector_norm_RDD(make_file_termFreq_norm_RDD(dirPath),N)\n",
    "print(file_wordHashVector_norm_RDD.take(2)) # test\n",
    "print(sum(file_wordHashVector_norm_RDD.take(1)[0][1])) # test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* We create the function *__make_label_point_RDD()__* that builds labelled-point objects. The function works with input arguments being either a path or an RDD and assigns filename 1 to texts marked as spam emails and filename 0 to files not marked as spam emails"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LabeledPoint(0.0, [0.0909090909091,0.0909090909091,0.0,0.0909090909091,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0454545454545,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0454545454545,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0454545454545,0.0,0.0,0.0454545454545,0.0,0.0454545454545,0.0,0.0,0.0454545454545,0.0,0.0,0.0,0.0,0.0,0.0454545454545,0.0454545454545,0.0,0.0,0.0,0.0454545454545,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0454545454545,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0454545454545,0.0,0.0,0.0,0.0454545454545,0.0,0.0,0.0,0.0,0.0909090909091,0.0454545454545,0.0,0.0,0.0454545454545]), LabeledPoint(0.0, [0.00691562932227,0.0138312586445,0.00968188105118,0.0235131396957,0.0,0.0124481327801,0.0179806362379,0.0138312586445,0.00276625172891,0.0193637621024,0.030428769018,0.00138312586445,0.00553250345781,0.00138312586445,0.00414937759336,0.00691562932227,0.00691562932227,0.0553250345781,0.00691562932227,0.0110650069156,0.00829875518672,0.0165975103734,0.00691562932227,0.00414937759336,0.0124481327801,0.00276625172891,0.00553250345781,0.00829875518672,0.0,0.0110650069156,0.0179806362379,0.00829875518672,0.00691562932227,0.00414937759336,0.0,0.00414937759336,0.0110650069156,0.00553250345781,0.00138312586445,0.0,0.00968188105118,0.00691562932227,0.0179806362379,0.00276625172891,0.0124481327801,0.00276625172891,0.00138312586445,0.00691562932227,0.00691562932227,0.0207468879668,0.0,0.00414937759336,0.0331950207469,0.015214384509,0.00414937759336,0.00414937759336,0.00968188105118,0.00276625172891,0.00829875518672,0.00414937759336,0.00276625172891,0.0,0.00276625172891,0.00414937759336,0.0110650069156,0.0373443983402,0.00829875518672,0.00414937759336,0.00968188105118,0.0124481327801,0.0401106500692,0.00553250345781,0.0,0.0110650069156,0.0110650069156,0.00968188105118,0.00829875518672,0.0276625172891,0.00968188105118,0.00414937759336,0.0331950207469,0.00276625172891,0.00276625172891,0.0290456431535,0.00553250345781,0.00553250345781,0.00553250345781,0.0138312586445,0.00138312586445,0.0235131396957,0.00138312586445,0.00414937759336,0.00276625172891,0.00691562932227,0.00414937759336,0.0193637621024,0.00691562932227,0.015214384509,0.0207468879668,0.015214384509]), LabeledPoint(1.0, [0.0335820895522,0.00373134328358,0.00746268656716,0.0335820895522,0.0298507462687,0.0,0.00746268656716,0.0111940298507,0.0186567164179,0.00746268656716,0.0111940298507,0.00746268656716,0.0,0.0223880597015,0.0149253731343,0.00746268656716,0.0223880597015,0.00746268656716,0.0,0.0,0.00746268656716,0.0186567164179,0.0149253731343,0.00373134328358,0.00373134328358,0.0,0.00373134328358,0.0223880597015,0.0,0.0,0.00373134328358,0.0149253731343,0.00373134328358,0.00373134328358,0.0,0.0335820895522,0.0,0.00373134328358,0.00746268656716,0.0,0.0,0.0335820895522,0.00746268656716,0.00746268656716,0.0111940298507,0.0111940298507,0.0,0.0111940298507,0.0,0.00373134328358,0.00746268656716,0.00373134328358,0.00373134328358,0.0149253731343,0.0149253731343,0.0,0.00373134328358,0.00746268656716,0.00746268656716,0.0111940298507,0.0111940298507,0.0111940298507,0.0,0.0186567164179,0.0186567164179,0.0298507462687,0.00746268656716,0.0,0.00746268656716,0.0223880597015,0.0111940298507,0.0111940298507,0.0111940298507,0.00746268656716,0.00746268656716,0.0111940298507,0.0,0.00373134328358,0.00373134328358,0.00746268656716,0.0111940298507,0.0111940298507,0.0111940298507,0.00746268656716,0.0,0.0149253731343,0.00373134328358,0.00373134328358,0.00746268656716,0.0373134328358,0.00373134328358,0.00373134328358,0.00373134328358,0.00746268656716,0.0149253731343,0.0149253731343,0.00373134328358,0.0111940298507,0.0335820895522,0.0410447761194])]\n"
     ]
    }
   ],
   "source": [
    "def make_label_point_RDD(inp,argN,trg): # function (f) creates labelled points where 1=spam; 0=non-spam and works when the argument inp is either a path (when trg =='path') or an RDD (when trg==''). The latter becomes useful at Task e  \n",
    "    file_wordHashVector_norm_RDD = make_file_wordHashVector_norm_RDD((make_file_termFreq_norm_RDD(inp) if trg=='path' else inp),argN) # retrive the hashed normalised vector using function (g)\n",
    "    label_point_RDD = file_wordHashVector_norm_RDD.map(lambda f_wVec: LabeledPoint(0 if (re.search('spmsg', f_wVec[0])==None) else 1,f_wVec[1])) # assign labelled points\n",
    "    return label_point_RDD\n",
    "\n",
    "label_point_RDD = make_label_point_RDD(dirPath, 100,'path')# test\n",
    "print(label_point_RDD.take(3)) # test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* We create the function *__trainModel()__* that trains a classifier for spam (=1) versus non-spam (=0) files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training Logistic Regression with Limited-memory the Broyden–Fletcher–Goldfarb–Shanno (BFGS)\n",
      "training data items: 289, correct: 289\n",
      "training accuracy 100.0%\n"
     ]
    }
   ],
   "source": [
    "def trainModel(argDir,argN): # function(g) trains a logistic regression model to detect spam vs. non-spam files\n",
    "    label_point_RDD_train = make_label_point_RDD(argDir, argN,'path') # uses function (f) to build labelled points when argument inp is a path (i.e. trg=='path')\n",
    "    logReg_model = LogisticRegressionWithLBFGS.train(label_point_RDD_train) # train the algorithm\n",
    "    correct = label_point_RDD_train.map(lambda lp: 1 if logReg_model.predict(lp.features) == lp.label else 0).sum() # calculate correctly classified data points\n",
    "    count = label_point_RDD_train.count() # counts the size of training set\n",
    "    print('training Logistic Regression with Limited-memory the Broyden–Fletcher–Goldfarb–Shanno (BFGS)')\n",
    "    print('training data items: {}, correct: {}'.format(count, correct))\n",
    "    print('training accuracy {:.1%}'.format(correct/count)) \n",
    "    return logReg_model\n",
    "\n",
    "logReg_model = trainModel(dirPath, 100) # test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* We test the classifier built in the function *__testModel()__* on a different corpus\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training Logistic Regression with Limited-memory the Broyden–Fletcher–Goldfarb–Shanno (BFGS)\n",
      "training data items: 289, correct: 289\n",
      "training accuracy 100.0%\n",
      "test data items: 291, correct:254\n",
      "testing accuracy 87.3%\n"
     ]
    }
   ],
   "source": [
    "def testModel(argDir, argN): # function (h) tests the logistic regression model that is trained with function (g)\n",
    "    label_point_RDD_test = make_label_point_RDD(argDir, argN,'path') # uses function (f) to build labelled points when armument inp is a path (i.e. trg=='path')\n",
    "    logReg_model = trainModel(dirPath, 100) # calls the sustantive model\n",
    "    correct_test = label_point_RDD_test.map( lambda lp: 1 if logReg_model.predict(lp.features) == lp.label else 0).sum() # calculates correctly classified data points after applying the substantive model\n",
    "    count_test = label_point_RDD_test.count() # counts data points in the test dataset\n",
    "    print('test data items: {}, correct:{}'.format(count_test,correct_test))\n",
    "    print('testing accuracy {:.1%}'.format(correct_test/count_test))\n",
    "\n",
    "N = 100\n",
    "logReg_model_test = testModel(dirPath_2, N) #test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* We create the function *__trainTestModel()__* that trains and tests a classifier for spam (=1) versus non-spam (=0) files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logistic Regression BFGS; training data items: 289, correct: 289\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:271\n",
      "Logistic Regression BFGS; testing accuracy 93.1%\n"
     ]
    }
   ],
   "source": [
    "def trainTestModel(train_inp,test_inp,argN,trg): # function (i) trains and tests the classifier\n",
    "    label_point_RDD_train = make_label_point_RDD(train_inp, argN,trg) # build label point from training data using function (f)\n",
    "    logReg_model_train = LogisticRegressionWithLBFGS.train(label_point_RDD_train) # train logistic regression\n",
    "    correct_train = label_point_RDD_train.map(lambda lp: 1 if logReg_model_train.predict(lp.features) == lp.label else 0).sum() \n",
    "    count_train = label_point_RDD_train.count()\n",
    "    label_point_RDD_test = make_label_point_RDD(test_inp, argN,trg) # build label point from test data using function (f)\n",
    "    correct_test = label_point_RDD_test.map( lambda lp: 1 if logReg_model_train.predict(lp.features) == lp.label else 0).sum()     \n",
    "    count_test = label_point_RDD_test.count()\n",
    "    print('Logistic Regression BFGS; training data items: {}, correct: {}'.format(count_train, correct_train))\n",
    "    print('Logistic Regression BFGS; training accuracy {:.1%}'.format(correct_train/count_train))\n",
    "    print('Logistic Regression BFGS; test data items: {}, correct:{}'.format(count_test,correct_test))\n",
    "    print('Logistic Regression BFGS; testing accuracy {:.1%}'.format(correct_test/count_test))\n",
    "    #return trainTestModel\n",
    "    \n",
    "trainTestModel_example = trainTestModel('hdfs://saltdean/data/spam/bare/part6','hdfs://saltdean/data/spam/bare/part10',100,'path') # test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* We define a dictionary for the four text corpora that we will examine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "setDict = {'EXPERIMENT A: Testing different vector sizes - No preprocessing':prefix+'spam/bare/',\n",
    "           'EXPERIMENT B: Testing different vector sizes - Stopwords removed':prefix+'spam/stop/',\n",
    "           'EXPERIMENT C: Testing different vector sizes - Lemmatised':prefix+'spam/lemm/',\n",
    "           'EXPERIMENT D: Testing different vector sizes - Lemmatised and stopwords removed':prefix+'spam/lemm_stop/'}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* We conduct **Experiment 1** that explores classification accuracy against the size of the training dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "***** Experiment 1 initiated *****\n",
      "\n",
      "***** EXPERIMENT A: Testing different vector sizes - No preprocessing\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part10\n",
      "\n",
      "**********add part 1**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part[1-1]\n",
      "Logistic Regression BFGS; training data items: 289, correct: 289\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:254\n",
      "Logistic Regression BFGS; testing accuracy 87.3%\n",
      "\n",
      "**********add part 2**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part[1-2]\n",
      "Logistic Regression BFGS; training data items: 578, correct: 578\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:266\n",
      "Logistic Regression BFGS; testing accuracy 91.4%\n",
      "\n",
      "**********add part 3**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part[1-3]\n",
      "Logistic Regression BFGS; training data items: 867, correct: 867\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:265\n",
      "Logistic Regression BFGS; testing accuracy 91.1%\n",
      "\n",
      "**********add part 4**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part[1-4]\n",
      "Logistic Regression BFGS; training data items: 1156, correct: 1156\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:265\n",
      "Logistic Regression BFGS; testing accuracy 91.1%\n",
      "\n",
      "**********add part 5**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part[1-5]\n",
      "Logistic Regression BFGS; training data items: 1446, correct: 1446\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:264\n",
      "Logistic Regression BFGS; testing accuracy 90.7%\n",
      "\n",
      "**********add part 6**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part[1-6]\n",
      "Logistic Regression BFGS; training data items: 1735, correct: 1708\n",
      "Logistic Regression BFGS; training accuracy 98.4%\n",
      "Logistic Regression BFGS; test data items: 291, correct:274\n",
      "Logistic Regression BFGS; testing accuracy 94.2%\n",
      "\n",
      "**********add part 7**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part[1-7]\n",
      "Logistic Regression BFGS; training data items: 2024, correct: 1990\n",
      "Logistic Regression BFGS; training accuracy 98.3%\n",
      "Logistic Regression BFGS; test data items: 291, correct:280\n",
      "Logistic Regression BFGS; testing accuracy 96.2%\n",
      "\n",
      "**********add part 8**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part[1-8]\n",
      "Logistic Regression BFGS; training data items: 2313, correct: 2258\n",
      "Logistic Regression BFGS; training accuracy 97.6%\n",
      "Logistic Regression BFGS; test data items: 291, correct:275\n",
      "Logistic Regression BFGS; testing accuracy 94.5%\n",
      "\n",
      "**********add part 9**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part[1-9]\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2540\n",
      "Logistic Regression BFGS; training accuracy 97.6%\n",
      "Logistic Regression BFGS; test data items: 291, correct:276\n",
      "Logistic Regression BFGS; testing accuracy 94.8%\n",
      "\n",
      "***** EXPERIMENT B: Testing different vector sizes - Stopwords removed\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/stop/part10\n",
      "\n",
      "**********add part 1**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/stop/part[1-1]\n",
      "Logistic Regression BFGS; training data items: 289, correct: 289\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:219\n",
      "Logistic Regression BFGS; testing accuracy 75.3%\n",
      "\n",
      "**********add part 2**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/stop/part[1-2]\n",
      "Logistic Regression BFGS; training data items: 578, correct: 578\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:232\n",
      "Logistic Regression BFGS; testing accuracy 79.7%\n",
      "\n",
      "**********add part 3**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/stop/part[1-3]\n",
      "Logistic Regression BFGS; training data items: 867, correct: 841\n",
      "Logistic Regression BFGS; training accuracy 97.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:221\n",
      "Logistic Regression BFGS; testing accuracy 75.9%\n",
      "\n",
      "**********add part 4**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/stop/part[1-4]\n",
      "Logistic Regression BFGS; training data items: 1156, correct: 1101\n",
      "Logistic Regression BFGS; training accuracy 95.2%\n",
      "Logistic Regression BFGS; test data items: 291, correct:248\n",
      "Logistic Regression BFGS; testing accuracy 85.2%\n",
      "\n",
      "**********add part 5**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/stop/part[1-5]\n",
      "Logistic Regression BFGS; training data items: 1446, correct: 1359\n",
      "Logistic Regression BFGS; training accuracy 94.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:257\n",
      "Logistic Regression BFGS; testing accuracy 88.3%\n",
      "\n",
      "**********add part 6**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/stop/part[1-6]\n",
      "Logistic Regression BFGS; training data items: 1735, correct: 1625\n",
      "Logistic Regression BFGS; training accuracy 93.7%\n",
      "Logistic Regression BFGS; test data items: 291, correct:255\n",
      "Logistic Regression BFGS; testing accuracy 87.6%\n",
      "\n",
      "**********add part 7**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/stop/part[1-7]\n",
      "Logistic Regression BFGS; training data items: 2024, correct: 1910\n",
      "Logistic Regression BFGS; training accuracy 94.4%\n",
      "Logistic Regression BFGS; test data items: 291, correct:260\n",
      "Logistic Regression BFGS; testing accuracy 89.3%\n",
      "\n",
      "**********add part 8**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/stop/part[1-8]\n",
      "Logistic Regression BFGS; training data items: 2313, correct: 2163\n",
      "Logistic Regression BFGS; training accuracy 93.5%\n",
      "Logistic Regression BFGS; test data items: 291, correct:255\n",
      "Logistic Regression BFGS; testing accuracy 87.6%\n",
      "\n",
      "**********add part 9**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/stop/part[1-9]\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2425\n",
      "Logistic Regression BFGS; training accuracy 93.2%\n",
      "Logistic Regression BFGS; test data items: 291, correct:251\n",
      "Logistic Regression BFGS; testing accuracy 86.3%\n",
      "\n",
      "***** EXPERIMENT C: Testing different vector sizes - Lemmatised\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm/part10\n",
      "\n",
      "**********add part 1**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm/part[1-1]\n",
      "Logistic Regression BFGS; training data items: 289, correct: 289\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:251\n",
      "Logistic Regression BFGS; testing accuracy 86.3%\n",
      "\n",
      "**********add part 2**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm/part[1-2]\n",
      "Logistic Regression BFGS; training data items: 578, correct: 578\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:250\n",
      "Logistic Regression BFGS; testing accuracy 85.9%\n",
      "\n",
      "**********add part 3**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm/part[1-3]\n",
      "Logistic Regression BFGS; training data items: 867, correct: 867\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:248\n",
      "Logistic Regression BFGS; testing accuracy 85.2%\n",
      "\n",
      "**********add part 4**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm/part[1-4]\n",
      "Logistic Regression BFGS; training data items: 1156, correct: 1156\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:259\n",
      "Logistic Regression BFGS; testing accuracy 89.0%\n",
      "\n",
      "**********add part 5**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm/part[1-5]\n",
      "Logistic Regression BFGS; training data items: 1446, correct: 1446\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:263\n",
      "Logistic Regression BFGS; testing accuracy 90.4%\n",
      "\n",
      "**********add part 6**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm/part[1-6]\n",
      "Logistic Regression BFGS; training data items: 1735, correct: 1712\n",
      "Logistic Regression BFGS; training accuracy 98.7%\n",
      "Logistic Regression BFGS; test data items: 291, correct:270\n",
      "Logistic Regression BFGS; testing accuracy 92.8%\n",
      "\n",
      "**********add part 7**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm/part[1-7]\n",
      "Logistic Regression BFGS; training data items: 2024, correct: 1990\n",
      "Logistic Regression BFGS; training accuracy 98.3%\n",
      "Logistic Regression BFGS; test data items: 291, correct:271\n",
      "Logistic Regression BFGS; testing accuracy 93.1%\n",
      "\n",
      "**********add part 8**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm/part[1-8]\n",
      "Logistic Regression BFGS; training data items: 2313, correct: 2266\n",
      "Logistic Regression BFGS; training accuracy 98.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:273\n",
      "Logistic Regression BFGS; testing accuracy 93.8%\n",
      "\n",
      "**********add part 9**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm/part[1-9]\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2541\n",
      "Logistic Regression BFGS; training accuracy 97.7%\n",
      "Logistic Regression BFGS; test data items: 291, correct:274\n",
      "Logistic Regression BFGS; testing accuracy 94.2%\n",
      "\n",
      "***** EXPERIMENT D: Testing different vector sizes - Lemmatised and stopwords removed\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm_stop/part10\n",
      "\n",
      "**********add part 1**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm_stop/part[1-1]\n",
      "Logistic Regression BFGS; training data items: 289, correct: 289\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:222\n",
      "Logistic Regression BFGS; testing accuracy 76.3%\n",
      "\n",
      "**********add part 2**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm_stop/part[1-2]\n",
      "Logistic Regression BFGS; training data items: 578, correct: 578\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:224\n",
      "Logistic Regression BFGS; testing accuracy 77.0%\n",
      "\n",
      "**********add part 3**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm_stop/part[1-3]\n",
      "Logistic Regression BFGS; training data items: 867, correct: 847\n",
      "Logistic Regression BFGS; training accuracy 97.7%\n",
      "Logistic Regression BFGS; test data items: 291, correct:219\n",
      "Logistic Regression BFGS; testing accuracy 75.3%\n",
      "\n",
      "**********add part 4**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm_stop/part[1-4]\n",
      "Logistic Regression BFGS; training data items: 1156, correct: 1116\n",
      "Logistic Regression BFGS; training accuracy 96.5%\n",
      "Logistic Regression BFGS; test data items: 291, correct:237\n",
      "Logistic Regression BFGS; testing accuracy 81.4%\n",
      "\n",
      "**********add part 5**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm_stop/part[1-5]\n",
      "Logistic Regression BFGS; training data items: 1446, correct: 1367\n",
      "Logistic Regression BFGS; training accuracy 94.5%\n",
      "Logistic Regression BFGS; test data items: 291, correct:254\n",
      "Logistic Regression BFGS; testing accuracy 87.3%\n",
      "\n",
      "**********add part 6**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm_stop/part[1-6]\n",
      "Logistic Regression BFGS; training data items: 1735, correct: 1636\n",
      "Logistic Regression BFGS; training accuracy 94.3%\n",
      "Logistic Regression BFGS; test data items: 291, correct:257\n",
      "Logistic Regression BFGS; testing accuracy 88.3%\n",
      "\n",
      "**********add part 7**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm_stop/part[1-7]\n",
      "Logistic Regression BFGS; training data items: 2024, correct: 1900\n",
      "Logistic Regression BFGS; training accuracy 93.9%\n",
      "Logistic Regression BFGS; test data items: 291, correct:259\n",
      "Logistic Regression BFGS; testing accuracy 89.0%\n",
      "\n",
      "**********add part 8**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm_stop/part[1-8]\n",
      "Logistic Regression BFGS; training data items: 2313, correct: 2162\n",
      "Logistic Regression BFGS; training accuracy 93.5%\n",
      "Logistic Regression BFGS; test data items: 291, correct:260\n",
      "Logistic Regression BFGS; testing accuracy 89.3%\n",
      "\n",
      "**********add part 9**********\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm_stop/part[1-9]\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2420\n",
      "Logistic Regression BFGS; training accuracy 93.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:255\n",
      "Logistic Regression BFGS; testing accuracy 87.6%\n",
      "\n",
      "***** Experiment 1 completed *****\n"
     ]
    }
   ],
   "source": [
    "N=100 # define indicative vector size\n",
    "\n",
    "print('\\n***** Experiment 1 initiated *****')\n",
    "for sp in sorted(setDict):\n",
    "    print('\\n*****',sp)\n",
    "    testPath = setDict[sp]+'part10' # define the path of the test data\n",
    "    dirPattern = setDict[sp]+\"part[1-{}]\" # define a pattern whereby train data will be loaded (i.e. part 1; part1 + part2; ...)\n",
    "    print(testPath)\n",
    "    for i in range(1,10):\n",
    "        print('\\n**********add part %d**********' %(i)) \n",
    "        trainPaths = dirPattern.format(i) \n",
    "        print(trainPaths) #just for testing, remove later\n",
    "        trainTestModel(trainPaths,testPath,N,'path')\n",
    "\n",
    "print('\\n***** Experiment 1 completed *****')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "**Key observations on the output of Experiment 1:** (i) For a given vector size N, increases in the size of the training dataset tend to enhance testing accuracy; (ii) For a given vector size N, removing stop words from the text corpus is linked to decreases in testing accuracy. A potential interpretation of this effect is that stopwords may be carrying information about the writing style of the texts, which is indicative of whether they are spam or not; and (iii) For a given vector size N, processing a lemmatised corpus yields fairly similar testing accuracy to processing an unprocessed corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* We conduct **Experiment 2** that explores classification accuracy against vector size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "***** Experiment 2 initiated *****\n",
      "\n",
      "***** EXPERIMENT A: Testing different vector sizes - No preprocessing\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part10\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/bare/part[1-9]\n",
      "\n",
      "***** size of vector N = 3\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2165\n",
      "Logistic Regression BFGS; training accuracy 83.2%\n",
      "Logistic Regression BFGS; test data items: 291, correct:233\n",
      "Logistic Regression BFGS; testing accuracy 80.1%\n",
      "*****processing duration 50.01354503631592 seconds*****\n",
      "\n",
      "***** size of vector N = 10\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2194\n",
      "Logistic Regression BFGS; training accuracy 84.3%\n",
      "Logistic Regression BFGS; test data items: 291, correct:248\n",
      "Logistic Regression BFGS; testing accuracy 85.2%\n",
      "*****processing duration 50.11001801490784 seconds*****\n",
      "\n",
      "***** size of vector N = 30\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2354\n",
      "Logistic Regression BFGS; training accuracy 90.5%\n",
      "Logistic Regression BFGS; test data items: 291, correct:255\n",
      "Logistic Regression BFGS; testing accuracy 87.6%\n",
      "*****processing duration 48.951175928115845 seconds*****\n",
      "\n",
      "***** size of vector N = 100\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2540\n",
      "Logistic Regression BFGS; training accuracy 97.6%\n",
      "Logistic Regression BFGS; test data items: 291, correct:276\n",
      "Logistic Regression BFGS; testing accuracy 94.8%\n",
      "*****processing duration 47.53315091133118 seconds*****\n",
      "\n",
      "***** size of vector N = 300\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2602\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:284\n",
      "Logistic Regression BFGS; testing accuracy 97.6%\n",
      "*****processing duration 46.57633996009827 seconds*****\n",
      "\n",
      "***** size of vector N = 1000\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2602\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:287\n",
      "Logistic Regression BFGS; testing accuracy 98.6%\n",
      "*****processing duration 44.77421808242798 seconds*****\n",
      "\n",
      "***** size of vector N = 3000\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2602\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:288\n",
      "Logistic Regression BFGS; testing accuracy 99.0%\n",
      "*****processing duration 50.02766227722168 seconds*****\n",
      "\n",
      "***** EXPERIMENT B: Testing different vector sizes - Stopwords removed\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/stop/part10\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/stop/part[1-9]\n",
      "\n",
      "***** size of vector N = 3\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2171\n",
      "Logistic Regression BFGS; training accuracy 83.4%\n",
      "Logistic Regression BFGS; test data items: 291, correct:240\n",
      "Logistic Regression BFGS; testing accuracy 82.5%\n",
      "*****processing duration 40.1761429309845 seconds*****\n",
      "\n",
      "***** size of vector N = 10\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2192\n",
      "Logistic Regression BFGS; training accuracy 84.2%\n",
      "Logistic Regression BFGS; test data items: 291, correct:237\n",
      "Logistic Regression BFGS; testing accuracy 81.4%\n",
      "*****processing duration 41.7465124130249 seconds*****\n",
      "\n",
      "***** size of vector N = 30\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2241\n",
      "Logistic Regression BFGS; training accuracy 86.1%\n",
      "Logistic Regression BFGS; test data items: 291, correct:246\n",
      "Logistic Regression BFGS; testing accuracy 84.5%\n",
      "*****processing duration 45.21866703033447 seconds*****\n",
      "\n",
      "***** size of vector N = 100\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2425\n",
      "Logistic Regression BFGS; training accuracy 93.2%\n",
      "Logistic Regression BFGS; test data items: 291, correct:251\n",
      "Logistic Regression BFGS; testing accuracy 86.3%\n",
      "*****processing duration 44.49857521057129 seconds*****\n",
      "\n",
      "***** size of vector N = 300\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2602\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:273\n",
      "Logistic Regression BFGS; testing accuracy 93.8%\n",
      "*****processing duration 44.04692459106445 seconds*****\n",
      "\n",
      "***** size of vector N = 1000\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2602\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:286\n",
      "Logistic Regression BFGS; testing accuracy 98.3%\n",
      "*****processing duration 41.19809532165527 seconds*****\n",
      "\n",
      "***** size of vector N = 3000\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2602\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:286\n",
      "Logistic Regression BFGS; testing accuracy 98.3%\n",
      "*****processing duration 45.568400382995605 seconds*****\n",
      "\n",
      "***** EXPERIMENT C: Testing different vector sizes - Lemmatised\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm/part10\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm/part[1-9]\n",
      "\n",
      "***** size of vector N = 3\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2171\n",
      "Logistic Regression BFGS; training accuracy 83.4%\n",
      "Logistic Regression BFGS; test data items: 291, correct:233\n",
      "Logistic Regression BFGS; testing accuracy 80.1%\n",
      "*****processing duration 44.64307975769043 seconds*****\n",
      "\n",
      "***** size of vector N = 10\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2169\n",
      "Logistic Regression BFGS; training accuracy 83.4%\n",
      "Logistic Regression BFGS; test data items: 291, correct:240\n",
      "Logistic Regression BFGS; testing accuracy 82.5%\n",
      "*****processing duration 37.971388816833496 seconds*****\n",
      "\n",
      "***** size of vector N = 30\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2329\n",
      "Logistic Regression BFGS; training accuracy 89.5%\n",
      "Logistic Regression BFGS; test data items: 291, correct:256\n",
      "Logistic Regression BFGS; testing accuracy 88.0%\n",
      "*****processing duration 39.89562463760376 seconds*****\n",
      "\n",
      "***** size of vector N = 100\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2541\n",
      "Logistic Regression BFGS; training accuracy 97.7%\n",
      "Logistic Regression BFGS; test data items: 291, correct:274\n",
      "Logistic Regression BFGS; testing accuracy 94.2%\n",
      "*****processing duration 38.74367308616638 seconds*****\n",
      "\n",
      "***** size of vector N = 300\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2602\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:285\n",
      "Logistic Regression BFGS; testing accuracy 97.9%\n",
      "*****processing duration 37.83487391471863 seconds*****\n",
      "\n",
      "***** size of vector N = 1000\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2602\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:287\n",
      "Logistic Regression BFGS; testing accuracy 98.6%\n",
      "*****processing duration 40.12249541282654 seconds*****\n",
      "\n",
      "***** size of vector N = 3000\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2602\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:287\n",
      "Logistic Regression BFGS; testing accuracy 98.6%\n",
      "*****processing duration 46.257266998291016 seconds*****\n",
      "\n",
      "***** EXPERIMENT D: Testing different vector sizes - Lemmatised and stopwords removed\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm_stop/part10\n",
      "hdfs://saltdean.nsqdc.city.ac.uk/data/spam/lemm_stop/part[1-9]\n",
      "\n",
      "***** size of vector N = 3\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2171\n",
      "Logistic Regression BFGS; training accuracy 83.4%\n",
      "Logistic Regression BFGS; test data items: 291, correct:237\n",
      "Logistic Regression BFGS; testing accuracy 81.4%\n",
      "*****processing duration 35.67483329772949 seconds*****\n",
      "\n",
      "***** size of vector N = 10\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2176\n",
      "Logistic Regression BFGS; training accuracy 83.6%\n",
      "Logistic Regression BFGS; test data items: 291, correct:235\n",
      "Logistic Regression BFGS; testing accuracy 80.8%\n",
      "*****processing duration 37.79244947433472 seconds*****\n",
      "\n",
      "***** size of vector N = 30\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2256\n",
      "Logistic Regression BFGS; training accuracy 86.7%\n",
      "Logistic Regression BFGS; test data items: 291, correct:240\n",
      "Logistic Regression BFGS; testing accuracy 82.5%\n",
      "*****processing duration 36.780619621276855 seconds*****\n",
      "\n",
      "***** size of vector N = 100\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2420\n",
      "Logistic Regression BFGS; training accuracy 93.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:255\n",
      "Logistic Regression BFGS; testing accuracy 87.6%\n",
      "*****processing duration 39.723405599594116 seconds*****\n",
      "\n",
      "***** size of vector N = 300\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2602\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:283\n",
      "Logistic Regression BFGS; testing accuracy 97.3%\n",
      "*****processing duration 40.441476345062256 seconds*****\n",
      "\n",
      "***** size of vector N = 1000\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2602\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:287\n",
      "Logistic Regression BFGS; testing accuracy 98.6%\n",
      "*****processing duration 42.52954292297363 seconds*****\n",
      "\n",
      "***** size of vector N = 3000\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2602\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:285\n",
      "Logistic Regression BFGS; testing accuracy 97.9%\n",
      "*****processing duration 45.03615999221802 seconds*****\n",
      "\n",
      "***** Experiment 2 completed *****\n"
     ]
    }
   ],
   "source": [
    "print('\\n***** Experiment 2 initiated *****')\n",
    "\n",
    "for sp in sorted(setDict):\n",
    "    print('\\n*****',sp)\n",
    "    testPath = setDict[sp]+'part10'\n",
    "    trainPaths = setDict[sp]+\"part[1-9]\"\n",
    "    print(testPath)\n",
    "    print(trainPaths)\n",
    "    for i in [3, 10, 30, 100, 300, 1000, 3000]:\n",
    "        print('\\n***** size of vector N = %d' %(i))\n",
    "        start_time = time.time()\n",
    "        trainTestModel(trainPaths,testPath,i,'path')\n",
    "        print(\"*****processing duration %s seconds*****\" % (time.time() - start_time))\n",
    "\n",
    "print('\\n***** Experiment 2 completed *****')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "**Key observations on the output of Experiment 2:** (i) Increasing the vector size tends to increase testing accuracy, but is  computationally more intensive, as computation timings suggest; (ii) For a given vector size N, increases in the size of the training dataset tend to enhance testing accuracy (as in Experiment 1); (iii) For a given vector size N, removing stop words from the text corpus is linked to decreases in testing accuracy (as in Experiment 1); (iv) For a given vector size N, processing a lemmatised corpus yields fairly similar testing accuracy to processing an unprocessed corpus (as in Experiment 1)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* Finally, we conduct **Experiment 3** that explores differently preprocessed datasets. We define the vector size N = 300, based on the outcome of experiment 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "***** Experiment 3 initiated *****\n",
      "EXPERIMENT 3: Testing differently preprocessed data sets\n",
      "training on parts 1-9, N = 300\n",
      "\n",
      "***** EXPERIMENT A: Testing different vector sizes - No preprocessing\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2602\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:284\n",
      "Logistic Regression BFGS; testing accuracy 97.6%\n",
      "\n",
      "***** EXPERIMENT B: Testing different vector sizes - Stopwords removed\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2602\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:273\n",
      "Logistic Regression BFGS; testing accuracy 93.8%\n",
      "\n",
      "***** EXPERIMENT C: Testing different vector sizes - Lemmatised\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2602\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:285\n",
      "Logistic Regression BFGS; testing accuracy 97.9%\n",
      "\n",
      "***** EXPERIMENT D: Testing different vector sizes - Lemmatised and stopwords removed\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2602\n",
      "Logistic Regression BFGS; training accuracy 100.0%\n",
      "Logistic Regression BFGS; test data items: 291, correct:283\n",
      "Logistic Regression BFGS; testing accuracy 97.3%\n",
      "\n",
      "***** Experiment 3 completed *****\n"
     ]
    }
   ],
   "source": [
    "print('\\n***** Experiment 3 initiated *****')\n",
    "\n",
    "N=300 # vector size of 300 appears faster with equivalent accuracy\n",
    "\n",
    "print('EXPERIMENT 3: Testing differently preprocessed data sets')\n",
    "print('training on parts 1-9, N = {}'.format(N))\n",
    "for sp in sorted(setDict):\n",
    "    print('\\n*****',sp)\n",
    "    testPath = setDict[sp]+'part10'\n",
    "    trainPaths = setDict[sp]+\"part[1-9]\"\n",
    "    #print(testPath)\n",
    "    #print(trainPaths)\n",
    "    trainTestModel(trainPaths,testPath,N,'path') \n",
    "    \n",
    "print('\\n***** Experiment 3 completed *****')\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "**Key observations on the output of Experiment 3:** (i) Training the lemmatised corpus yields marginally higher testing accuracy compared to training the unprocessed corpus; (ii) Perhaps *surprisingly*, training a corpus where stop words have been removed and words have been lemmatised yields substantially higher testing accuracy compared to training a corpus where stop words have been removed but lemmatisation hasn't been applied."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "* We define a function that creates TF.IDF RDD that will be used for experiment 4. **[Task i]**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def make_f_wtfiL_RDD(path): # define function (j) that creates tf.idf\n",
    "    # Calculuate the IDFs\n",
    "    fw_RDD = read_file_word_RDD(path)\n",
    "    fw_u_RDD=fw_RDD.distinct() # maintains unique (file, word) pairs\n",
    "    fw_uwf_RDD=fw_u_RDD.map(lambda fw:(fw[1],[fw[0]])) # reorganises (file, word) -to (word,[file])\n",
    "    fw_uwfn_RDD=fw_uwf_RDD.reduceByKey(add) # joins the lists of files with reduceByKey\n",
    "    vocSize = fw_RDD.map(lambda fw: fw[0]).distinct().count() # calculates the vocabulary size (i.e. the count of text files)\n",
    "    #print('\\nvocSize: {}'.format(vocSize)) # print the vocabulary size \n",
    "    wIdf_RDD=fw_uwfn_RDD.map(lambda wf: (wf[0],log(vocSize/(1+len(wf[1]))))) # calculates the IDF  \n",
    "    # print('\\nwIdf_RDD.count(): ',wIdf_RDD.count()) # testing\n",
    "    # print('\\ncalculated idf',wIdf_RDD.take(2)) # testing\n",
    "\n",
    "    # Gets the normalise word counts (TFs) and organise by word (word,(file,count))\n",
    "    f_wcLn_RDD = make_file_termFreq_norm_RDD(path) # creates the normalised word count lists \n",
    "    #print('f_wcLn_RDD: ',f_wcLn_RDD.map(\n",
    "    #        lambda x: sum([c for (w,c) in x[1]]).histogram([0,10,100,1000,10000]))) # checks for the per-file word counts\n",
    "    w_fcn_RDD=f_wcLn_RDD.flatMap(lambda fwc:[(w,(fwc[0],c)) for (w,c) in fwc[1]]) #creates a list of tuples [(word,(file,count)), ..] and uses flatmap print('w_fcn_RDD.count(): {}'.format(w_fcn_RDD.count())) # for testing\n",
    "    # print('\\nmade the tf as (w,(f,cn))',w_fcn_RDD.take(2)) # testing\n",
    "\n",
    "    # now we can join the IFDs and TFs by the words (word,(file,coun)) join (word,idf) to (word,((file,count),idf))\n",
    "    w_fcnIdf_RDD=w_fcn_RDD.join(wIdf_RDD) #Join the IDF and TF RRD's\n",
    "    # print( '\\nw_fcnIdf_RDD.count(): ', w_fcnIdf_RDD.count())\n",
    "    # print( '\\njoined(w,((f,cn),idf))', w_fcnIdf_RDD.take(2))\n",
    "\n",
    "    # we have doubly nested tuples (word,((file,count),idf)) in the RDD, \n",
    "    # but they let us calculate the TF.IDF per file and word (file,[(word,count*idf)]).\n",
    "    f_wtfiL_RDD=w_fcnIdf_RDD.map(lambda w_fcnIdf:(w_fcnIdf[1][0][0],[(w_fcnIdf[0],(w_fcnIdf[1][0][1]*w_fcnIdf[1][1]))])) #Map to (f,[(w,Tf*idf)])\n",
    "    f_wtfiL_RDD=f_wtfiL_RDD.reduceByKey(add)\n",
    "    # print('\\nf_wtfiL_RDD.count()', f_wtfiL_RDD.count())\n",
    "    # print('\\n Calculated TF.IDF',str(f_wtfiL_RDD.take(2)))\n",
    "\n",
    "    return f_wtfiL_RDD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* We conduct **Experiment 4** that explores TF.IDF accuracy against the size of the training dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EXPERIMENT 4: Testing differently preprocessed data sets with TF.IDF\n",
      "\n",
      "***** Experiment 4 initiated *****\n",
      "training on parts 1-9, N = 100\n",
      "\n",
      "***** EXPERIMENT A: Testing different vector sizes - No preprocessing\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2336\n",
      "Logistic Regression BFGS; training accuracy 89.8%\n",
      "Logistic Regression BFGS; test data items: 291, correct:251\n",
      "Logistic Regression BFGS; testing accuracy 86.3%\n",
      "\n",
      "***** EXPERIMENT B: Testing different vector sizes - Stopwords removed\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2278\n",
      "Logistic Regression BFGS; training accuracy 87.5%\n",
      "Logistic Regression BFGS; test data items: 291, correct:245\n",
      "Logistic Regression BFGS; testing accuracy 84.2%\n",
      "\n",
      "***** EXPERIMENT C: Testing different vector sizes - Lemmatised\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2353\n",
      "Logistic Regression BFGS; training accuracy 90.4%\n",
      "Logistic Regression BFGS; test data items: 291, correct:261\n",
      "Logistic Regression BFGS; testing accuracy 89.7%\n",
      "\n",
      "***** EXPERIMENT D: Testing different vector sizes - Lemmatised and stopwords removed\n",
      "Logistic Regression BFGS; training data items: 2602, correct: 2292\n",
      "Logistic Regression BFGS; training accuracy 88.1%\n",
      "Logistic Regression BFGS; test data items: 291, correct:250\n",
      "Logistic Regression BFGS; testing accuracy 85.9%\n",
      "\n",
      "***** Experiment 4 completed *****\n"
     ]
    }
   ],
   "source": [
    "#Aplication of trainModel and test Model to RDDs created with make_f_wtfiL_RDD\n",
    "\n",
    "N=100 # indicative vector size\n",
    "\n",
    "print('EXPERIMENT 4: Testing differently preprocessed data sets with TF.IDF')\n",
    "print('\\n***** Experiment 4 initiated *****')\n",
    "print('training on parts 1-9, N = {}'.format(N))\n",
    "for sp in sorted(setDict):\n",
    "    print('\\n*****',sp)\n",
    "    test_RDD = make_f_wtfiL_RDD(setDict[sp]+'part10') # apply function (j) to create TF.IDF on test data\n",
    "    train_RDD = make_f_wtfiL_RDD(setDict[sp]+\"part[1-9]\") # apply function (j) to create TF.IDF on train data\n",
    "    trainTestModel(train_RDD,test_RDD,N,'') # apply function (i) to train and test the data\n",
    "print('\\n***** Experiment 4 completed *****')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "**Key observations on the output of Experiment 4:** (i) For N =100, training on data with tf.idf yields systematically lower testing accuracy compared to training on hashed normalised word counts (see Experiment 2); (ii) For N=100, testing accuracy is maximised when training on lemmatised data."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}