nltk-session-1-updated-Python3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<img style=\"float:left\" src=\"http://ipython.org/_static/IPy_header.png\" />\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Session 1: Orientation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "Welcome to the *IPython Notebook*. Through this interface, you'll be learning a lot of things:\n",
    "\n",
    "* A Programming language: **Python**\n",
    "* A Python library: **NLTK**\n",
    "* Overlapping research areas: **Corpus linguistics**, **Natural language processing**, **Distant reading**\n",
    "* Additional skills: **Regular Expressions**, some **Shell commands**, and **tips on managing your data**\n",
    "\n",
    "You can head [here](https://github.com/resbaz/lessons/blob/master/nltk/README.md) for the fully articulated overview of the course, but we'll almost always stay within IPython. \n",
    "Remember, everything we cover here will remain available to you after ResBaz is over, including these Notebooks. It's all accessible at the [ResBaz GitHub](https://github.com/resbaz/lessons/tree/master/nltk).\n",
    "\n",
    "**Any questions before we begin?**\n",
    "\n",
    "Alright, we're off!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text as data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Programming languages like Python are great for processing data. In order to apply it to *text*, we need to think about our text as data.\n",
    "This means being aware of how text is structured, what extra information might be encoded in it, and how to manage to give the best results. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is the Natural Language Toolkit?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "We'll be covering some of the theory behind corpus linguistics later on, but let's start by looking at some of the tasks NLTK can help you with. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NLTK is a Python Library for working with written language data. It is free and extensively documented. Many areas we'll be covering are treated in more detail in the NLTK Book, available free online from [here](http://www.nltk.org/book/).\n",
    "\n",
    "> Note: NLTK provides tools for tasks ranging from very simple (counting words in a text) to very complex (writing and training parsers, etc.). Many advanced tasks are beyond the scope of this course, but by the time we're done, you should understand Python and NLTK well enough to perform these tasks on your own!\n",
    "\n",
    "We will start by importing NLTK, setting a path to NLTK resources, and downloading some additional stuff."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function, division\n",
    "# clear output from download\n",
    "from IPython.display import display, clear_output\n",
    "# import: all the nltk basics\n",
    "import nltk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oh, we've got to import some corpora used in the book as well..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** Introductory Examples for the NLTK Book ***\n",
      "Loading text1, ..., text9 and sent1, ..., sent9\n",
      "Type the name of the text or sentence to view it.\n",
      "Type: 'texts()' or 'sents()' to list the materials.\n",
      "text1: Moby Dick by Herman Melville 1851\n",
      "text2: Sense and Sensibility by Jane Austen 1811\n",
      "text3: The Book of Genesis\n",
      "text4: Inaugural Address Corpus\n",
      "text5: Chat Corpus\n",
      "text6: Monty Python and the Holy Grail\n",
      "text7: Wall Street Journal\n",
      "text8: Personals Corpus\n",
      "text9: The Man Who Was Thursday by G . K . Chesterton 1908\n"
     ]
    }
   ],
   "source": [
    "from nltk.book import *  \n",
    "# asterisk means 'everything'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Importing the book has assigned variable names to ten corpora. We can call these names easily: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Text: Sense and Sensibility by Jane Austen 1811>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text2\n",
    "#text3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploring vocabulary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NLTK makes it really easy to get basic information about the size of a text and the complexity of its vocabulary.\n",
    "\n",
    "*len* gives the number of symbols or 'tokens' in your text. This is the total number of words and items of punctuation.\n",
    "\n",
    "*set* gives you a list of all the tokens in the text, without the duplicates.\n",
    "\n",
    "Hence, **len(set(text3))** will give you the total number unique tokens. Remember this still includes punctuation. \n",
    "\n",
    "sorted* places items in the list into alphabetical order, with punctuation symbols and capitalised words first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "44764"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(text3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2789"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(text3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'Achbor',\n",
       " u'Adah',\n",
       " u'Adam',\n",
       " u'Adbeel',\n",
       " u'Admah',\n",
       " u'Adullamite',\n",
       " u'After',\n",
       " u'Aholibamah',\n",
       " u'Ahuzzath',\n",
       " u'Ajah']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(set(text3)) [25:35]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can investigate the *lexical richness* of a text. For example, by dividing the total number of words by the number of unique words, we can see the average number of times each word is used. \n",
    "We can also count the number of times a word is used and calculate what percentage of the text it represents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(text3)/len(set(text3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "147"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text4.count(\"American\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Challenge!** \n",
    "\n",
    "How would you calculate the percentage of Text 4 that is taken up by the word \"America\"?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.13174597728754245"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "100.0*text4.count(\"America\")/len(text4) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploring text - concordances, similar contexts, dispersion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "'Concordance' shows you a word in context and is useful if you want to be able to discuss the ways in which a word is used in a text. \n",
    "'Similar' will find words used in similar contexts; remember it is not looking for synonyms, \n",
    "although the results may include synonyms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Displaying 11 of 11 matches:\n",
      "ong the former , one was of a most monstrous size . ... This came towards us , \n",
      "ON OF THE PSALMS . \" Touching that monstrous bulk of the whale or ork we have r\n",
      "ll over with a heathenish array of monstrous clubs and spears . Some were thick\n",
      "d as you gazed , and wondered what monstrous cannibal and savage could ever hav\n",
      "that has survived the flood ; most monstrous and most mountainous ! That Himmal\n",
      "they might scout at Moby Dick as a monstrous fable , or still worse and more de\n",
      "th of Radney .'\" CHAPTER 55 Of the Monstrous Pictures of Whales . I shall ere l\n",
      "ing Scenes . In connexion with the monstrous pictures of whales , I am strongly\n",
      "ere to enter upon those still more monstrous stories of them which are to be fo\n",
      "ght have been rummaged out of this monstrous cabinet there is no telling . But \n",
      "of Whale - Bones ; for Whales of a monstrous size are oftentimes cast up dead u\n"
     ]
    }
   ],
   "source": [
    "text1.concordance(\"monstrous\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "imperial subtly impalpable pitiable curious abundant perilous\n",
      "trustworthy untoward singular lamentable few determined maddens\n",
      "horrible tyrannical lazy mystifying christian exasperate\n"
     ]
    }
   ],
   "source": [
    "text1.similar(\"monstrous\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "very exceedingly so heartily a great good amazingly as sweet\n",
      "remarkably extremely vast\n"
     ]
    }
   ],
   "source": [
    "text2.similar(\"monstrous\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a_pretty is_pretty a_lucky am_glad be_glad\n"
     ]
    }
   ],
   "source": [
    "text2.common_contexts([\"monstrous\", \"very\"])  # this function takes two arguments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also find words that typically occur together, which tend to be very specific to a text or genre of texts. We'll talk more about these features and how to use them later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "United States; fellow citizens; four years; years ago; Federal\n",
      "Government; General Government; American people; Vice President; Old\n",
      "World; Almighty God; Fellow citizens; Chief Magistrate; Chief Justice;\n",
      "God bless; every citizen; Indian tribes; public debt; one another;\n",
      "foreign nations; political parties\n"
     ]
    }
   ],
   "source": [
    "text4.collocations()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python also lets you create graphs to display data.\n",
    "To represent information about a text graphically, import the Python library *numpy*. We can then generate a dispersion plot that shows where given words occur in a text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEZCAYAAAC5AHPcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFk9JREFUeJzt3X2wXHV9x/H3kpuQADEhhOFKeEgkU402TRQqCKEuYBUc\naMn4hAKCdrQdjYpVkGJHQFtFW4YIKFRtC0hVfKi3Co4PaG5kCAGDJFEDaoQACSFADGCQQB62f3zP\nz3PuZm9y88t93LxfMzv37Hn8/c7ZPZ89v9/esyBJkiRJkiRJkiRJkiRJkiRJ2gMdD9zXD+tZBZy0\nG8ufCfygH8rRX/prv+yqbcCLhmC7kvYAq9i9E/VAeQA4sZdp1wHPAU8Xj18AnwReMCglG16mEiHx\nh+LxAPCRyvS+Bsi5wG39XDaNIHsNdQE0IjWKx0jSAD5NBMZk4B3AMcDtwD5DWK6hfA9OAMYDbwU+\nBrx2CMuiEcgAUX+qARcCK4EngJuA/Ytp1wDfrMz7aeDWYrgOPFyZdijwv8BjxXquKsYfAfykGPc4\ncCNxEtyV8gE8DywB/gY4gAgT6PmJugZcAawDngKWAy8tpl0HXAv8kLia6QYOq2znJcCPgPVEE9Sb\nKtOuI/bF94CNRN1fD6wo1rUa+FAxb52e+2VGsa0NwC+B05rW+zng5mI9i+l7M9Ri4FfAn7eYNgG4\ngTgWq4CPEvtmRlGPVxFXMb/v47Yk7eF6ayr6ALAIOBgYTZxkv1JMGwf8GjiHaNt/vJgPep4oRwHL\ngMuLZfYGjiumHUE0nY0mriIWEif5nZUL4L+BT7QYfz3wtWL4XMoAeR0RMqmJ68VAZzF8HXGSngOM\nAeZXltu3qMs5xAe02UVdZ1SWfZI48QKMBdZW6jgBeHkxXKfcL6OJYL4Q6ABOKMrwZ5X1PgEcRezD\nG4GvtqgvlE1Yo4gwOA54plgn9GzCugH4dlGvw4lj+M5i2jnYhCVpF/V2ol7RNP6FxKf9dKX7SuKT\n6irgLZX56pQnylcRn3b7cnV8OvDzPpQLeg+Qy4grCegZICcSJ8ujW5TlOspghDi5bgEOIer106b5\n/4NoIkrLXtc0/UHg3WzfH1On3C/HE0FT9RXg4sp6v1CZdgpwL61NJUJiA3E8VgDzKtNTgIwi+o1e\nUpn2bmBBMXwuBsgezSYs9aepxKfVDcVjBXFiPaiYfhdwfzH8jV7WcShxQt3WYtpBxNXCaqJZ6ctE\nE9TumEI0NTX7CXA10Sy0jgiB8cW0RlGG5BniRHww8Sn9aMp9sAF4G+U+aNCzWQrgDUQz1iqiieqY\nFuU5uMVyD1JexTWKcibPAvu1WE/VAcAkomnu6hbTJxNXPg9Wxj1E7DPJAFG/egg4mej3SI99KD85\nv5do8nkEuKCXdTxM9CeMajHtk8BWoq1+AnA2u/Yabu743w94Db1/ir6KaBJ6KdFUdH4xvkYEXXU9\nk4A1xD5YSM99MJ6oe2+WEFdTBwJdwNdbzPNIsc1aZdzhxTYHyhPAZuKDQXIYZXiOtC9SqJ8ZIMo1\nhmi/T48Oos/jk5QdygcSHdUQJ+BPEP9r8XYiQGa1WO9dROBcRoTPWODYYtp+xKf9p4lPwee3WL43\nNcqT797AkcTJej3RvNXsKOJKYjTwR2ATEV7J64m+gzFFve4gTua3FHU9q1h2NPCXlM1A1QCgmH4m\nEYhbiQ7prWzvzqIcFxTL1IFTKftvmtfbH7YSYfavxL4/HPgg0b8CccVzSFEe7YEMEOX6HnFCS4+P\nAZ8FvkP57aQ7iH6PUURz02XE/1+sBC4qxqWTT/o0u5X4dtF04tP8w8Cbi2mXAq8gmq++C3yLvn8K\nbhAn36eJT9bXAz8jwunZyjxpfS8g+hRSn80TwL9V5kv9D+uJTu+ziml/IL4OewYRKGuBTxFB07yN\n5Cyi/+Ypoo/hzKZyQ/QlnUb0bTxONDmdDfxmB+vd0b7p67T3EaF9P3Gl9j+Ugftj4ttbjxL9VpKk\nneitQ17ao3gFIu26gWgukkYcA0TadSPxP/ElSZIkSRrBhn1b7qxZsxrLli0b6mJI0kizjLiVzoAZ\n9n0gy5Yto9FotO3j4osvHvIyWDfrZ/3a70Hr/7PqV8M+QCRJw5MBIknKYoAMsXq9PtRFGDDtXDew\nfiNdu9dvMAz7TnSgUbTnSZL6qFarwQCf470CkSRlMUAkSVkMEElSFgNEkpTFAJEkZTFAJElZDBBJ\nUhYDRJKUxQCRJGUxQCRJWQwQSVIWA0SSlMUAkSRlMUAkSVkMEElSFgNEkpTFAJEkZTFAJElZDBBJ\nUhYDRJKUxQCRJGUxQCRJWQwQSVIWA0SSlMUAkSRlMUAkSVkMEElSFgNEkpTFAJEkZTFAJElZDBBJ\nUhYDRJKUxQCRJGUxQCRJWQwQSVIWA0SSlMUAkSRlMUAkSVkMEElSFgNEkpTFAJEkZTFAJElZDBBJ\nUhYDRJKUxQCRJGUxQCRJWQwQSVIWA0SSlMUAkSRlMUAkSVkMEElSFgNEkpTFAJEkZTFAJElZDBBJ\nUhYDRJKUxQCRJGUxQCRJWQwQSVIWA0SSlMUAkSRlMUAkSVkMEElSFgNEkpTFAJEkZTFAJElZDBBJ\nUhYDRJKUxQCRJGUxQCRJWQwQSVIWA0SSlMUAkSRlMUAkSVkMEElSFgNEkpTFAJEkZTFAJElZDBBJ\nUhYDRJKUxQCRJGUxQCRJWQwQSVIWA0SSlMUAkSRlMUAkSVkMEElSFgNEkpTFAJEkZTFAJElZDBBJ\nUhYDRJKUxQCRJGUxQCRJWQwQSVIWA0SSlMUAkSRlMUAkSVkMEElSFgNEkpTFAJEkZdnNAGls3MX5\nz4XGVbu3TUnScLC7VyCNAZ4fgJkzoaMDxo+HadNg7txyWhru7oZ58+JvUh2umj+/9fP58+Mxb17r\n5VqNnz8/ytC83eZtz5nTukzpeXd3zDNmTAynsqTh5vXPnQu1WuyX9OjsjHXMnduzjmn+tI3u7nJf\npulpG2metI40T3WfV9ebjse0abFcenR0tK5jtW5z5pT7NJU3Hcfqvq4uUx3X2RnzjRoF48ZFGSZN\ninFp3pkzY1wq4/z5UffOzrLMc+fGY86cslzjxpWvuVSWOXNifdVl5s0rt5HWn15DM2f2rFO13nPn\nxvTOzhg3c2bUo7OzPI6dnbHspEnxSMcobTft67R8mmfSpLLcqRxp3nnzyrpW93t1n6XjlcpaLX9a\nR3qdpLpUt1Ndb6vj33xs0/FNdWp+//T2Pm4en9aRlq8+0vFLj2nTer4f0us9lS3to3SMUr3UU8eO\nJzfOBzZB7SpoXAH8BdROgsaJwN8V8/wLcCrwLPC3UHsMGqcBHwXGAOuBM2N8j3UfCFwDHFaMOA9q\ni1qV4t57YetW2LgRnnkGNmwopy1YEH+7u+Hmm2HyZKjXy3FpuKqrC847b/vnXV3xfNUquPrq7Ze7\n+ebtx3d1wdKlMGtWz+1Cz20vWdK6TOl5d3fMs3lzzxf9k0/G8JNP9lx/qvfWreW61q2L+caOjX2U\n6pjmnzUrttHdXe7LND1tI5U1rePBB2Nc2l5Vd3dM37ABnn4a1q4tp1XLVa1jdf8sWQKrV8c+Tccg\nHUco93V1f1TrtG5dzLttG2zaFGVpNGLc1Kkxb3rtLFgQZezqirpv3BgBvHZt1BViHR0dUa7nnotx\nGzeWx33JEtiyJeZJy0ycGHXYujX2X/U1tHp1WdZUpur+3bgxluvqinJu2xZ1gljXc8/Fsun13tUV\n4xcsiO0++miM7+go69nVVc6f9uPq1WWgr14d69i0qVzneef13GfpeHV1la+XND0ds8mTy9f02LHl\nfkjbqb7Hmo9/9X1UfU2sWhV1an7/9PY+bh7f1RXrmDx5+3nr9bK8AM8/X77n0vtjw4byvb9gQeyj\nLVvKY6vt7ewK5KfA8cXwUcC+0OgA5gAL4zl3QG12Me+7inlvg9oxUHsFcBNwQTG+Vln3Z4EroPZK\n4I3Al3a3MpKkwbOTKxB+DhwJjfHAJmAJESTHA+8HnofaLcW8dwN/XQwfCo2vA53EVcj9Ldb9GmBG\npVVrPDT2gdofm2fctu2SPw03GnWgvpNiS9Kepbu7m+7e2vsGyE4CpLYZGg8A5wKLgOXAicARULsX\nGpsrM2+rrO8q4N+hdjM0Xg1c0mrlwNFQe35nhdxrr0v+1CRSq+14XknaE9XrdeqVNr1LL710wLfZ\nl07024APE01WtwH/ANyzk2VeADxSDJ/byzw/JK5iCo3ZfSiLJGmY2FkTFkRoXET0dTwLjWeLcdDz\nW1WNyvNLgG9AYwPwE+DwFvO8H/gcNJYV5VgIvKdVAWbMiE7CceOig2x2JWpOOCH+1uvwxBM9O9Va\ndbwBnH566+fp78qVrZc79dTW69p//51v96ijWk9Lz+t1uPVWuOuuGJ44McbPnh3D1TrX67BsWXQa\njhpVjp88GaZPhwMPhFe/evv50zbqdbj88rKzMW0vbePWW8t1pE70tJ+by3799bHc0qUwZUo5bfHi\n1nWEsm5btpTbTPs+Hcfm7aRlquOuvTaOyTXXxLfXOjvhqadi3PTpMd+MGbBmTdRl6dLYzt13w777\nxutpypSoK8Djj8f0LVtins2bY5503I86KtY/YUK5zJQpsHBhbGPChJ6voYULe9YplWnLllh+5cpy\nm+vXw4oV5XqnT4/p9TpceWW5j2bPjmM5ZUrUJ41fvz7KcPrpsHx5lD2Ve+HCKBvE8mvWxHar+726\nz9JxSvuiWv50zNJrCaLMaT+k7VTfY83Hv3p8q6+JlSujTr29R5o1jz/99HKftZLegxD7oPp+WLYs\nXiPpvX/CCbGPnnoq/qZ9o55GQoNQo9HI+vavJO2xatHeP6DneP8TXZKUxQCRJGUxQCRJWQwQSVIW\nA0SSlMUAkSRlMUAkSVkMEElSFgNEkpTFAJEkZTFAJElZDBBJUhYDRJKUxQCRJGUxQCRJWQwQSVIW\nA0SSlMUAkSRlMUAkSVkMEElSFgNEkpTFAJEkZTFAJElZDBBJUhYDRJKUxQCRJGUxQCRJWQwQSVIW\nA0SSlMUAkSRlMUAkSVkMEElSFgNEkpTFAJEkZTFAJElZDBBJUhYDRJKUxQCRJGUxQCRJWQwQSVIW\nA0SSlMUAkSRlMUAkSVkMEElSFgNEkpTFAJEkZTFAJElZDBBJUhYDRJKUxQCRJGUxQCRJWQwQSVIW\nA0SSlMUAkSRlMUAkSVkMEElSFgNEkpTFAJEkZTFAJElZDBBJUhYDRJKUxQCRJGUxQCRJWQwQSVIW\nA0SSlMUAkSRlMUAkSVkMEElSFgNEkpTFAJEkZTFAJElZDBBJUhYDRJKUxQCRJGUxQCRJWQwQSVIW\nA0SSlMUAkSRlMUAkSVkMEElSFgNEkpTFAJEkZTFAJElZDBBJUhYDRJKUxQCRJGUxQCRJWQwQSVIW\nA0SSlMUAkSRlMUAkSVkMEElSFgNEkpTFAJEkZTFAJElZDBBJUhYDRJKUxQCRJGUxQCRJWQwQSVIW\nA0SSlMUAkSRlMUAkSVkMkCHW3d091EUYMO1cN7B+I127128wGCBDrJ1fxO1cN7B+I127128wGCCS\npCwGiCQpS22oC9AHS4FZQ10ISRphlgGzh7oQkiRJkiRJklo6GbgP+C3wkSEuy86sApYD9wB3FeMm\nAT8CfgP8EJhYmf+fiHrdB7y2Mv5I4BfFtM9Wxu8N3FSMXwwc3t8VaPJfwLqiLMlg1eecYhu/Ad6+\n+1VpqVX9LgFWE8fwHuCUyrSRVr9DgQXAr4BfAu8vxrfDMeytbpfQHsdvLHAn0fe7AvhUMb4djt2g\nGQWsBKYCo4mdOWMoC7QTDxAHuOozwAXF8EeAy4rhlxL1GU3UbyXllxnuAl5ZDH+PCFGA9wCfL4bf\nAnyt/4re0vHAy+l5gh2M+kwCfke8OSZWhvtbq/pdDPxji3lHYv06KTtP9wN+Tbx/2uEY9la3djp+\n+xR/O4gT/Bza49gNmlcB3688v7B4DFcPAAc0jbsPOKgY7iyeQ3xaqF5RfR84BnghcG9l/BnAtZV5\nji6GO4DH+6XUOzaVnifYwajPW4FrKstcWyw3EKayfYB8qMV8I7V+VV3Aa2i/Ywhl3drx+O0D/Ax4\nGcP02A3X/wOZAjxceb66GDdcNYBbgSXAu4pxBxHNJBR/08E/mKhPkurWPH4NZZ2r+2ML8BTbX/EM\ntIGuzwE7WNdgeR/x1cf/pPzkNdLrN5W42rqT9juGU4m6LS6et8vx24u4qlhH2Vw3LI/dcA2QxlAX\nYBcdR7yQTwHeSzSRVDUYeXXakXarD8Qnr2lE88ha4PKhLU6/2A/4FvAB4A9N00b6MdwP+CZRt420\n1/HbRtTjEOCvgBOapg+bYzdcA2QN0VmWHErPZBxu1hZ/Hwe+TbQ7riMuNSEuJx8rhpvrdghRtzXF\ncPP4tMxhxXAHMAH4ff8Vv08Guj7rW6xrMI/7Y5RvzC9Rth2P1PqNJsLjy0QzD7TPMUx1u5Gybu12\n/CCuDG4hOsPb5dgNig6iA2cqMIbh3Ym+DzC+GN4XuJ34JsRnKNsmL2T7Tq8xxCem31F2et1JtE3W\n2L7TK7VNnsHAd6LD9n0Eg1GfScD9RPPD/pXhgTCVnvV7YWX4g8BXiuGRWL8acANwRdP4djiGvdWt\nXY7f5Mo6xwE/BU6iPY7doDqF+IbFSqKjaLiaRhzApcTXClNZJxH9Iq2+dncRUa/7gNdVxqev3a0E\nrqyM3xv4OuXX7qb2cx2afRV4BHieaCt9B4NXn3cU439LfKVwIDTX753ESWk50YbeRdnGDCOvfnOI\nZpCllF9rPZn2OIat6nYK7XP8ZgI/J+q3HDi/GN8Ox06SJEmSJEmSJEmSJEmSJEmSpOQK4rYWyQ+A\nL1aeX078g1mOOvDdXqbNIf5h697i8a7KtAOLaXcX872JuEX3jzPKcFHGMpKkPngD8RsHELfoWULc\nFSBZRHl7i51pvsVPndYB0gk8SHl78QOK7b6+eH4GPUPs+8CxfSxDs+Z7WUmS+snBwEPF8EzgOuKE\nPZH4z9sNxC1yTiL+03c5cdfWMcUyq4jbQ9xN/EbCycQVxd3Ej/G0CpBPED9kVHUicfuJWUS4PEb8\nt/THiBC4j7gtxcuI32u4h/jv6SOK5c8irlruIW6nvVdRri3FuC/3cX9IknbB/cQN4N4N/D3wceI2\nF8cBC4kgeQiYXsx/PWWz1wPAh4vhscV86aR+E/CdFtv7FnBa07h0YzqIW0FUbyGxAHhFMXwl8LZi\nuKPY5oxiO6OK8Z8Hzi6GvQLRsDNc78Yr5VhENBEdC9xRPI4lfqDsduDFRFCsLOa/nrhddpKawF5S\nzPe74vmNlDeoa9bb+DStt+l3EP0aFxD3ItpEXB0dSTSD3UNczUzbwfqlIWWAqJ3cTlxtzCRuIreY\nMlAWtZi/Rs/fVXiml/X2FgIriBN+1ZHETTV35qvE1cuzxJ1S028+XE/8tszLiSD7eB/WJQ0JA0Tt\nZBFwKtGE1CD6PSYSVyCLiDuZTqVsmjqbaNpqdl8x34uK52/tZXufA84l+jsgOtEvI/o4dmYacZVz\nFfB/ROj9GHgj8e0tiDuwpt9t2Ew0dUnDhgGidvJL4iS+uDJuOfAk8QNcm4jbVX+jGL+F8neiq1ci\nm4h+lFuITvR1tP4FuEeJTu8vEh3utxMd87dU1tnbL8e9uSjvPUSH+g3FOv6ZuF33suJv+hGhLxRl\nthNdkiRJkiRJkiRJkiRJkiRJkiRJkiRJA+//AbcetJRNG4P/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x109d3d990>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy\n",
    "%matplotlib inline\n",
    "text1.dispersion_plot([\"whale\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "906"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text1.count('whale')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Challenge!**\n",
    "<br>\n",
    "Create a dispersion plot for the terms \"citizens\", \"democracy\", \"freedom\", \"duties\" and \"America\" in the innaugural address corpus.\n",
    "What do you think it tells you? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAEZCAYAAAAt5touAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuUXFWd6PHvMYmCgjSvJaJiePhAjQmCIkwwFWR83aDJ\n1RF8IOBao45G1MFRRhxJ9OpEHS+NKDI6V3mIbzAD0QF8pIKCiIHQvAJOhKCgIEEQ1AEE9v1j70Od\nPn2qH6Gre3f6+1mrVp3aZz9+53RV/ersc7oKJEmSJEmSJEmSJEmSJEmSJEmSVHMQcP049LMReOmj\naP8m4IJxiGO8jNd+GauHgT0mYVxJGhcbeXTJoFduAg7usu404H7gnnS7GvgE8MQJiSwvs4mJ6N50\nuwn4YGX9aJPUUcBPxjk2TRGPmewApGGEdJtKAvBJYlLaCTgaeDFwMfD4SYxrMl/r2wHbAm8APgK8\nbBJj0RRjktJUVADHARuATcA3ge3Tui8A36nU/STww7TcAn5TWfc04Bzg96mfk1P5nsCPU9kdwFeJ\nb7RjiQ/gAWAt8GpgR2LCgsFHBgVwInA78EfgKuA5ad1pwKnAhcSjsjawW2WcZwM/AO4kTtf9XWXd\nacR98X3gT8RtfxVwXerrFuDYVLfF4P2ydxrrLuAa4NBav58HVqV+LmX0U3aXAtcCz2tYtx1wBvFv\nsRE4nrhv9k7bcQDxaOwPoxxLknqu27Tae4BLgF2BWcQ38q+ldVsDNwBHEs+13JHqweA34xnAAPCZ\n1OZxwN+kdXsSpxlnEY+G1hATyUhxAXwF+FhD+enAN9LyUXSS1MuJiaycDnwWsEtaPo2YCOYDjwX6\nK+2ekLblSOKHzXlpW/eutL2b+OYOsBXwu8o2bgfsk5ZbdPbLLGLyPw6YCSxMMTyz0u8mYD/iPvwq\n8PWG7YXOdN8MYsL5G+DPqU8YPN13BvDdtF1PJ/4N35rWHYnTfZIy1C0ZXFcrfzLxqKWcGXgR8RP3\nRuCwSr0WnTfjA4if2kczm7AYuGIUcUH3JLWCeEQEg5PUwcQ35P0bYjmNTvKF+Ab+IPBU4nZdVKv/\n78TptLLtabX1NwNvY+j5sRad/XIQMZlVfQ04odLvFyvrXgmsp9lsYiK6i/j3uA5YWllfJqkZxPN4\nz66sexuwOi0fhUlq2nK6T1PRbOKn7rvS7Trim/eT0vrLgBvT8re79PE04pv2ww3rnkQ86rmFOAV3\nJnG67tF4CnFaru7HwOeIU2i3ExPNtmldSDGU/kx8s9+VeLSxP519cBfwRjr7IDB4Cg/gtcQpv43E\n6bwXN8Sza0O7m+kcjYYUZ+l/gG0a+qnaEdiBOI35uYb1OxGP4G6ulP2auM80zZmkNBX9GngF8TxU\neXs8nSOAdxGnx34LfKBLH78hnt+Z0bDuE8BDxHMn2wFHMLbXSv1ij22AQ+h+NHAycfrsOcRptX9K\n5QUxmVb72QG4lbgP1jB4H2xL3PZu1hKPCncGVgLfaqjz2zRmUSl7ehqzVzYBfyV++CjtRidBT7WL\nZzSOTFLK3WOJ51PK20ziOahP0LmIYGfixQkQ3+Q/RvxfpLcQk9Tchn4vIya1FcQEtxVwYFq3DfGo\n5R7ip/l/amjfTUHnDf5xwL7EhHAncSqwbj/iEdEs4C/AfcQEWXoV8VzOY9N2/YyYML6XtvXNqe0s\n4IV0psyqSYa0/k3EpPsQ8SKEhxjq5ymOD6Q2LWARnfNp9X7Hw0PEhPlx4r5/OvA+4vkuiEduT03x\naJoxSSl33ye+aZa3jwAnAefSuertZ8TzUDOIU3MriP+ftAH4UCor3+DKT+UPEa9a24t4VPIb4PVp\n3XLgBcSpvvOAsxn9p/lAfIO/h3iEcDrwC2IC/J9KnbK/JxLP8ZTn0DYBn67UK88H3Um80OHNad29\nxEu5Dycmrd8B/0pMZvUxSm8mnk/7I/Gcz5tqcUM8t3co8VzTHcTpuSOAXw7T73D7ZrTr3k38YHAj\n8YjzLDpJ/UfEqwJvI55HlCRloNtFGNK04ZGUlK9eTK1JU4pJSsrXVPzGDUmSJEmSNKmc807mzp0b\nBgYGJjsMSZpqBohfy9UTnpNKBgYGCCFkfzvhhBMmPYYtIUbjNM7cb1MlTpr/D3HcmKQkSdkySUmS\nsmWSmmJardZkhzCiqRAjGOd4M87xNVXi7DUvnOgIaX5VkjRKRVFAD3OJR1KSpGyZpCRJ2TJJSZKy\nZZKSJGXLJCVJypZJSpKULZOUJClbJilJUrZMUpKkbJmkJEnZMklJkrJlkpIkZcskJUnKlklKkpQt\nk5QkKVsmKUlStkxSkqRsmaQkSdkySUmSsmWSkiRlyyQlScqWSUqSlC2TlCQpWyYpSVK2TFKSpGyZ\npCRJ2ZrEJBXeDuGItHwUhCdX1n0Jwt6TEpYkKRvFZAcQhdXA+6G4fDKDCCFM4vCSNPUURQE9zCUT\neCQV3gJhAMKVEM6AcAKEYyG8FtgPOAvCFRC2gtCGsC+EQyGsS7cbINyY+to31VkL4XwIu6TyNoQV\nEH6e6s9P5c9NZetSDHs1Rdhud24A/f2Dy8rb0qWd+/7+Tr3+/sF9Vcur7auPodNPfZx6vSVLmvds\nPeZSWb9cV9++ettq3SZLlsRY58yJ9+V4S5Y0b39ZVm5HWb8eT33salm1PnT2ffVxvU23bajXKWOr\nxtIttnrc5bj1vqv7oFre9Dyq9j2W/dK0ffX9NNz+6DZOXVO74epDfG6M5u863Dil+r7sVt7UvlpW\nPj9HW79p3XDb0+3139RH02uv/j5QPrfmzx/6flCON51MUJIKzwWOBxZCMQ94T7kCirOBtcAboXgB\nFPfFcgIU50GxT7xxJfBpCDOBk4HXQrEf8BXg453+mAHF/sB7gRNS+TuAk1I/+wK3NEVZf9KsXNn8\n5rJqVed+5cpOvZUrB/dVLa+2rz6GTj/1cer1Vq9u3rv1mEtl/fFKUqtXx1jXr4/35XirVzdvf1lW\nbkdZfyxJqlofOvu++rjeZrRJqoytGku32Opxl+PW+67ug2r5cEmqvo82J0nV99NkJan168cvSdX3\nZbfykZJO+fwcbf2mdcNtT7fXf1MfwyWp8n2gfG6tXTv0/aAcbzqZOUHjHAx8C4o/xIfFXTGfDDLM\n4WL4APAXKL4A4XnAc4Efpj5mAL+tVD4n3V8BzE7LlwDHQ3hqXF9s2PxNkSRNlIlKUoGR5yy7nBAK\nhwCvBV6SCgrgWigO7NLP/en+IR7ZvuLrEC4FFgHfjxdtFEOOS9rtZZXlFtAaIWRJml7a7TbtkQ6p\nx9FEJakfA9+F8H/j0VTYIZWXiete4IlDm4WnA58HXgZFmXxuAHaG8GIoLoUwC3gGFNd1Hz7sAcWN\nwMkQdgPmAEOSVKu1rLI8hq2TpGmi1WrRqrxBLl++vKfjTVCSKq6D8HFgDYSHgHXARjpHT6cBp0L4\nC1AeIRXAkcAOwMpU9VYoFkF4HfBZCNulbTgRaEpSZf+vh/Bm4K/A7+icw5IkZWyijqSA4gzgjC7r\nzqFzLglgYbq/HPhoQ/0BYEFD+cLK8iZgj7S8AlgxUoT1o6fFi2HevKH1Nm2KdTdtgr3SdYLz5kFf\n3+C++vo65dV+6o8XLYr91Meq11u4kEbVuBcvHlq/XN90dFgvG+4IcuFCeMpTYM0aWLCgM97223e2\nt95PWVZuR3Wfdhu7vj31fVVdv2jR0DbdtqFepynebo/rcW/a1Fy/uv+r5cPt17LNaPdLU1/1/TTc\n/hhNTMO1G87ee499vG7l9X3ZrXyk5/XChSM/P4bbtpG2p9z3Y3k+NZWV7wMb0hnz/fYbXKfsv9t+\n2VJl8n9SWfD/pCRpjLag/5OSJGlsTFKSpGyZpCRJ2TJJSZKyZZKSJGXLJCVJypZJSpKULZOUJClb\nJilJUrZMUpKkbJmkJEnZMklJkrJlkpIkZcskJUnKlklKkpQtk5QkKVsmKUlStkxSkqRsmaQkSdky\nSUmSsmWSkiRlyyQlScqWSUqSlC2TlCQpWyYpSVK2TFKSpGyZpCRJ2TJJSZKyNcYkFZZBOLYnkUiS\nVDPWI6nQkyhGP/zMyR0/H+32+Pc11j7b7fGNQ5Orv3/8/6bd+irL58/vjNvfHx9XYymXqzH2Ip7x\n1LQfly7tlNVvZVz112G1/XR+nY0iSYXjIdwA4SfAs1LZnhD+C8JaCBdBKMtPg3AKhJ9B+BWEFoTT\nIVwH4SuVPt8A4SoIV0NYUSl/BYTLIVwJ4QepbBmEMyH8FDgdwtPTmJen2wGV9h9M/V4J4RMQ9oh1\nHln/jMGPpy6TlMbbypUTn6TWru2Mu3JlfFyNpVyuxtiLeMZT035ctcoktblGODIJ+wKHAXOBWcAV\nwOXAvwPvgGIDhP2BU4CXpkZ9UBwA4dXAucABwHXALyDMBe4AVgAvAO4GLoTwGuAS4IvAQVDcDKGv\nEsizgflQ3A9ha+Bv0/IzgK8BL4TwSuDVwIuguC+2L+6G8Mc4bjEAHA18+VHsL0nSBBpp+uwg4Jz4\nps99EM4FtgIOBL5dmf17bLoPwHlp+RrgNiiuTauuBWanWxuKO1P5WcBLgIeAi2KCgphgHunz3JiU\nHhnrcynhPQQ8I5UfAnw5xVpt/x/A0RD+EXg98MJuG7ts2bJHllutFq1Wq1tVSZqW2u027Qk8tBsp\nSQWgqJU9Brgbin26tHkg3T8M3F8pfziN99da/Xr/Tf5SWX4f8DsojoAwA7hvmFgBzgZOAH4MrIXi\nrm6DVJOUJGmo+gf45cuX93S8kc5JXQQshrAVhG2BQ4kJ4yYIr4tVQgHh+aMcLwCXAQsg7JiSzOFA\nG7gUeAmE2anqDl36eCJwW1p+CzAjLf+AeMS0dWq/fbwv7gcuAL4AVM6LSZJyN8KRVLEOwjeBAeD3\nxAQTgDcBX4DwYeK5qq8DV6VG1SsAG64GLG6DcBywmnjkswqKNEUY3gacA+ExwO3Ayxv6OQU4G8Jb\ngPOBP6V+L4AwD1gL4QHge8CHU5uvAUuAC4ff3qljPGciy77G2qezoVuWxYth3rzx7bPbc6Qs32+/\nzrh9ffDgg0NjWbx4cIy9iGc8Ne3HRYuGH7u6run1OJ1fa6OZatsChPcD20JxwnCVQpjkK+wlaYop\nigJ6mEumwf8dhe8CuwMHT3YkkqSxmSZHUqPikZQkjVGvj6T87j5JUrZMUpKkbJmkJEnZMklJkrJl\nkpIkZcskJUnKlklKkpQtk5QkKVsmKUlStkxSkqRsmaQkSdkySUmSsmWSkiRlyyQlScqWSUqSlC2T\nlCQpWyYpSVK2TFKSpGyZpCRJ2TJJSZKyZZKSJGXLJCVJypZJSpKULZOUJClbJilJUrZMUpKkbPUw\nSYVjIFwH4cxx7rcNYd/x7VOSlKNeHkn9A3AIFEd0isLMceg3pNukWroU2u3O4/7+znK1fLy12/FW\njlcfa7zGLvspx+vWb7VetWwscYw0xkQqY1iyZGjZcPXHum609ZYuba43Ut+jGbv6nC2Xuz2Pq/uj\nWyzVv+HmvB7K9uVY5fO8vLXbMGfO4D77+2P9ss7SpfF+/vzmWOqvm2ps9X1djt/0Wihf/9Xlso8y\nnqZ9NNLrtmxX3rfbcVvK2+67D17X1MeWpkdJKpwK7AGcD+FuCGdA+ClwOoSdIHwHwmXpdmBq8wQI\nX4bwcwhXQHh1Kt8awjfSUdk5wNaVcd4A4SoIV0NYUSn/E4RPQbgGwg8gvBjCGgi/gnDoeGzhqlWD\nnxQrV3aWJyJJleOZpMZXGcPq1UPLhqs/1nWjrbdqVXO98UhS1edsudzteVzdH91iqf4NN+f1ULYv\nxyqf5+Wt3Yb16wf3uXJlrF/WWbUq3q9d2xxL/XVTja2+r8vxm14L5eu/ulz2UcbTtI9Get2W7cr7\ndjtuS3m7+ebB65r62NL0KEkV7wB+C7SAE4G9gZdC8Sbgs7GseBHwOuA/UqPjgR9BsT9wMPBpCI8n\nHpH9CYrnACcAaaov7AqsABYC84AXQnhN6uvxqa/nAfcCH019LknLkqQpYDym34ZTpPtzobg/LR8C\n7F2Zsds2HkXxMuBQCO9P5Y8DdgMOAk5K3V0dj5wogBcCbSjujOvCWcBLgP8EHoDigtTP1cB9UDwU\nj6yY3S3YZcuWPbLcarVotVpj3mBJ2pK1223aE3jY1uskVfpLZbkA9ofigcFVAsD/huK/G8oLhqqf\nlyoqZX+tlD8MpLGKh4c7L1ZNUpKkoeof4JcvX97T8SbjEvQLgWM6D8PctHBBrXyftHAR8MZU9jzg\n+cRkdBmwAMKOEGYAhwNrehi3JGmC9fJIKnRZPgb4PISBNP4a4J3Ax4D+NJ33GOBG4NXAF4CvxAsn\nWA+sjd0Ut0E4DlhNPIpaBcV5DeMNF8tmW7QIqrOBixd3lns5S1j23dfXPNZ4jV32M1J/TfXGGkNO\ns6plLAsXDi0brv5Y14223qJFzfVG+3cZTvU5Wy53ex5X90e3WB7t66GsNzDQeVw+zwHmzYOzzx5c\nd/Fi2H57WLAgPt6wAfbaCx58cHCdelxNMTft676+OG697qZNncfV5UWL4NZbYzxlu3q89b7q6+fN\nGzzuD3/YqXfrrZ06dTm9jsZT0zTadBVCmPQr2yVpSimKAnqYS/zGCUlStkxSkqRsmaQkSdkySUmS\nsmWSkiRlyyQlScqWSUqSlC2TlCQpWyYpSVK2TFKSpGyZpCRJ2TJJSZKyZZKSJGXLJCVJypZJSpKU\nLZOUJClbJilJUrZMUpKkbJmkJEnZMklJkrJlkpIkZcskJUnKlklKkpQtk5QkKVsmKUlStkxSkqRs\nmaQkSdkySUmSspVhkgrLIBw7zPq5EF5ZeXwohA/2PCxJ0oTLMEkRRli/D/CqzsPiPCg+2cuAANrt\neCuXq/cA/f2dOtW6AEuXxvVNfdb76TYmwJIlgx9X25f9V8dpGrNuNHXq43UrX7q0eX017nKb6mXd\nYmoac7Rlw5WPNMZI/ZTtm/4Gw/XX9DzqptpfWbd8ntXHHM5otm9z22xO3+Mdw5Y6PjS/3puM5XU8\n1WSSpMLxEG6A8BPgWalsNYR90/JOEG6CMAv4KHAYhHUQXg/hKAgnp3o7Q/gOhMvS7cBUviDVXwfh\nCgjbjDXCkZLUypXdk9SqVXF9U5/1frqNCbB6dfckVfZfHadpzLrR1KmP16181arm9dW4y22ql3WL\nqVdJaqQxRuqnbN/0NxivJFXtr6xbPs/qYw7HJDU1x4fRJ6mxvI6nmpmTHUBKRIcBc4FZwBXA5eXK\nwXWLv0L4F2BfKI5JVY6sVDgJOBGKiyHsBpwPPAc4FngnFD+D8Hjg/l5tjSRp/GSQpDgIOAeK+4D7\nIJw7Qv0i3ZocAuxdyW3bQngCcDFwIoSz0li3NjVetmzZI8utVotWqzXabZCkaaHdbtOewMPMHJJU\noDnpPAjMSMtbjbKvAtgfigdq5Z+EsAr4X8DFEF4OxQ31xtUkJUkaqv4Bfvny5T0dL4dzUhcBiyFs\nBWFb4NBUvhFI56R4XaX+PcC2lcfVBHchcEznYZiX7veE4looPgX8gkfOe0mScpbBkVSxDsI3gQHg\n98BlxKOrfwO+BeFtwPfozOGtBo6LF0Hwr6m8XHcM8HkIA8RtWwO8E3gPhIXAw8A1wH+NNcrqzF+5\nXC1bvBjmzWtuu2gR7LVX9z67zSrWyxcu7B5HX18njmpMIxlNnW7x1MsXLWpeX427vB8Y6L791Zia\nxhxt2XDlI40xUj9l+6a/wXD9Nf39uqmPUZaVz7PqmMPZnFnr0bbp5Yz4ZM+2T/b4MPrny1hex1NN\nt3M701EIYaSr3yVJVUVRQA9zSQ7TfZIkNTJJSZKyZZKSJGXLJCVJypZJSpKULZOUJClbJilJUrZM\nUpKkbJmkJEnZMklJkrJlkpIkZcskJUnKlklKkpQtk5QkKVsmKUlStkxSkqRsmaQkSdkySUmSsmWS\nkiRlyyQlScqWSUqSlC2TlCQpWyYpSVK2TFKSpGyZpCRJ2TJJSZKyZZKSJGXLJCVJylaPk1RYDOFh\nCM/qUf/7QjipN31LkiZb0dvuwzeBrYEroFg2zn3PhOLB8ewwhDCO3UnSlq8oCuhhLunhkVTYBtgf\nWAoclspaENZAWAnhVxBWQDgCwmUQroKwR6q3M4TvpPLLIByYypdBOBPCT4EzICyAcF5nvPCV1M8A\nhCWp/BQIv4BwTWw/sv5+aLfjcrvduU2E6ri96HdLMNHbMpX2XX9/vI21zdKl8Va2Lfupl8HQ/dFu\nx3r1svJ1NNz+69bXkiXDx1xvV93ueixN8S9d2umjjHOXXeL9kiWw++5DY6+2L7evfFz2Ud3m6q2/\nH+bMictz5sQx5s/vrCv39Zw58b66bs6c+Li/P7Yr/1b1/qfS83QsZvaw79cA50Pxawh3QHhBKn8+\n8GzgLuAm4EtQvAjCMcC7gfcBJwEnQnExhN1iPzwntX82MB+K+2PSe8S/xD6L58eHoS+VHw/FXRBm\nAD+EMAeKq4cLfOVKuPtuaLUG/+FbrS4NxlG73Rl3PMcb7/4m00Rvy1TadytXxvv3vndsbTZujMuz\nZ8e2ZT8bNw4ue+97h+6PdhtWrYLPfW5wWbsdX0fQff9166ts1029XXW767GUcVfjX7UKdtop9lG+\n3m+/Pa5bvRruuafz2i/HqbYvYygfl9ta3eZ6vOvXd+5/8xu4777BiXDjRrjlFrj3Xrjtts669eth\n5sx4u/JK6EvvbDvtNLj/8j1rS9PLJPUG4MS0/O30eBXwCyhuj8VhA3BBqnMNsDAtHwLsDY9Mv20L\n4QnEgnNjghripTxyxAZQlE+VwyD8PXFbn0xMdsMmKUlSHnqUpMIOxITzPAgBmEFMMN8Dqgnm4crj\nhyvxFMD+UDxQ6xfgL8MMXJsXDbsDxwL7QfHHOB3IVt0aL1u2DIifaDZubAGtYYaSpOmn3W7TnsC5\nxV4dSb0OOAOKf+gUhTbwklG2vxA4Bvi31HYuFAMjtPkB8C7idCFpuu+JwJ+BeyA8CXglsLpbB2WS\narfjFIckabBWq0WrMq+4fPnyno7XqwsnDge+Wys7O5V3u4QuVNYdA+yXLoC4Fnh7rV5Tm/8DbA/h\naghXAq2U2NYB1wNnAT/dvM2RJE2GHh1JFQc3lJ0MnFwrW1hZXgOsSct3EhNavY9ayh7U5s/AUQ1t\njh5NxFWLF8O8eXF5ok9EluON97hb0gnVyfqbTAWLF29emw0b4vJeew3uZ8OGoWX1/dFqwaZNQ8v6\n+jqvo2669XXrrWNrV93uRYua11Xj37Sp00f5ej/11Fg2MBAvUOg2RlleXsBQ3dZu29zXB3feGeue\nfXbcp3fc0WkLcV+vWQMLFnTGL9ttt10cf/vt4SlPGboPRrOvp6oe/5/UlOL/SUnSGE3h/5OSJOnR\nMUlJkrJlkpIkZcskJUnKlklKkpQtk5QkKVsmKUlStkxSkqRsmaQkSdkySUmSsmWSkiRlyyQlScqW\nSUqSlC2TlCQpWyYpSVK2TFKSpGyZpCRJ2TJJSZKyZZKSJGXLJCVJypZJSpKULZOUJClbJilJUrZM\nUpKkbJmkJEnZMklJkrJlkpIkZcskJUnKlklqimm325MdwoimQoxgnOPNOMfXVImz10xSU8xUeOJO\nhRjBOMebcY6vqRJnr5mkJEnZMklJkrJVTHYAGbkSmDvZQUjSFDMAzJvsICRJkiRJkiRpCngFcD3w\n38AHJ2C8pwGrgWuBa4BjUvkOwA+AXwIXAn2VNv+c4rseeFmlfF/g6rTupEr544BvpvJLgac/inhn\nAOuA8zKOsw/4DrAeuA7YP9M4/5n4d78a+FrqN4c4vwzcnvosTVRcR6Yxfgm8ZTPi/DTx7z4AnANs\nN8lxNsVYOhZ4mLhvJzPG4eJ8N3F/XgN8MoM4p70ZwAZgNjCLePHE3j0ecxc6Jxm3AW5IY34K+EAq\n/yCwIi0/J8U1K8W5gc4FL5cBL0rL3ycmXIB3Aqek5cOAbzyKeP8ROAs4Nz3OMc7Tgbem5ZnEN6rc\n4pwN3Eh88UJ8AR+ZSZwHAfsw+A1rIuLaAfgVMQH2VZbHEuff0rlKeUUGcTbFCPHD6fnATXSSVG77\nciHxg8ms9HjnDOKc9g4gPnFKx6XbRFoJHEL8hPKkVLZLegzxE0z1CO984MXAk4mfeEqHA6dW6uyf\nlmcCd2xmbE8Ffkh88pZHUrnFuR3xzb8utzh3IH4g2T71cR7xDTaXOGcz+A1rIuJ6A/CFSptTU7ux\nxFm1BPhqBnE2xfht4PkMTlK57ctvAQc31JvUOKf7/0k9BfhN5fEtqWyizCZ+mvk58Q3h9lR+O503\niF1TXKUyxnr5rXRir27Xg8AfGTzFMFonAv9EnKIo5Rbn7sQXwFeAK4AvAU/IMM4/AJ8Bfg38Frib\n+Kk1tzhLvY5rx2H62lxvJX6azy3O16Q2V9XKc4oR4BnAS4jTc21gvxzinO5JKkzi2NsAZwPvAe6t\nrQtMbmwAi4DfE89Hdft/uhzinAm8gDi18ALgzww9Gs4hzj2B9xI/mOxK/Pu/uVYnhzib5BpX1fHA\nA8RzfTl5PPAh4IRKWa7/nzqTeKT/YuKH029NbjjRdE9StxLniktPY3CW75VZxAR1JnG6D+Kn1V3S\n8pOJCaIpxqemGG9Ny/Xyss1uabk8R/OHMcZ4IPBq4vTE14nTAGdmGOct6faL9Pg7xGR1W2Zx7gdc\nAtxJ/GR5DnG6Obc4S73+O9/Z0Nfmvv6OAl4FvKlSlkucexI/mAwQX0tPBS4nHpnmEmPpFuLzEuLr\n6WFgpwzjnFZmEk/czQYey8RcOFEAZxCn0qo+RWfe9ziGngB+LHFq61d0Pon9nDjvWzD0pGU573s4\nj+7CCYAFdM5J5RjnRcAz0/KyFGNucc4lXjG1der/dOBdGcU5m6EXTvQ6rh2I5xP7iJ/gy+WxxPkK\n4hWTO9XqTWac9Rirmi6cyGVfvh1YnpafSZyaziHOae+VxBPaG4gnCHttPvETypXEqbR1xD/sDsSL\nFJou+f1Qiu964OWV8vLyzw3AZyvljyMeqpeXf85+lDEvoHN1X45xziV+8qtehpxjnB+gcwn66cQj\n6hzi/DoRu8tjAAAC+klEQVTxPNkDxPMIR09gXEen8v8mXu04ljjfmtrdTOe1dEql/mTEWcZ4P519\nWXUjg88TTva+rMY5izhbcjXxaK+VQZySJEmSJEmSJEmSJEmSJEmSJEnqjROJX2VVuoD4fYGlzwDv\n28y+W3T+abpuPvEfJ9en299X1u2c1l2e6v0d8SdLfrQZMXxoM9pIkjLxWuLPbED8WrG1wMWV9ZfQ\n+emCkdS/lqxFc5LahfgPq+VPvOyYxn1Venw4gxPl+cSvtdoc9e+RlCRNIbvS+bqYOcBpxKTQR/wv\n+7uIX7f1UuK3sl8F/D/iV8oAbCR+xdDlxN/ZeQXxyOhy4o/GNSWpjxG/7qnqYOLXQc0lJrDyy4A/\nQkw01xO/2ui5xN/8WUf8Ro49U/s3E4++1hF/LuExKa4HU9mZo9wfkqTM3Ej8Ysy3Eb/v7KPEr9f6\nG2ANMVn9Gtgr1T+dzhThTcD70/JWqV6ZOL5J5+unqs4GDq2VlV/YCfGrZapfSbOa+CW7pPI3puWZ\nacy90zgzUvkpwBFp2SMpZWW6fwu6tDkuIU6nHQj8LN0OJH6r+cXAs4jJaEOqfzrxd3pK5XThs1O9\nX6XHX6X7zzgM9/MOxTDrf0Y8z/QB4ven3Uc8ytuXOGW4jnhUtvsw/UuTxiQljd3FxKOmOcQv17yU\nTtK6pKF+weDfY/pzl367JZrriEmlal/it6qP5OvEo7D/IX5L9cJUfjrxBzf3ISbLj46iL2nCmaSk\nsbuE+KOQdxKTz13Ec1IHpHW/JB61lNN4RxCnAeuuT/X2SI/f0GW8zxN/M2luerwj8fzRp0YR6+7E\no7WTgf8kJtYfAa8jXhUI8Vu5y9/++StxWlDKgklKGrtriIni0krZVcSfhP8DcUrtaODbqfxB4sUJ\nMPiI6j7iea3vES+cuJ3mX8C9jXihw5eIF1lcTLwY43uVPrv9cu7rU7zriBdRnJH6+DDxJzgG0n35\nA4dfTDF74YQkSZIkSZIkSZIkSZIkSZIkSZIkSZIkbcn+P+9qUm9WhFuOAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x116accc10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "text4.dispersion_plot([\"citizens\", \"democracy\", \"freedom\", \"duties\", \"America\"]) # plot five words longitudinally"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The IPython Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, we've been writing Python code in an IPython notebook. Why?\n",
    "\n",
    "1. The main strength of IPython is that you can run bits of code individually, so you don't have to keep repeating things. For example, if you scroll up to the last function and replace the 50 with 2, you can re-run that code and get the new answer. \n",
    "2. IPython allows you to display images alongside code, and to save the input and output together.\n",
    "3. IPython makes learning a bit easier, as mistakes are easier to find and do not break an entire workflow.\n",
    "\n",
    "You can get more information on IPython, including how to install it on your own machine, at the [IPython Homepage](http://ipython.org)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Significant Whitespace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One thing that makes Python unique is that whitespace at the start of the line (use four spaces for consistency!) is meaningful. \n",
    "In many other languages, whitespace at the start of lines is simply a readability convention."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Phew, fixed.\n"
     ]
    }
   ],
   "source": [
    "# Fix this whitespace problem!\n",
    "\n",
    "string = 'user'\n",
    "if string == 'user':\n",
    "    print('Phew, fixed.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, whitespace tells both Python and human readers where things start and stop."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining a function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll talk about *functions*. Here's an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def welcomer(name):\n",
    "    print('Welcome, %s!' % name)# here '%s' tells Python to expect a string and how many strings to expect.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that it doesn't do anything by itself. It needs to actually be *called*, and given some data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Welcome, Kim!\n"
     ]
    }
   ],
   "source": [
    "welcomer('Kim')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Advantages of functions:\n",
    "1. Save you typing\n",
    "2. You can be sure you're doing exactly the same operation every time\n",
    "<br>\n",
    "> **Note** Learn to love tab-completion! Typing the first one or two letters of a command you've used previously then hitting tab \n",
    "will auto-complete that command, saving you typing (i.e. time and mistakes!). \n",
    "\n",
    "You may wish to repeat an operation multiple times looking at different texts or different terms within a text. Instead of re-entering the formula every time, you can assign a name and set of actions to a particular task.\n",
    "\n",
    "Previously, we calculated the lexical diversity of a text. In NLTK, we can create a function called **lexical diversity** that runs a single line of code. We can then call this function to quickly determine the lexical density of a corpus or subcorpus.\n",
    "<markdown cell>\n",
    "**Challenge!**\n",
    "\n",
    "Write a function to calculate the lexical diversity of a text; test it out on the books in the NLTK corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def lexical_diversity(text):\n",
    "    return len(text)/len(set(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#After the function has been defined, we can run it:\n",
    "lexical_diversity(text2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The parentheses are important here as they sepatate the the task, that is the work of the function, from the data that the function is to be performed on. \n",
    "\n",
    "The data in parentheses is called the argument of the function. When we use a function, we say that we 'call' it. \n",
    "\n",
    "Other functions that we've used already include len() and sorted() - these were predefined. *lexical_diversity()* is one we set up ourselves; note that it's conventional to put a set of parentheses after a function, to make it clear what we're talking about."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python treats a text as a long list of words. First, we'll make some lists of our own, to give you an idea of how a list behaves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sent1 = ['Call', 'me', 'Ishmael', '.']\n",
    "# Note we use Square brackets here to define our list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Call', 'me', 'Ishmael', '.']"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sent1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(sent1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The opening sentences of each of our texts have been pre-defined for you. You can inspect them by typing in 'sent2' etc.\n",
    "\n",
    "You can add lists together, creating a new list containing all the items from both lists. You can do this by typing out the two lists or you can add two or more pre-defined lists. This is called concatenation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Fellow',\n",
       " '-',\n",
       " 'Citizens',\n",
       " 'of',\n",
       " 'the',\n",
       " 'Senate',\n",
       " 'and',\n",
       " 'of',\n",
       " 'the',\n",
       " 'House',\n",
       " 'of',\n",
       " 'Representatives',\n",
       " ':',\n",
       " 'Call',\n",
       " 'me',\n",
       " 'Ishmael',\n",
       " '.']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sent4 + sent1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also add an item to the end of a list by appending. When we append(), the list itself is updated. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Call', 'me', 'Ishmael', '.', 'Please']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sent1.append('Please')\n",
    "sent1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Extension: Loops**\n",
    "\n",
    "We can ask Python to do something, such as performing a function, all items in a list. For instance, we could ask it to tell us the size of all the files in a directory. We do this by creating something called a *loop*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current fruit : banana\n",
      "Current fruit : apple\n",
      "Current fruit : mango\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "fruits = ['banana', 'apple', 'mango']\n",
    "for fruit in fruits:        \n",
    "    print('Current fruit :', fruit)\n",
    "    print('Done!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Challenge** \n",
    "\n",
    "Define a list called Library with the 9 NLTK books in it. Write a for loop that will run our lexical_diversity() function over each book and tell you its score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Text: Moby Dick by Herman Melville 1851> 13\n",
      "<Text: Sense and Sensibility by Jane Austen 1811> 20\n",
      "<Text: The Book of Genesis> 16\n",
      "<Text: Inaugural Address Corpus> 14\n",
      "<Text: Chat Corpus> 7\n",
      "<Text: Monty Python and the Holy Grail> 7\n",
      "<Text: Wall Street Journal> 8\n",
      "<Text: Personals Corpus> 4\n",
      "<Text: The Man Who Was Thursday by G . K . Chesterton 1908> 10\n"
     ]
    }
   ],
   "source": [
    "Library = [text1, text2, text3, text4, text5, text6, text7, text8, text9]\n",
    "for book in Library:\n",
    "    score = lexical_diversity(book)\n",
    "    print(book, score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Indexing Lists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can navigate this list with the help of indexes. Just as we can find out the number of times a word occurs in a text, we can also find where a word first occurs. We can navigate to different points in a text without restriction, so long as we can describe where we want to be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "173\n"
     ]
    }
   ],
   "source": [
    "print(text4.index('awaken'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This works in reverse as well. We can ask Python to locate the 158th item in our list (note that we use square brackets here, not parentheses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the\n"
     ]
    }
   ],
   "source": [
    "print(text4[158])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As well as pulling out individual items from a list, indexes can be used to pull out selections of text from a large corpus to inspect. We call this slicing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[u'U86', u'thats', u'why', u'something', u'like', u'gamefly', u'is', u'so', u'good', u'because', u'you', u'can', u'actually', u'play', u'a', u'full', u'game', u'without', u'buying', u'it']\n"
     ]
    }
   ],
   "source": [
    "print(text5[16715:16735])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we're asking for the beginning or end of a text, we can leave out the first or second number. For instance, [:5] will give us the first five items in a list while [8:] will give us all the elements from the eighth to the end. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[u'[', u'Sense', u'and', u'Sensibility', u'by', u'Jane', u'Austen', u'1811', u']', u'CHAPTER']\n",
      "[u'upon', u'us', u',', u'we', u'carried', u'forth', u'that', u'great', u'gift', u'of', u'freedom', u'and', u'delivered', u'it', u'safely', u'to', u'future', u'generations', u'.', u'Thank', u'you', u'.', u'God', u'bless', u'you', u'.', u'And', u'God', u'bless', u'the', u'United', u'States', u'of', u'America', u'.']\n"
     ]
    }
   ],
   "source": [
    "print(text2[:10])print(text4[145700:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To help you understand how indexes work, let's create one.\n",
    "\n",
    "We start by defining the name of our index and then add the items. You probably won't do this in your own work, but you may want to manipulate an index in other ways. Pay attention to the quote marks and commas when you create your test sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The\n",
      "brown\n"
     ]
    }
   ],
   "source": [
    "sent = ['The', 'quick', 'brown', 'fox']\n",
    "print(sent[0])\n",
    "print(sent[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the first element in the list is zero. This is because we are telling Python to go zero steps forward in the list. If we use an index that is too large (that is, we ask for something that doesn't exist), we'll get an error.\n",
    "\n",
    "We can modify elements in a list by assigning new data to one of its index values. We can also replace a slice with new material."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['The', 'quick', 'furry', 'child']\n"
     ]
    }
   ],
   "source": [
    "sent[2] = 'furry'\n",
    "sent[3] = 'child'\n",
    "print(sent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Defining variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Python, we give the items we're working with names, a process called assignment. For instance, in the NLTK corpus, 'Sense and Sensibility' has been assigned the name 'text2', which is much easier to work with. \n",
    "We also assigend the name 'sent' to the sentence that we created in the previous exercise, so that we could then instruct Python to do various things with it. Assigning a variable in python looks like this:  \n",
    "variable = expression   \n",
    "You can call your variables (almost) anything you like, but it's a good idea to pick names that will be meaningful and easy to type. You can't use words that already have a meaning in Python, such as import, def, or not. If you try to use a word that is reserved, you'll get a syntax error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Challenge**\n",
    "\n",
    "- Create a list called 'opening' that consists of the phrase \"It was a dark and stormy night; the rain fell in torrents\"\n",
    "- Create a variable called 'clause' that contains the contents of 'opening', up to the semi-colon\n",
    "- Create a variable called 'alphabetised' that contains the contents of 'clause' sorted alphabetically\n",
    "- Print 'alphabetised' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "opening = ['It', 'was', 'a', 'dark', 'and', 'stormy', 'night', ';', 'the', 'rain', 'fell', 'in', 'torrents']\n",
    "clause = opening[0:7]\n",
    "alphabetised = sorted(clause)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that assigning a variable just causes Python to remember that information without generating any output. \n",
    "\n",
    "If you want Python to show you the result, you have to ask for it (this is a good thing when you assign a variable to a very long list!)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['It', 'was', 'a', 'dark', 'and', 'stormy', 'night']\n"
     ]
    }
   ],
   "source": [
    "print(clause)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['It', 'a', 'and', 'dark', 'night', 'stormy', 'was']\n"
     ]
    }
   ],
   "source": [
    "print(alphabetised)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Frequency distributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use Python's ability to perform statistical analysis of data to do further exploration of vocabulary. For instance, we might want to be able to find the most common or least common words in a text. We'll start by looking at frequency distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nltk.probability import FreqDist\n",
    "from collections import Counter\n",
    "fdist1 = FreqDist(text1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fdist1.most_common(50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "906"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fdist1['whale']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEiCAYAAADeViTIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXmYFNW1wH8IsoMjoGwiYwQVFATZFQiK4g5uIJooEZMY\nUYnRGAeNijHxKT439Bl3ASMKIlERQVEZFNkHhh1ZFGQJCEgDgw5rvT/OrVR1T3VPT093dXXP+X1f\nfVV1q+qc29VV99Q95y6gKIqiKIqiKIqiKIqiKIqiKIqiKIqiKIqiKIqiKIqiKIqSMK8D24ClHsfu\nBo4A9Vxpw4A1wCqgjyu9g5GxBnjWlV4NGGfS5wDNXccGAavNcmN5foSiKIqSfnoA7SlpUJoBU4Hv\ncAxKa6AQOBrIBdYClcyxeUBns/0xcJHZHgK8YLavBd4x2/WAdUCOWextRVEUJYPJpaRBeRdoS7hB\nGQbc6zpnKtAVaAysdKUPBF50ndPFbFcBtpvt64B/uq550VynKIqipJCjfNbXD9gELIlIb2LSbTYB\nTT3SN5t0zHqj2T4E7Abqx5ClKIqipJAqPuqqCdwHXOBKqxTlXEVRFCXD8NOgnIy4wBab/ROAAsRt\ntRmJreA6tsmkn+CRjjl2IrAF+R3HADtNei/XNc2ALzwzdPLJ1rp16xL8OYqiKBWSxUC7dCjOxbuV\nF3gH5asCJyGBdLv2MhcxOpUoGZS3YyUDCQ/Kf4sE4o91bXthefHQQw95pid6LOjy/NRV0eT5qSvo\n8vzUFXR5fupKtjzAilbgp7KG8jbwSySusRF4EHjDXZi7tlcA4836EGIs7ONDgFFADcSgTDXprwFv\nIs2Gd+IE3n8EHgHmm/2HgVBZMl5cXJzUY0GX56euiibPT11Bl+enrqDL81NXKvIejVQalOtKOf6L\niP1HzRJJAdDGI30/MCCK7DcIN16KoihKiqmc7gykmeHDhw8vkVilShVyc3M9L0jkWNDl+amrosnz\nU1fQ5fmpK+jy/NSVbHkPP/wwiOenBBW9lZVxCSqKoijxUKlSJYhiO/zuh5IRhELRQy6JHAu6PD91\nVTR5fuoKujw/dQVdnp+6UpH3aKhBURRFUZKCurzU5aUoihI36vJSFEVRUo4aFA+C7sNUf27myPNT\nV9Dl+akr6PL81KUxFEVRFCXj0BiKxlAURVHiRmMoiqIoSspRg+JB0H2Y6s/NHHl+6gq6PD91BV2e\nn7o0hqIoiqJkHBpD0RiKoihK3GgMRVEURUk5alA8CLoPU/25mSPPT11Bl+enrqDL81OXxlAURVGU\njENjKBpDURRFiRuNoSiKoigpRw2KB0H3Yao/N3Pk+akr6PL81BV0eX7q0hiKoiiKknFoDEVjKIqi\nKHGjMRRFURQl5ahB8SDoPkz152aOPD91BV2en7qCLs9PXdkSQ3kd2AYsdaU9AawEFgMTgWNcx4YB\na4BVQB9XegcjYw3wrCu9GjDOpM8BmruODQJWm+XG8v8URVGUisXhw2W/JpUxlB5AETAGaGPSLgA+\nB44Aj5m0PKA1MBboBDQFPgNaAhYwD7jdrD8GRgJTgSHAGWZ9LXAlMBCoB8xHDBFAgdn2MrcaQ1EU\npcKyfz+sWwerVzvLN9/Iul8/ePnlktfEiqFUSWFevwJyI9KmubbnAleb7X7A28BBYD2wFugCbADq\nIMYExDhdgRiUvsBDJv094HmzfSHwKY4BmQZcBLxTvp+jKIqSeRw5Ahs3ehuNDRvkuBffflt2XemM\noQxGahwATYBNrmObkJpKZPpmk45ZbzTbh4DdQP0YsuIm6D5M9edmjjw/dQVdnp+6gi4vFbo2bQox\ndy6MGQP33w/9+0PbttCtW4jcXOjTB26/HUaOhE8+ge++g3btQrRoAZdcAnfeCS+8AJ99Bt9/D+PH\nlz2GksoaSizuBw4gbi5FURQlDg4dcmoX33wTvjRpAoWFJa9p1w4aNYJTToFTT5W1vdSvD8cd560r\ngZh8WgzKb4BLgN6utM1AM9f+CUjNYrPZjky3rzkR2IL8jmOAnSa9l+uaZsAX0TKTl5dH9erVAejY\nsSPdu3cnJycHcL4GIvdtIo/baaVdH0R5OTk5Ki9F8qLtVzR5se5vRZPnTvO6fudOWLEixMaNsHhx\nDqtWgWWF2LIFCgrk2nbt5PzCQrk+NxeuvjpE5co5nHoqtG0bolkzaNUqh7p1E89fYWEh+fn5FBcX\nUxqp7tiYC0zCCcpfBDwJ/BLY4TrPDsp3xgnKt0CC8nOBoUgcZTLhQfk2wK1IMP4KnKD8AuAs5PcV\nmG0ve6tBeUVR0oJlwaZNsGIFrFzprFeuhJ07va+pVAlOPFFqGpFL06ZwlA9BjFhB+VTyNlJ7OIDE\nOgYjTXw3AIvM8oLr/PuQYPwqJLBuYzcbXosYE5tqwHicZsO5rmM3mfQ1SBPiaFhe7Nq1yzM90WNB\nl+enroomz09dQZfnp64gyTt82LK+/dayPvrIsh5/3LIGDbKsTp0sq1u3XZaYlfClXbtdVq1altWh\ng2Vdf71l/e1vljVunGUtXmxZ//lP+u8F8qHvSSpdXtd5pL0e4/xHzRJJAU4Nx81+YEAUWW+YRVEU\nxReOHJFWU8uXO8v+/fDRR/DTTyXPb9dO4hetW0OrVrK0bg3Nm0OLFlIbiSSRuIaf6Fhe6vJSFKUM\n2K6qZcvEaNjrlSth3z7vaxo3FmPRujWcfrpjRBo08DfvySBd/VAURVEymp07YckSWWzDsXw57Nnj\nfX6jRmIw3EurVlCvnr/5VtKDLz7HoMvzU1dFk+enrqDL81NXWa85eNCyli2zrAkTdln33mtZF19s\nWU2ahMc13HGOBg0sq1cvy7rtNssaNWqX9eWXlrVjR3ry7rc80hRDURRFCRy7d8PChVLrWLxYlhUr\nJN7Rrl14X45ataBNG+jdGwYPlhrHGWfA8cc754RC4GppW6HRGIrGUBQla/nhB+m3YS8LFkj8w4uT\nToIzz5SlbVtZfvELf5riZhIaQ1EUJevZvRvmz4d582RdUCBjWEVi1zratnUMSJs2ULeu/3nONtT2\nehCUsXkyeVwhlee/rqDLS6auAwfEaIwaFWLQIAl85+TABRfIOFbr10sv89q1oUcPGafqzTelJdaG\nDSFmz4aXXoIhQ+Cccxxjkon3wm95sdAaiqIogcayZCDDuXNhzhxZL1pUMuZRtSq0bw+dO0P37lLr\nOOUUqFw5XF7Q+3JkMhpD0RiKogSKvXul9jFnjrNs317yvFNPhS5dxIB06SIurKpV/c9vRUNjKIqi\nBBLLkpFyv/7aqX0sWybpburXF6PRtausO3WCY49NT56V6GgMxYOg+zDVn5s58vzUFXR5ADt3higo\ngKefhquugoYNJf7x/PMhXn0Vli4VF1XHjjJ3x7/+BcuWhdi+HSZPhgcekHk9bGOSyfcik/MeDa2h\nKIqSMvbvl6a6M2bAl19KD/PZs8PPadgQevaEG26QGkj79lCjhnM8FPIe10oJHhX9b9IYiqIkkZ9/\nFrfVjBmyzJ4NkdNonHyytLyyl2gDISrBRGMoiqKkhOJiiX3k58P06bJ94ED4Oa1bwy9/KbWQnj1l\nZkElO9EYigdB92GqPzdz5Pmpyw95xcViPJ55JkSvXtL349xz4eGHxaV18KDMGjh0KLz3nvRUX75c\n5iofOBBq1syee5EpujSGoihKINi/X1xYdg1k9uzw/h+VKsl2r16y9OghQ5Xo2FYVk4ruudQYiqK4\nOHhQ+oB88YUYkVmzJC7i5swzxYV17rniwtKh2SsWGkNRFMWTQ4dk5N3p02WZObPkJFFt2kjtwzYg\n9eunJatKBqAxFA+C7sNUf27myPNTVzzXWJbENEaOhH79xDjcckuIvDz45BMxJq1ayRhX778f4ocf\nZJj3kSPhyisdY5IN9yKd8vzUpTEURVGSxvffS0/0jz4SV9bWreHHmzaVnufnnis1kcaNJT0U0liI\nUjY0hqIxFCXL2LVL3FeffSbLmjXhxxs1gvPPl0mjeveGZs3Sk08lM9EYiqJkMfv3Sw3ENiAFBXDk\niHO8bl2pedhGpFUr7UiopAaNoXgQdB+m+nMzR14qdO3aFWLpUnjqKbj4YhnXqndvmDIlxPz5MhZW\nr17wyCPSzHfnThg9OsQdd0gnw0hjksn3IlPl+akrW2IorwOXAj8AbUxaPWAc0BxYDwwA7FwPAwYD\nh4GhwKcmvQMwCqgOfAz80aRXA8YAZwE7gWuBDebYIOB+s/13c56iZCy7dknQfMoUmcL2iy/Cj7dt\nCwMGwGOPyVwgtWqlJ59KxSaVFd8eQBFSmNsGZQSww6zvBY4F8oDWwFigE9AU+AxoCVjAPOB2s/4Y\nGAlMBYYAZ5j1tcCVwEDEaM1HDBFAgdn2MrcaQ1ECid0aa/JkWWbNgsOHneONG8vshH36iCurYcP0\n5VWpWKQrhvIVkBuR1hf4pdkeDeQjBqUf8DZwEKm5rAW6IDWOOogxATFOVyAGpS/wkEl/D3jebF+I\n1G5sAzINuAh4Jwm/SVFSxv79EkyfNEmMyIYNzrEqVaQV1qWXwkUXebuuFCXd+B1DaQhsM9vbzD5A\nE2CT67xNSE0lMn2zScesN5rtQ8BuoH4MWXETdB+m+nMzR15purZvh1Gj4OqrpY/HxRfLuFfHHhvi\n+ONh0CB4913YsUPcXHffDU2bhjyNSabfi4okz09d2RJDKQ3LLIpSYbAsWLUKPvxQJpMaOzZ8dsIz\nz4TLL4dLLpGZCY/SZjNKBuG3QdkGNAK2Ao2RgD1IzcPdGv4EpGax2WxHptvXnAhsQX7HMUhwfjPQ\ny3VNMyAihOmQl5dH9erVAejYsSPdu3cnx/Tmsi105L5N5HE7rbTrgygvJydH5aVIXu3aOcyYAV9/\nHWLWLJg8WY63awcdOoSoXz+Hvn2hd+8QDRva10eXF20/0fz5JS/W/a1o8txpQZdXWFhIfn4+xZET\n23gQjxe2NvAz0vrqVLNMQeIdpZELTCI8KL8TeByJneQQHpTvjBOUb4HUYOYirb7mAZMJD8q3AW5F\ngvFX4ATlFyCtvyohQfmz0KC84iPFxdIqa8IEiYfs2uUcq18fLrtMaiJ9+kCdOunLp6KUlVhB+XhY\nCNRECvr1wLvAW3Fc9zZSeziAxDpuQgr7z4DVSOA8x3X+fUgwfhUSWLfpACw1x0a60qsB44E1wBzC\nGwDcZNLXIE2Io2F5sWvXLs/0RI8FXZ6furJZXnGxZU2aZFk33GBZdetaljizLKtdu13WKadY1j33\nWNZXX1nWoUPBy7sf8vzUFXR5fupKtjxihCricXlVAn4CbgZeQGoZi+O47roo6edHSX/ULJEU4NRw\n3OxH+rF48YZZFCWlHDwovdNnzIAXX4Tdu51j7dtD//7SMqtt2/TlUVH8Ip5qyyLEvfQ0YlSWIzUG\nr0I+0zAGV1Hi58gR+OorePttcWnt3OkcO/NM6WDYvz+0bJm+PCpKqihvP5Q7kV7s/0aMycnA9GRl\nTlEyAcuSeUPefhvGjZPe6jatW8v0tgMGwKmnpi+PipJu4mmU2BDpRPi42V8HzExZjgJA0NuBa5t4\n/+StXi3zpZ92Gvz2tyGefFKMSfPmkJcHixfDsmVwxx2hqMYkW+5FJukKujw/dQWtH8owJPhdWpqi\nZAVbtkgtZOxYWLDASe/VC26/Ha6/Hrp21Z7qihJJrFfiYuASZJysd1zn1kGa+XZObdZ8QWMoCgA/\n/QTvvSe91qdPdzob1qkjvdivv16GPqmiEz4oFZxEYyhbkBZW/czaFrAH+FMS86coacGypAby2msS\nG9mzR9KrVpWWWddfL+saNdKbT0XJFGLFUBYjw8afjAzkOMosE4Fd0S7KBoLuw1R/bvmu2bEDXn01\nxJlnQufO8NJLYky6dIExY0Js2wYTJ8I114QbkyDkPZPl+akr6PL81BW0GEoXZFTfXNf5FvCLMmtT\nlDRx5Ij0F3n1VXj/fTj9dBlLq0EDuOEGuPlmSQuFdB51RUmUeMKK3yBNhxciw6/Y7EhJjvxFYyhZ\nzvffwxtvyGIPB3/UUXDhhWJELr9cXFyKosRHefuhhJCxuxQlIzhwQEbzffVV+PRTJ8CemytG5De/\ngRNOiCVBUZREiKcfynTgCaAbMsiivWQtQfdhqj/XO331avjLX8RY9O8P27aFOPpo6XT42Wewbh38\n9a9yXO+t//L81BV0eX7qCloMpSsSM+kYkX5umbUpSpI5cADeegteeUXG07I54wzpM3LFFTK6r6Io\nqaeid83SGEqGsny5GJExY5yh4WvWlNrI734nLba046GiJJ/yxlAeQmoolQgftvhv5c6ZopSBfftg\n/HgxJLNnO+kdOogRue46qFs3fflTlIpOPDGUfWYpAo4gvedzU5intBN0H2ZF8+cuWgQPPRSiSRMY\nPFiMSZ068Ic/wNy5IRYsgFtuKWlMgvB7/dQVdHl+6gq6PD91BS2G8r8R+08gk2MpSsrYtw/eeUc6\nHc6fL9Pl7tkD3bpJbWTAAKhVS/qNKIoSDBLxMtdDpuNtkeS8pAONoQSMpUvFiLz5pjMUSk6OdD78\n/e8l2K4oSvoobwxlqWv7KOB4NH6iJJHiYnj3XZnxcNYsJ71bN3Fl9e8vAXdFUYJNPDGUy81yGdAH\naAI8l8pMpZug+zCzxZ/77bdw773SL+TGG+Gnn0LUqQNDhsg8I7NmwaBBYkz03ma+PD91BV2en7qC\nFkNZD7QDeiCtvL4ivjnlFaUEhw/D5MnwwgswZYrTi719e7jrLuk3Urt2evOoKEpixBND+SPwO2SU\n4UrAFcArwMgU5ssvNIbiE8XF8PLL8PTTsH69pFWrBtdeKzWSzp2134iiZAKxYijxvMJLkd7y+8x+\nLWAO0CYZmUszalBSzP79MqbWo4/KTIgAJ50Et94KN90ko/0qipI5xDIo8cRQQPqfeG1nJUH3YWaC\nP3f79hAvvggtWsgQKFu2SNPfSZNCrF0L99xT0pgE4V5kwr3NVHl+6gq6PD91+RlDicegvAHMBYYD\nDyO1k9fLrCmcYcBypPYzFqiGNEeeBqxG+rnkRJy/BliFNAyw6WBkrAGedaVXA8aZ9DlA83LmV4mT\ngwelJ/uvfy21kE2boE0bmbCqoAC6d5fh4xVFyT7i9Vp3ALrjBOUXlUNnLvAF0ArYjxT8HwOnI3Os\njADuBY4F8pD568cCnYCmwGdAS5OXecDtZv0xEteZCgwBzjDra4ErgYEeeVGXV5I4ckQ6Ij74oIzq\nCzJh1fDhcNVVakQUJVtI1OXVGRlmBWRO+WeRArsxYmASZQ9wEKiJtDKricxf3xeZahizvsJs9wPe\nNtesB9Yis0g2BuogxgRgjOsat6z3gN7lyK8SA8uCSZPEnfWrX4kxOeUUmaN9yRKZRleNiaJUDGK9\n6o8DKzzSV1ByOJay8CPwJPA9YkhCiKurIbDNnLPN7IP0e9nkun4TUlOJTN9s0jHrjWb7ELAbcanF\nRdB9mEHx5+bnhzjnHOjbV3q4N2smAfjly+Gii0KehiTo9yIo9zYb5fmpK+jy/NQVlH4odZAaQSTr\ngfK0zTkZmVI4Fyno3wV+HXGORfjIxkqAWLlS+oxs3QqFhRJcv/9+GayxevV0505RlHQRy6DkxDhW\noxw6OwKzgJ1mfyIyG+RWoJFZNwZ+MMc3A81c15+A1Ew2m+3IdPuaE5EaUBXgGKRmVIK8vDyqm1Kw\nY8eOdO/enZwc+em2hY7ct4k8bqeVdn0Q5eXk5JQqb9OmEKNHw/DhORw6BGefDc8+G+Kmm3KoU0fO\nLy6OX16y8xdUedH2K5q8WPe3oslzpwVdXmFhIfn5+RQXF1MasYLyLyFB8r/i1BaOQlp6NQR+X6p0\nb84E3kKC7MXAKCQO0hwxMo8jwfgcwoPynXGC8i1MnuYCQ831kwkPyrcBbkWC8VegQfmEOXIERo2C\nYcPghx+kA+Lvfw+PPALHHZfu3CmK4ieJBuXvRtxT65BaxESkGe4p5liiLEYC6AuAJSbtZeAx4AKk\n2fB5Zh8kZjPerKcgxsK2AkOAV02+1iLGBOA1oL5JvxMxTHETdB+mn/7cWbNCdO0KN98sxqR7d2n+\n++KLcPTR6c970OX5qSvo8vzUFXR5fuoKSgylCPmqPxlp0mshhfq6MmspyQizuPkROD/K+Y+aJZIC\nvHvs7wcGJJw7he3b4S9/kRhJYSE0bQpPPCFT7OoQKYqieFHRiwZ1eUVw+LB0TLzvPpmrvWpV+POf\nxd2lgzYqilLe+VCUCkJBgfRunz9f9i+8EJ57Dlq2TG++FEXJDLTLmQdB92EmW96uXTB8eIhOncSY\nNG0KEybI8PLHHRfsvAddnp+6gi7PT11Bl+enLj9jKPEalB7ATWb7OOCkMmtSAodlwdixcNpp8MEH\nULmyDNq4ahVcfbXGShRFKRvxFBnDkaFWTkVaeDVFWl2dk7ps+UaFjaFs2CAdEaeadnE9e8qkV6ef\nnt58KYoSbMo7fP2VyHha9nwom5Fe9EoGcvgwPPusGI6pUyEnB157DfLz1ZgoilI+4jEo+wmfA6VW\nivISGILuw0xUXkFBiLPPhjvvhH37YMAAGUZl8GDYvTvYec9UeX7qCro8P3UFXZ6fuoLSD8XmXaTX\nfA7SO34w0plQyRAOHIC//U1qJAUFcMIJ4t66/PJ050xRlGwi3rBrH5yJrT5BRgfOBrI+hrJqlQwr\nv3ChBNmHDJHpeOvWTXfOFEXJRMo7p/zdwDtI7CTbyFqDYlnw0ksyKvDPP0NuLrz5pgydoiiKkijl\nDcrXQabknYnMjtgw9umZT9B9mKVds3079OsnnRR//hluvBEWL4Yzzgh+3rNNnp+6gi7PT11Bl+en\nrqD1QxmOjOV1GzKs/JfA52XWpPjCvHkyh/ukSXDMMTJz4ujR6uJSFCX1lKXrWmPgGuA6oDbQNiU5\n8pescXkdOSLjbY0wQ2727CkurhNPTG++FEXJLsrr8hoC5CO1kgbAb8kOY5I1HDokTX9HjIAqVSTo\n/sUXakwURfGXeAzKicicIq2Bh/CeZz6rCLoP051eXAz9+4tbq2ZN+PjjEMOGyTAqydal8oKtK+jy\n/NQVdHl+6gpKP5S6wB7gCWQulHoRxz2n1FX8Y+9euOIKqY3k5MDHH0OrVunOlaIoFZVYMZTJwKXA\nepwZEt1kwwCRGRtD2bkTLrlEgvANG8Knn0JbdUQqipJiytsPJZvJSIOyZQv06QPLl0v/kmnToEWL\ndOdKUZSKQHmD8l5NhLO62XCQfZjffgs33xxi+XJo3Rpmzgw3JkHOe0WU56euoMvzU1fQ5fmpKygx\nlBpATWT+E3f8pC4yhL3iM998A+edB8cfD506yQRY9eunO1eKoihCLJfXncAfgSbAFlf6XuBl4PkU\n5ssvMsbltWwZnH8+bNsGPXrA5MlQRycRUBTFZ8obQxkKjExmhgJERhiURYvgggskEN+7t8yuWCvr\nJxFQFCWIlDeGMhI4AxgA3OhaspYg+TDnzxc3l92qa9IkOHhQ/bmZIs9PXUGX56euoMvzU5efMZR4\nx/J6DnFxnQuMAPqWWVM4OcAEYCXSUbILEqeZBqxGBqPMcZ0/DFgDrMIZRh9kauKl5tizrvRqwDiT\nPgdoXs78poWlS6VGEgpJf5OJE6FGjXTnSlEUxZt4XF7LgDOBhWbdEHgLOL8cekcDM4DXkYYBtYD7\ngR2IwboXOBbIQ3rojwU6IY0BPgNaIn1j5iEjIM8DPkZqU1OR4WLOMOtrkWmMB3rkI7Aur/x8uOwy\nZ2bFf/0Ljj463blSFKWiU16X18/AYeAQcAzwA9CsHPk5BuiBGBOM3N1IrWe0SRsNXGG2+wFvAweR\nTpZrkRpNY2Ro/XnmvDGua9yy3gN6lyO/vvPll+Le2rcPbrgB3npLjYmiKMEnHoMyH6ktvAIsABYB\ns8qh8yRgO/AGUut5BamhNAS2mXO24cy70gTY5Lp+E1JTiUzfjNOcuSmw0WzbBity6JiopNOHuWiR\nTM37888wbFiIN96QAR9Tmb9Yx1Re5ugKujw/dQVdnp+6gtIPxWaIWb+ITP9bF1hcZk3hOs9CXFXz\ngWcQ15YbC+/hXpJOXl4e1atXB6Bjx450d01paN/QnJyc/+4XFRWF7buPFxUVlTjfTWnyli0Lcddd\nsGdPDv37wy23FLF3b+Ly4s1frH2VVz55biq6vNLub0WTV97ywi95hYWF5OfnU1xcTGnEiqF0IHah\nvrBU6d40AmbjjAXWHQm6/wIJ+m9F3FnTgdNwjM1jZj0VGfV4gznHHg7xOqAncKs5ZzgSkK8C/Afp\noBlJYGIomzfDOefAhg3SRHjSJKhWLd25UhRFCSdWDCVWDeVJYhuUcxPMz1bEHXUK0qLrfGC5WQYB\nj5v1++b8D5Gg/FOIK6slEjexkNGQu5j9G3D6y3xoZMxBJgUL9FAxO3fK2FwbNkDXrtKaS42JoihK\nfJyJuLsWAxORQH09pAWXV7Ph+5Bg/CrgQle63Wx4LeGdL6sB43GaDedGyYflxa5duzzTEz0W65ot\nW3ZZXbpYFljW6adb1s6d5ZPnZ95VXrB0BV2en7qCLs9PXcmWR4yKRjwxlEFRBIyJ49poLEaaAUcS\nrSnyo2aJpABo45G+H+mIGWj274cHHoC5c2XU4E8+gXpxNx1QFEUJFvH0Q3kex6DUAM5D4ifXpCpT\nPmIMrv8cPgzXXQfvviuDPc6cCS1bpiUriqIocZNoDMXm9oj9HKQXupIglgW33SbGpG5dqZmoMVEU\nJdOJpx9KJD+RHbM1RiXV7cAfeABeegmqV4dJk0K0a1c+eak8pvIyR1fQ5fmpK+jy/NQVtH4ok1zb\nRyFDoYwvsyYFgGeegX/8AypXhvHjddpeRVGyh3hiKL1c24eQ/h8bvU/NOHyNobz5JtxoxmkeNQoG\nDfJNtaIoSlJI1pzydQmv0fxYjjwFBd8MyuTJ0K+fBOOffBLuussXtYqiKEmlvIND3oJ0RlyKNNMt\nQMb0ylqS7XP86qsQ11wjxiQvL9yYBMUnmqn+3KDL81NX0OX5qSvo8vzUFbQYyj3IUPA7yixdYelS\nGDYMioshq+oPAAAgAElEQVTht7+FR7160yiKomQB8bi8PkXmE9mX4rykg5S6vH76Cdq3h9Wr4aqr\nYNy4kiMHK4qiZBLl7YeShwzmOBs4YNIsZK55JQb33CPG5PTTZYIsNSaKomQz8cRQXkbG2JqDxE7s\nOErWkgyf45Qp8MILMjHWqFGhqFP3BsUnmqn+3KDL81NX0OX5qSvo8vzUFbQYSmVA2ySVgZ07YfBg\n2X7kEWjRIr35URRF8YN4YiiPIn1PPkQGXbTRZsOeAqF/f3jvPejRA6ZPl06MiqIo2UB5+6Gsx3u0\n4WwYfiXpBmXMGOmwWKcOLFkiowgriqJkC+Xth5KLGI/IJWtJ1K+4alWI281Qms895xiTTPCJZqo/\nN+jy/NQVdHl+6gq6PD91BS2Gkor5ULKOw4elj8nevdJE2B5iRVEUpaKg86EkyeU1YgTcey80aiSd\nGRs0SIpYRVGUQJGssbxscpD5UC4s7cQMICkGZckS6NgRDh6Ejz+Giy9OQs4URVECSHljKJHofCgu\nDh+Gm28WY/LggyFPY5IJPtFM9ecGXZ6fuoIuz09dQZfnp66gxVB0PpQYjBwJCxbACSfALbekOzeK\noijpI5H5UNYDm1KRmTRQLpfXd9/BGWfImF0ffQSXXprEnCmKogSQRMfyagk0BPIj0rsD1YB1Schb\nxmJZUiP56Se47jo1JoqiKLFiKM8AezzS95hjWUs8fsUxY2DaNKhXT6b1jXVdJvhEMzXvQZfnp66g\ny/NTV9Dl+anLzxhKLIPSEFjikb6E5ATlKwOLcGI09YBpwGpkyPwc17nDgDXAKqCPK70DMvHXGuBZ\nV3o1pCXaGmRQy+ZJyO9/2bYN/vQn2X7mGTj++GRKVxRFyUxixVDWAtGGNYx1LF7uQgxCHaAvMAKZ\nxGsEcC9wLDJ0fmtgLNAJaIqMfNwS6RszD7jdrD8GRgJTgSHIpGBDgGuR+VwGeuQhoRjKwIEyt0mf\nPjB1KlRKpPG1oihKBpJos+EFwO890n9H+YevPwG4BHjVlbG+wGizPRq4wmz3A94GDiINAtYCXYDG\niDGaZ84b47rGLes9oHc58/tfJk0SY1KzJrz0khoTRVEUm1gG5U7gJmAG8JRZZgA3m2Pl4WlkauEj\nrrSGwDazvc3sAzQhvFXZJqSmEpm+2aRj1hvN9iFgN+JSi4tovsM9e+C55+TYP/5RcuDHTPaJZmre\ngy7PT11Bl+enrqDL81NXUPqhbAXOBs5F3EcW8BHwRZm1hHMZ8AMSP+kV5RwL7/HDkk5eXh7Vq1cH\noGPHjnTv3v2/x+wbmpMj4ZynngpxzDFFdO6cwx13lDxeVFQUth/5h0SeHwqFKCoqKnG+3/Ji7au8\n8slzU9HllXZ/K5q8TCkvCgsLyc/Pp7i4mNJIh8PmUeAGpOZQHagLTERiJL0QQ9YYmA6chsRRAB4z\n66nAQ8gcLdOBVib9OqAncKs5ZzgSkK8C/Ac4ziMvccdQ5syBbt1kGt+FC6FNm7guUxRFySqSPfRK\nebkPaIa0FBuI1HhuQCbwGmTOGQS8b7Y/NOdVNde0ROImW5EmzF2QH3cD8IHrGlvWNcDn5cnwoUNw\n662yfc89akwURVG8SIdBicSuIjwGXIA0Gz4Pp0ayAhnqZQUwBWm5ZV8zBAnsr0GC9VNN+mtAfZN+\nJ04tJy4iq4r/939QWCgxk6FDo/sVM9knmql5D7o8P3UFXZ6fuoIuz09dQYmh+MEMs4BMKXx+lPMe\nNUskBYBXfWE/MKDcuQO2bIEHHpDt554DE25RFEVRIqjojV5LjaFcey2MHw/9+sH778c8VVEUJetJ\n9nwo2URMg/Lpp3DhhdLnZMUKaJ7U/vaKoiiZR9CC8oEnFApRXAy33Sb7Dz7oGJMg+DDVn5s58vzU\nFXR5fuoKujw/dfkZQ1GDEoXHH4e1a6F1a2fcLkVRFCU66vLycHmtXSvznOzfDzNmQM+eaciZoihK\nAFGXVxmwLLj9djEmN96oxkRRFCVe1KBEMGECbNsW4thj4YknSh4Pgg9T/bmZI89PXUGX56euoMvz\nU5fGUNLI44/L+h//0HlOFEVRyoLGUFwxlE2boFkzaSa8YwfUqJHGnCmKogQQjaHEySQzd2SfPmpM\nFEVRyooaFBcffijrAQOC7cNUf27myPNTV9Dl+akr6PL81KUxlDSwdy988YXMwNitW7pzoyiKknlo\nDMXEUCZMgP794ZxzYObMNOdKURQloGgMJQ5sd1ffvunNh6IoSqaiBgWZQGvyZNnu1y/4Pkz152aO\nPD91BV2en7qCLs9PXRpD8Zmvv4Yff4RTToFTT013bhRFUTITjaFYFnffDU89BX/+s3fveEVRFEXQ\nGEoMLAs+MDPR9+uX3rwoiqJkMhXeoKxcCevWQYMGTnPhoPsw1Z+bOfL81BV0eX7qCro8P3VpDMVH\n7NZdl10GlSunNy+KoiiZTIWPoXTtajFnDkycCFdeme7sKIqiBBudUz46VqVKFlWrymCQtWunOzuK\noijBRoPyMbAsOP/8cGMSdB+m+nMzR56fuoIuz09dQZfnp65sj6E0A6YDy4FlwFCTXg+YBqwGPgVy\nXNcMA9YAq4A+rvQOwFJz7FlXejVgnEmfAzSPlSHtHa8oilJ+0uHyamSWQqA2UABcAdwE7ABGAPcC\nxwJ5QGtgLNAJaAp8BrQELGAecLtZfwyMBKYCQ4AzzPpa4EpgoEdeLLDYvBmaNEnBL1UURckyguby\n2ooYE4AiYCViKPoCo036aMTIAPQD3gYOAuuBtUAXoDFQBzEmAGNc17hlvQf0jpaZzp3VmCiKoiSD\ndMdQcoH2wFygIbDNpG8z+wBNgE2uazYhBigyfbNJx6w3mu1DwG7EpVYCL3dX0H2Y6s/NHHl+6gq6\nPD91BV2en7r8jKFUKfMVyaM2Unv4I7A34phllpTz/fd5DB9eHYCOHTvSvXv3/x6zb2hOTs5/94uK\nisL23ceLiopKnO8mqPJi7au88slzU9HllXZ/K5q8TCkvCgsLyc/Pp7i4mNJIV7Pho4GPgCnAMyZt\nFdALcYk1RgL3pyFxFIDHzHoq8BCwwZzTyqRfB/QEbjXnDEcC8lWA/wDHeeTDOnLEolJFbzytKIoS\nJ0GLoVQCXgNW4BgTgA+BQWZ7EPC+K30gUBU4CQnIz0MMzx4knlIJuAH4wEPWNcDnUTOjxkRRFCUp\npMOgnAP8GjgXWGSWi5AayAVIs+HzcGokK4DxZj0Fabllu8OGAK8izYPXIjUTEINV36TfiVPLiYug\n+zDVn5s58vzUFXR5fuoKujw/dWV7DGUm0Q3Z+VHSHzVLJAVAG4/0/cCAsmdNURRFSZSK7vD575zy\niqIoSukELYaiKIqiZCFqUDwIug9T/bmZI89PXUGX56euoMvzU5efMRQ1KIqiKEpS0BiKxlAURVHi\nRmMoiqIoSspRg+JB0H2Y6s/NHHl+6gq6PD91BV2en7o0hqIoiqJkHBpD0RiKoihK3GgMRVEURUk5\nalA8CLoPU/25mSPPT11Bl+enrqDL81OXxlAURVGUjENjKBpDURRFiRuNoSiKoigpRw2KB0H3Yao/\nN3Pk+akr6PL81BV0eX7q0hiKoiiKknFoDEVjKIqiKHGjMRRFURQl5ahB8SDoPkz152aOPD91BV2e\nn7qCLs9PXRpDURRFUTIOjaFoDEVRFCVuNIaiKIqipJxsNygXAauANcC98V4UdB+m+nMzR56fuoIu\nz09dQZfnpy6NoSSHysDziFFpDVwHtIrnwpkzZyb1WNDl+amrosnzU1fQ5fmpK+jy/NSVirxHI5sN\nSmdgLbAeOAi8A/SL58IFCxYk9VjQ5fmpq6LJ81NX0OX5qSvo8vzUlYq8RyObDUpTYKNrf5NJUxRF\nUVJANhuUhJtvFRcXJ/VY0OX5qauiyfNTV9Dl+akr6PL81JWKvEcjm5sNdwWGIzEUgGHAEeBx1zlr\ngZP9zZaiKEpGsxhol+5M+E0VYB2QC1QFCokzKK8oiqIokVwMfIPURIalOS+KoiiKoiiKopRGNsdQ\n4qEx8COwv4zXNQK2lkNvPaAlUM2V9mUZ9CWa77JS3t+plM7dEfsW8JTZvgF409/sKBlMfeAPwM/A\nq8AevzNQxW+FAeNfSFB+AvDnMlz3GnBpgjp/BwwFTkDiOl2B2cB5ZdAXme9zkFiR/X9awBigOhDZ\nVKM6sNJs/wB0iaH3Y+CsWD/GxR+BZ137RSY/Bz3OtYC6wABgKvLgP2B0PQIsBLoj96cIKVjbG/nV\ngRcQY3c60BboC/zdyI52L8rKdLP+Ebg6gevjpQ7RWyTWLOXaicizMQVpcOKmOxDZM+0c4GuiPxfx\nNOupS3i58aNZn0nJ+z4R+Z/+gTTZtzsZdwPeQu5r5DV/iyMPIPfmpzjPHRSxbwEv4/1s2sc7Efs5\ni8bnQG+PtPORd35jiStiU9r/4n7e85BRQdYCc4DLkThyJDWBZkg4wE0t4C7gRKScagmcCnxUxjxX\naI4CeiIv5lST1hq4OcY19lfjnRHpRcDeKIv9tbAMqIEUlgCnAf92yWiEPAiXAceXku/TEeMyC3n4\nn3MtIAVzJF5p0VgUsf8EUqAcjbwkO5DC3utccH5jNJaadXcgH/nNc13HKiEF1SLgNmAGUpPr4tJX\nCVhutmPdi2h5vxM4xsh5zci9EGhulhMi8lzaf1wFKSy9iKarNLxqKW8CFwBjgW+Bx5CX38br/7DT\n4nkuXo7YvwWprW4AvjPLt+bYG8ACYLTZtheQd+paYInZPxp5Bz4BxgF/QWpp9gLSiOaPwHtmucNc\nB3A2sAKnYG6H/N+nIv+r/Sy0Bf5qtp/HeRZeMfmeYI79HRiCPBt1gVuRj5pYz1kjSpYXf0BqCEsQ\nD4S95CLDP1Uyv9uLWsgH1StmvyXyLoAYhFlI69RLkefHJvJ534HzvF9o7tFSs/2uSe+LGJL1Zr89\n8KHZHo8MUWX/zlpIiy6ljER76KOxAmiC8/DUJ/whivaQgrx4IIVtdZc8kC/2DcgX9RjkT+9fSt5X\nUtJ12RjogDzIZ5nts4BeJi1ehkTs2w/XlcgLdYzJ7yQgZNb2ko+84LGwDc5jwK/M9qKI9UPAb832\nQpz75y4wbTle9yJW3pfg/OcXIob9DCPbLjTnUnZmEu7OtImmqzQiz6mC88wA5CAF2iaj4x2zfRdO\nQT0cKSjifS4ida4FGkTJ3wqi3/do/1es9+s1xDidh3ztj0JcOADzkC9ot7zlxDYAkeQgBg2c/8TN\nkhj5Bu/y4j/I87If59n5zpxzuzlvNDKCRySlFeTNkffjn8j7Fu15/xoxYDZHIR9ENZHyCuQdyon4\nXfZ/UeDxm8tkUCq6y8umAfK1lGf2DwKHYpz/IlJY/gLnT3CzF/lCsvkn8mA9gHw1HAu8D0wDduF8\nLfwVqWr/YPaPM3reJTrLEAOyxZXWB/gN4mZ4MiJf98WQFckLEfv283IZ8oW3G9hndBwH/C/OA76X\n0h/GzciX8AWIUamO09nWzuuvgR7I2GxHIwVlC5eMa5CXGbzvRay8Wy59lyJf/fbLdVKUPNcr5Tf9\niBQkM5EvP9stY+Hcm0hd0bgPaZ1YA7kfNgdxahANkHv0a6SwWIS0bjwOcafZ7EEKtP8lvudie8T+\nt4hv3ov5yFe6VwFeRLgh6orc+2+Qd8SrQO9E+PvzecR530ecfwgpNN3G3yK6S+snnP93H3Lv3jb7\nA02edxP9OfMqL7YhtaU7cGoJkXQ1ujYYve58DjC6cR0DMQjnIO9AO+Qef2WORT7vNxP+IXMEeV/s\n32znNXLUR9tduh951mxOpoxxWjUoQhFSy7CxH/pojDTLP4GXEJeZhfzRhUhMxOshBflCBvlizEdq\nMHbVuRLhL/JOSm84cRzyhTgP58+3gHORl2BClOsSYRLyJVuM1LqORwqjfOSelZUBiF/9CeQhbwzc\nY45dC1wPDEZcLSciheFM5J6firxI35r8TAJq430v+kbJezHylfcp8nGQh/wfkfEINwuJHvOwjJx1\nZjnK5KmSOVbg0jUsDl2PmuV/kHvUEqdWC1LLOQ2pzV6OU+A9hLg6HvaQOYL4novjIvbzkOd6NnDA\npFlIPPANk76V8PveFqkdfYD85llG7jVILeomnK969zWHkMJ8rUk/GecD73ukgAVxjQ1F/sNjiG4A\nJrnSj0KM33izfz0Sm3vG7H9t0iojRvs05Dn7DqcWHau82APcSEnGIB96xyLGAaS82IWUE9EK8u8R\ng/0/yHNr4XgBYj3v0VhufkcV5HkaivwvIGXSVMSIjUXu829iyCpBRW/lZdMB+ao4Hbnh9kNf2hf2\nH5Hg1USzfyXiB52EPKRnm/SvzbnrS5H3BBIzGIv8N3a1+i8xrukVJT3frC9DXiB3QRRv4NOL+kjh\nfxipmk9HqvFFlCxo7eB7MqmOE8yth7zAzZEX1l0LcOdhRpS810Fqg+2Qwqkq8t83RT4YSsNuree+\ntzNc23YNwa5dHIX4rNeZfNQ3ury+0t1Ea8jxBPLMnoPzQfNPxFBO95BjIW6kHMTo9DTp+cgz4f6I\nWmTyarMAcSstRYygbSRHm9/zJ+SL2W0g1yMF5e2Im28PEiweCTTEu3DdgLi53kAKcZD/dzDwBfL/\nPIsEuSshBnqo+U0vI+/cLhwDsB7nHbEQw/Q9pQfHKyPPSW3kf3O3mIpVXjyP8x7UQO73QnM8Wnmx\nCrgfeU+n4RTk05HyoIdZTkSC7tuAyZT+vHtRC6mN9jH7nyDu+P3I81gJ5+Nwrvn93xEnalAcjsYJ\nan5D9Oqym6XIzberqLWQF6ZNgnkYgfyJ3ZEHY6aRH8ugxOIlnIf6FSQeM5fYDQ686I24Ha7GeVns\nZ8duzZMsvkZeqGgGajZSGBcgL7zNk8j9i7xXj+PMhdMGGS2hhkt2Ncre6g5it9Zrgxg4+yt2L/Jx\nUDVChl0ol9ZQYhniBpqNGL9WSM3lEFLQ/cvIuh75Uu8PdHRdbxvhQ0gNcCLy7I42192A1Ayucl0z\nhHCXZ6SBcTMbabnlxbtR8jgT78J1JFJz/QT5aOhnZN9PfA1KaiEGYG9EeiPkHlrIF73tVq6BvA+R\nH13nI1/r4xBDZj8vlZH//Tmk9lIJKS8O4E2OkXEh0cuLvyPvl7sgd3sq6iDvRE/E8wFiXH6B1MJs\nV2QN8ztjGYBOiEHJJbx1XVukpnIxzodFa+T/Oz2GPCUK5yBfNIOQKqtXtTWSpYRXVWuYtOORF+AV\nnBYvr8chzytAu9QjDaTgBe9WR/bXlH2t/QVcm5JNSePBdp2MIrwVj7s1j1/EijvEun/DkS++H5A8\nb0XcPqW1uouVD/d1rVzXzUZcjjYfIS9rvslD5FIa0RpyrPA41yvNZr5Ze9W8S6uNP4q09GpMeAMU\nEMMzFplz6Gqz2MYpWh6XIgWqTS2c/8qr9d9WwlvuuZeRyH35FfLePYjUwB40cmI1dpmAfKF/i7z7\n04y8WshHwL/Ntc/j1Kbs+xgPVYHVrt/lVV54xWFtFiDP2suIMWnuOlZA+EdKtTjythpxif0CMSr2\nAhLb+xIpJzogta8yjdmlMRThX8gNLiT8q7e0/gtvIF8TE5EvlSsQw/EB8sdMw3EBxBr9+Fbki/Bk\nwg1IHRzDEYntR64dQ6795fIT4lrZiXzBlJWHzPoPlOw74DezKBnMjef+XYO4DxYivvuGSPPeYpz7\nVB1xP7ib30Yj8rqVrutqEm4oLkMK7LNJjGgNOX5Evt5nm/O64hRO7sYDRyE1Ftv9+DNSONrB3e6U\n3qfjeuQZzotIPwn5vQdw3Cg2E5H77ZXHdoS7x9zb9jt4GfJR9hHikrENq5eb5wOcmmtkv41YjV1a\nIM9GP6TGNhb56NqH1CzGIfd+JGLcKpvjz5tj+wivacaK10QrLxoi/clseTY/Ape48h1JZcJrRvsp\nWQuOZDtOM+FIJpvrpyHlylWU7KsSEzUoQgfkjy/rkPdPIf5K20X1G+QreRBlmHIYeYinIC2d7iW8\npdTOMubJzSTkZXgCp6B5JfrppRLrpU01tqGoTMlgbiWk8Il1/35GCqpDiMvlB6Rz1wqit7qLRazW\net8hLfreNHn5FU6fjbMpaZBL+3CJbMgxCqmZnoQYzI3I83ciTgHgbjxwyOTNdnX+ASk8c8z+j5Qe\nfM2Ncczr2qVmqRIlj6/gXbiCd+u/kMlzNO4hep+eWI1d7AJ5N+Kq3IrTIKEXUku5CPnyH2DS2+Pd\nCfNcwlvPHUJqN3a8Jlp5sd7s3xYh7ySTv6fxjnftQAzhB+ZYP5MWi4eRZtmf4fz2wTjPJ8iHxzok\n9mU3vIgLjaEI7yIBM6/mponwd+SLbHKS5CWD6jgvZqIsQ/pOpIPcGMdqI3mrj/dHwY+IW+Z+pIC4\nG/kSXIQYJ5teOK3uovnEvbCvuxYxHnchhYFdi/wKMQbP4V0TvqMMuiD2vbCQQqwmUmvrjnz9z0Tu\nQTFOTOVkxKjYTai9Gmt4xc/cTEQM80ijC6R2/gTRh+2x89gBp3D9CsdlWQspxJcgQejGSGH/KeJO\n/gvyAVjDJW8tUmvwauAQq7HL75DOk22QGkRtxFWWh/xP45APs6ISUkunAWK8Ep6bidjxrhZILdvu\nY7LJHF9bUsx/eQupSS/HqRW2QPr5uO2BHfC3G17ERUU3KHb1tDby1VGW5nexKMJxA9jB/VS0eIoH\ne2iGyq60RIYiAflqjPbSppPJiP83WjDyJMStOQMpWH9G/otk/44VOMHcXpR8Qb8msZpwIkQLhvdH\nAt7RGjZE8jDi8hyFd75vQr523zK6QIzqr5AaRrKZhhTyf0ZiOr9Bah+XIwWjVzPkoUhhaxu8r3Di\nXe5Wg+5a4zNE7zrgNZzMr5B37UckJvMmYlAqI/HYKTF+U6whTxYjxtBNZJrt9o7H6H2DxAlT8gxW\ndJeX/QKNQKqL7gJgRDnk1sa7SanfJBobiiSWu8l+adOJPc7ZLMRofIUzXpnN60jcYCRS8Cw05z1D\n8ojV4dXuhxKt42WyOR0p6Gy+wAmQNyW+IV8gvvjZcYQ3zhiFNCNOBfWRr+mhyH89A4mtXIR3M2SQ\nGMVQpAb0Ok6/L4juxj2AuHwia0KDcRqn3G/S1yAGoR9itKeb/MxBCu93iG1Q3jD67RjbFqSxwEd4\nx7vsMd7uJtww2DWKp4jOLEp2Qn0X+dDwagBUpve7ohuUfLM+mpJtt2uQOIkMAJkKEo0NRXJ5EvLi\nB68hL9xIxJ2zCMdofIG4Yjoi/8MfEPddMg2K3eH1RSPfxq4JN6DsHdESJVowHLwbNpRGrPjZTsTV\nYruUBlK6Lz9RbFfkViRutgUxJFcQ3gz5TZxmyPcjMS17BInnkED5a0Q3rhOQj5KLkFrar3E+Urx6\nyoO45EBch3PM9ipKf/9OJnpPea941ztmO3Jg0UqUrqsbUia5Pwpt70WmvOeB5VbEKv+EE0RcigTJ\nog3wFw+xmpT6ybs4/tWKQhWk8LwP6cBmB6k/R17yp5Ev7VgDbyabXmaZB/zStW+nJRP7GV6J+Mg3\nIM/zEZwCcSVSCK52nV+acYnVXLs5YjS3m+UDxH2TCi5HCtc2yAfhQsQgx2qGbNMO6RT5DdIBdBHi\nPvL6ArffXfd4XfbQLvlITcmO+XQlvN9LZPP10sZrm4XUOuzzTsZ5Luzm0A8iNQ93c+hEyI2yJIWK\nXkNJVeuqWE1K/cAdG/LrizgIfI4UJLORWElHnCaXS8z+GUhsYZc5L9r4VMkk36yTXRP2ItZXpv31\nenECcqPVaqogfVT8+rq1n+0Q4aNE/IPozZD/iMQxdiLusj/jdJI8Fal1Rbpx7VqYV+uvu00+3MPJ\n1MQxKpFjr5X2Hz+ElENeQ564a4abI647HqmV5RLeSXFwDF3rPdK8OhHblCn2W9ENym6zDCztxDIS\nq0mpH6QqNhR0YhkN26dfB3lZ30CCq16jAiebRPoZJcr6JJ1jE0/8rDlyH1M94RtEL0Sj9fEAiWde\nhdTWbGwD6B4zzM0F5rq/IoW63foLpHD/JWUfWSMag5CGJROQ+zsUx2UYK95V1v5u0YjVl61MVPRW\nXn7Qi8SaoiYDr+EylpL40DCZgm00/oxjNO5AgpsdkJf2K7N84UN+jkE+MJJdE/aD3FKOr0caebRG\nCjj36MqxgsOJMhspRAsIL0TfI3oz5ESI1vrLHjXCPbGVXYgn2nryPOTZ7E7JBiOxWlYWUsae7KlG\nDUp24v4ids/YZn8R/8rroiwgltG4BymIFlK+r0nF4U0kGB9CYlOReI12XF78KkRjNa2O1nqyrP2J\n3FQhvMHIz0gNaCXRm0MHsb+bkoUcg3w9vYO4I3LNUj/qFdnBPcgkS0eXdqKSFLwmmnMvqeDvJD79\ndlmI1Qgh1kRuiRCrwUiux7IPZ9y+I0i8J3IcP0VRlIzCnoskcpbC7wgfyiMZuAdB9aMQfZno/S+S\n3XryaaQmPQ2p1Z1HfI013gJ+j7QiDQTq8lIUpbxE9rtJJW8RvfNqMnA3QmhJuKupudGd7JE1bLxi\nf7E4D4m79KBkv6u0oAZFUZRMItWFaG6MY12R5sMjEPdqZOtJr/ni46E8DUaixV4URVGUOIjWedUv\nyjJvUTwkGvtLZ2ddTyp6PxRFUTKLWJ1XU02q+hM9keB16eysqyiKkvEkGsBOBkFtPVkHcZttwJ/O\npYqiKFmFFqLy+8cjfc0+Q4Zw8XsA2jDU5aUoSiYRGcB+HWdo94pGdaSjpXbWVRRFSQDtvKooiqIo\niqIoiqIoiqIoiqIoiqIoilIBuR8ZoXYx0pM60WE44iEfaeWkKIFCmw0rSvnphgyp3h5pvlmP1M4E\naZHYzHyKklKOSncGFCULaIRM2Wr3BfgR+A/wADIi7VLgJdf5+chshvOREXM7Af8GVgOPmHNygVXI\nZE4rkCHTvXqE90HmNS9AOrnVMumPAcuRGlOiQ3soiqIoPlMLcXN9A/wf0NOkH+s6ZwxwmdmeDvyP\n2VYla0kAAAF0SURBVB4KbAEaAlWBjea6XGTej27mvNeAu13XnwU0QIZTtw3NvYgRq4cYI5u65fht\nihI3WkNRlPKzD4lp/B7YDowDBiHDYMxBBvE7D5l33eZDs15mlm3AAWRiqmbm2EZksD+Qmkp31/WV\nkBF3WyM1lEXAjcCJwG5kAqrXgCvRwQIVn9AYiqIkhyNIbWEG4uL6A9AGMTSbkXGWqrvO3++6zj0W\n1RGc99IdJ6mEd9xkGnC9R3pnoDdwDXC72VaUlKI1FEUpP6cgs/vZtEdcThawE5nhr38Cck9EaiEg\nRsM9ZpWF1H7OQYZTB3G9tTTrHGAKcBdwZgK6FaXMaA1FUcpPbeA5pBA/BKwBbgFCiDtrKzA3yrWx\nWmx9A9yGDIC4HPhnxPEdyJSxb+O0KrsfmWf9A6RGVAn4Uxl/j6IoipJF5FK+WQAVxXfU5aUowUX7\nmiiKoiiKoiiKoiiKoiiKoiiKoiiKoiiKoiiKoiiKoiiKoihB4v8BSfNRHDG2a5MAAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x116bdb310>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fdist1.plot(50, cumulative = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Challenge!**\n",
    "\n",
    "Let's compare the 15 most common tokens of the texts in the NLTK book. You could do this manually, but it will save you time and typing if you define a function and then loop it over the list, 'Library', that you created earlier.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(u',', 18713),\n",
       " (u'the', 13721),\n",
       " (u'.', 6862),\n",
       " (u'of', 6536),\n",
       " (u'and', 6024),\n",
       " (u'a', 4569),\n",
       " (u'to', 4542),\n",
       " (u';', 4072),\n",
       " (u'in', 3916),\n",
       " (u'that', 2982),\n",
       " (u\"'\", 2684),\n",
       " (u'-', 2552),\n",
       " (u'his', 2459),\n",
       " (u'it', 2209),\n",
       " (u'I', 2124)]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def common_words(text):\n",
    "    return FreqDist(text).most_common(15)\n",
    "common_words(text1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Text: Moby Dick by Herman Melville 1851> [(u',', 18713), (u'the', 13721), (u'.', 6862), (u'of', 6536), (u'and', 6024), (u'a', 4569), (u'to', 4542), (u';', 4072), (u'in', 3916), (u'that', 2982), (u\"'\", 2684), (u'-', 2552), (u'his', 2459), (u'it', 2209), (u'I', 2124)]\n",
      "<Text: Sense and Sensibility by Jane Austen 1811> [(u',', 9397), (u'to', 4063), (u'.', 3975), (u'the', 3861), (u'of', 3565), (u'and', 3350), (u'her', 2436), (u'a', 2043), (u'I', 2004), (u'in', 1904), (u'was', 1846), (u'it', 1568), (u'\"', 1506), (u';', 1419), (u'she', 1333)]\n",
      "<Text: The Book of Genesis> [(u',', 3681), (u'and', 2428), (u'the', 2411), (u'of', 1358), (u'.', 1315), (u'And', 1250), (u'his', 651), (u'he', 648), (u'to', 611), (u';', 605), (u'unto', 590), (u'in', 588), (u'that', 509), (u'I', 484), (u'said', 476)]\n",
      "<Text: Inaugural Address Corpus> [(u'the', 9281), (u'of', 6970), (u',', 6840), (u'and', 4991), (u'.', 4676), (u'to', 4311), (u'in', 2527), (u'a', 2134), (u'our', 1905), (u'that', 1688), (u'be', 1460), (u'is', 1403), (u'we', 1141), (u'for', 1075), (u'by', 1036)]\n",
      "<Text: Chat Corpus> [(u'.', 1268), (u'JOIN', 1021), (u'PART', 1016), (u'?', 737), (u'lol', 704), (u'to', 658), (u'i', 648), (u'the', 646), (u'you', 635), (u',', 596), (u'I', 576), (u'a', 568), (u'hi', 546), (u'me', 415), (u'...', 412)]\n",
      "<Text: Monty Python and the Holy Grail> [(u':', 1197), (u'.', 816), (u'!', 801), (u',', 731), (u\"'\", 421), (u'[', 319), (u']', 312), (u'the', 299), (u'I', 255), (u'ARTHUR', 225), (u'?', 207), (u'you', 204), (u'a', 188), (u'of', 158), (u'--', 148)]\n",
      "<Text: Wall Street Journal> [(u',', 4885), (u'the', 4045), (u'.', 3828), (u'of', 2319), (u'to', 2164), (u'a', 1878), (u'in', 1572), (u'and', 1511), (u'*-1', 1123), (u'0', 1099), (u'*', 965), (u\"'s\", 864), (u'for', 817), (u'that', 807), (u'*T*-1', 806)]\n",
      "<Text: Personals Corpus> [(u',', 539), (u'.', 353), (u'/', 110), (u'for', 99), (u'to', 74), (u'and', 74), (u'lady', 68), (u'-', 66), (u'seeks', 60), (u'a', 52), (u'with', 44), (u'S', 36), (u'ship', 33), (u'&', 30), (u'relationship', 29)]\n",
      "<Text: The Man Who Was Thursday by G . K . Chesterton 1908> [(u',', 3488), (u'the', 3291), (u'.', 2717), (u'a', 1713), (u'of', 1710), (u'and', 1568), (u'\"', 1336), (u'to', 1045), (u'in', 888), (u'I', 885), (u'he', 858), (u'that', 841), (u'his', 765), (u'was', 716), (u'you', 580)]\n"
     ]
    }
   ],
   "source": [
    "for book in Library:\n",
    "    words = common_words(book)\n",
    "    print(book, words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring Vocab continued"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As well as counting individual words, we can count other features of vocabulary, such as how often words of different lengths occur. We do this by putting together a number of the commands we've already learned.\n",
    "\n",
    "We could start like this: \n",
    "\n",
    "     [len(word) for word in text1]\n",
    "\n",
    "... but this would print the length of every word in the whole book, so let's skip that bit!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fdist2= FreqDist(len(word) for word in text1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fdist2.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.19255882431878046"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fdist2.freq(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These last two commands tell us that the most common word length is 3, and that these 3 letter words account for about 20% of the book. \n",
    "We can see this just by visually inspecting the list produced by *fdist2.most_common()*, but if this list were too long to inspect readily, or we didn't want to print it, there are other ways to explore it.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a number of functions defined for NLTK's frequency distributions:\n",
    "\n",
    " | Function | Purpose  |\n",
    " |--------------|------------|\n",
    " | fdist = FreqDist(samples) | create a frequency distribution containing the given samples |\n",
    " | fdist[sample] += 1 | increment the count for this sample |\n",
    " | fdist['monstrous']  | count of the number of times a given sample occurred |\n",
    " | fdist.freq('monstrous') | frequency of a given sample |\n",
    " | fdist.N()  |  total number of samples |\n",
    " | fdist.most_common(n)   |  the n most common samples and their frequencies |\n",
    " | for sample in fdist:   |  iterate over the items in fdist, when in the loop, we refer to each item as sample |\n",
    " | fdist.max() | sample with the greatest count |\n",
    " | fdist.tabulate()   |  tabulate the frequency distribution |\n",
    " | fdist.plot()  |   graphical plot of the frequency distribution |\n",
    " | fdist.plot(cumulative=True) | cumulative plot of the frequency distribution |\n",
    " | fdist1 < fdist2 | test if samples in fdist1 occur less frequently than in fdist2 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is possible to select the longest words in a text, which may tell you something about its vocabulary and style"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'RESPONSIBILITIES',\n",
       " u'antiphilosophists',\n",
       " u'constitutionally',\n",
       " u'contradistinction',\n",
       " u'discountenancing',\n",
       " u'disqualification',\n",
       " u'enthusiastically',\n",
       " u'instrumentalities',\n",
       " u'internationality',\n",
       " u'irresponsibility',\n",
       " u'misappropriation',\n",
       " u'misrepresentation',\n",
       " u'misunderstanding',\n",
       " u'responsibilities',\n",
       " u'sentimentalizing',\n",
       " u'transcontinental',\n",
       " u'uncharitableness',\n",
       " u'unconstitutional']"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab = set(text4)\n",
    "long_words = [word for word in vocab if len(word) > 15]\n",
    "sorted(long_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use numerical operators to refine the types of searches we ask Python to run. We can use the following relational operators:\n",
    "\n",
    "\n",
    "### Common relationals\n",
    " |  Relational | Meaning |\n",
    " |--------------:|:------------|\n",
    " | <    |  less than |\n",
    " | <=   |   less than or equal to |\n",
    " | ==  |    equal to (note this is two \"=\" signs, not one) |\n",
    " | !=   |   not equal to |\n",
    " | \\>   |   greater than |\n",
    " | \\>= |   greater than or equal to |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Challenge!**\n",
    "\n",
    "Using one of the pre-defined sentences in the NLTK corpus, use the relational operators above to find:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Words longer than four characters\n",
    "- Words of four or more characters\n",
    "- Words of exactly four characters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['family', 'Dashwood', 'settled', 'Sussex'] ['family', 'Dashwood', 'long', 'been', 'settled', 'Sussex'] ['long', 'been']\n"
     ]
    }
   ],
   "source": [
    "longer = [word for word in sent2 if len(word) > 4]\n",
    "more = [word for word in sent2 if len(word) >= 4]\n",
    "exact = [word for word in sent2 if len(word) == 4]\n",
    "print(longer, more, exact)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "family\n",
      "Dashwood\n",
      "settled\n",
      "Sussex\n"
     ]
    }
   ],
   "source": [
    "for word in sent2:\n",
    "    if len(word) > 4:\n",
    "        print(word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can fine-tune our selection even further by adding other conditions. For instance, we might want to find long words that occur frequently (or rarely).  \n",
    "\n",
    "**Challenge!**\n",
    "\n",
    "Can you find all the words in a text that are more than seven letters long and occur more than seven times?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'#14-19teens',\n",
       " u'#talkcity_adults',\n",
       " u'((((((((((',\n",
       " u'........',\n",
       " u'Question',\n",
       " u'actually',\n",
       " u'anything',\n",
       " u'computer',\n",
       " u'cute.-ass',\n",
       " u'everyone',\n",
       " u'football',\n",
       " u'innocent',\n",
       " u'listening',\n",
       " u'remember',\n",
       " u'seriously',\n",
       " u'something',\n",
       " u'together',\n",
       " u'tomorrow',\n",
       " u'watching']"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fdist5 = FreqDist(text5)\n",
    "sorted(word for word in set(text5) if len(word) > 7 and fdist5[word] > 7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Common operators\n",
    "\n",
    " | Operator  | Purpose  |\n",
    " |--------------|------------|\n",
    " | s.startswith(t) | test if s starts with t |\n",
    " | s.endswith(t)  |  test if s ends with t | \n",
    " | t in s         |  test if t is a substring of s | \n",
    " | s.islower()    |  test if s contains cased characters and all are lowercase | \n",
    " | s.isupper()    |  test if s contains cased characters and all are uppercase | \n",
    " | s.isalpha()    |  test if s is non-empty and all characters in s are alphabetic | \n",
    " | s.isalnum()    |  test if s is non-empty and all characters in s are alphanumeric | \n",
    " | s.isdigit()    |  test if s is non-empty and all characters in s are digits | \n",
    " | s.istitle()    |  test if s contains cased characters and is titlecased (i.e. all words in s have initial capitals) | "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'comfortableness',\n",
       " u'honourableness',\n",
       " u'immutableness',\n",
       " u'indispensableness',\n",
       " u'indomitableness',\n",
       " u'intolerableness',\n",
       " u'palpableness',\n",
       " u'reasonableness',\n",
       " u'uncomfortableness']"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(w for w in set(text1) if w.endswith('ableness'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['29', '61']"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(n for n in sent7 if n.isdigit())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Bonus!**\n",
    "\n",
    "You'll remember right at the beginning we started looking at the size of the vocabulary of a text, but there were two problems with the results we got from using:\n",
    "\n",
    "     len(set(text1)).\n",
    "\n",
    "This count includes items of punctuation and treats capitalised and non-capitalised words as different things (*This* vs *this*). We can now fix these problems. We start by getting rid of capitalised words, then we get rid of the punctuation and numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19317"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(text1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17231"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(word.lower() for word in text1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16948"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(word.lower() for word in text1 if word.isalpha()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img style=\"float:left\" src=\"http://images.catsolonline.com/cache/custom/CEN/CE651-250x250.jpg\" />\n",
    "<br>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}