empty/Session 5 - Ngram Language Model with NLTK.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_uuid": "10bbe6360f9659217f9f40455301aa72d6b614a6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already up-to-date: pip in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (19.3.1)\n",
      "Collecting dill\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/c7/11/345f3173809cea7f1a193bfbf02403fff250a3360e0e118a1630985e547d/dill-0.3.1.1.tar.gz (151kB)\n",
      "\u001b[K     |████████████████████████████████| 153kB 5.3MB/s eta 0:00:01\n",
      "\u001b[?25hBuilding wheels for collected packages: dill\n",
      "  Building wheel for dill (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for dill: filename=dill-0.3.1.1-cp36-none-any.whl size=78532 sha256=16cf7606c2ec2eedcc4a2d86795134ac2796ecc82ced001abb24463070cffa76\n",
      "  Stored in directory: /Users/liling.tan/Library/Caches/pip/wheels/59/b1/91/f02e76c732915c4015ab4010f3015469866c1eb9b14058d8e7\n",
      "Successfully built dill\n",
      "\u001b[31mERROR: thinc 6.10.3 has requirement dill<0.3.0,>=0.2.7, but you'll have dill 0.3.1.1 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: thinc 6.10.3 has requirement wrapt<1.11.0,>=1.10.0, but you'll have wrapt 1.11.2 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: spacy 2.0.12 has requirement dill<0.3,>=0.2, but you'll have dill 0.3.1.1 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: google-gax 0.15.16 has requirement dill<0.3dev,>=0.2.5, but you'll have dill 0.3.1.1 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: google-cloud-trace 0.17.0 has requirement google-api-core<0.2.0dev,>=0.1.1, but you'll have google-api-core 1.10.0 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: google-cloud-trace 0.17.0 has requirement google-cloud-core[grpc]<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.29.1 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: google-cloud-speech 0.30.0 has requirement google-api-core<0.2.0dev,>=0.1.1, but you'll have google-api-core 1.10.0 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: google-cloud-speech 0.30.0 has requirement google-cloud-core[grpc]<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.29.1 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: google-cloud-spanner 0.29.0 has requirement google-api-core<0.2.0dev,>=0.1.1, but you'll have google-api-core 1.10.0 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: google-cloud-spanner 0.29.0 has requirement google-cloud-core[grpc]<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.29.1 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: google-cloud-firestore 0.28.0 has requirement google-api-core<0.2.0dev,>=0.1.1, but you'll have google-api-core 1.10.0 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: google-cloud-firestore 0.28.0 has requirement google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.29.1 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: google-cloud-datastore 1.4.0 has requirement google-api-core<0.2.0dev,>=0.1.1, but you'll have google-api-core 1.10.0 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: google-cloud-datastore 1.4.0 has requirement google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.29.1 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: google-cloud-bigtable 0.28.1 has requirement google-api-core<0.2.0dev,>=0.1.1, but you'll have google-api-core 1.10.0 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: google-cloud-bigtable 0.28.1 has requirement google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.29.1 which is incompatible.\u001b[0m\n",
      "Installing collected packages: dill\n",
      "  Found existing installation: dill 0.2.8.2\n",
      "    Uninstalling dill-0.2.8.2:\n",
      "\u001b[31mERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/dill-0.2.8.2-py3.6.egg-info/PKG-INFO'\n",
      "Consider using the `--user` option or check the permissions.\n",
      "\u001b[0m\n",
      "Requirement already up-to-date: nltk in /Users/liling.tan/Library/Python/3.6/lib/python/site-packages (3.4.5)\n",
      "Requirement already satisfied, skipping upgrade: six in /Users/liling.tan/Library/Python/3.6/lib/python/site-packages (from nltk) (1.12.0)\n"
     ]
    }
   ],
   "source": [
    "!pip3 install -U pip\n",
    "!pip3 install -U dill\n",
    "!pip3 install -U nltk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "65559326e76d2b7841244e1dd6ea448671411b95"
   },
   "source": [
    "# N-grams Language Models (N-grams LM)\n",
    "\n",
    "Nowadays, everything seems to be going neural... \n",
    "\n",
    "Traditionally, we can use n-grams to generate language models to predict which word comes next given a history of words. \n",
    "\n",
    "We'll use the `lm` module in `nltk` to get a sense of how non-neural language modelling is done.\n",
    "\n",
    "(**Source:** The content in this notebook is largely based on [language model tutorial in NLTK documentation by Ilia Kurenkov](https://github.com/nltk/nltk/blob/develop/nltk/lm/__init__.py))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "_uuid": "1b213fb0dd5534ce82d6f1c716e9695ba5cf9758"
   },
   "outputs": [],
   "source": [
    "from nltk.util import pad_sequence\n",
    "from nltk.util import bigrams\n",
    "from nltk.util import ngrams\n",
    "from nltk.util import everygrams\n",
    "from nltk.lm.preprocessing import pad_both_ends\n",
    "from nltk.lm.preprocessing import flatten"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "800f7632b6a834d36b95211a912b390fcb9dc11b"
   },
   "source": [
    "If we want to train a bigram model, we need to turn this text into bigrams. Here's what the first sentence of our text would look like if we use the `ngrams` function from NLTK for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "_uuid": "1c1716261478af5e75bba7799bfefc10bd1ea4ea"
   },
   "outputs": [],
   "source": [
    "text = [['a', 'b', 'c'], ['a', 'c', 'd', 'c', 'e', 'f']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "_uuid": "a5a1c775bab7cf9c67656d4349d8bfca02a80738"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('a', 'b'), ('b', 'c')]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(bigrams(text[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "_uuid": "0219988318e39dd0576913b8087af8b35cbdab2f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('a', 'c', 'd'), ('c', 'd', 'c'), ('d', 'c', 'e'), ('c', 'e', 'f')]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(ngrams(text[1], n=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e3a92b25dc25f3ae86fd265880b7f410e981c670"
   },
   "source": [
    "Notice how \"b\" occurs both as the first and second member of different bigrams but \"a\" and \"c\" don't? \n",
    "\n",
    "Wouldn't it be nice to somehow indicate how often sentences start with \"a\" and end with \"c\"?\n",
    "\n",
    "\n",
    "A standard way to deal with this is to add special \"padding\" symbols to the sentence before splitting it into ngrams. Fortunately, NLTK also has a function for that, let's see what it does to the first sentence.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "_uuid": "1ec58c94f58429c9160b4496c939f211f88ade54"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<s>', 'a', 'b', 'c', '</s>']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.util import pad_sequence\n",
    "list(pad_sequence(text[0],\n",
    "                  pad_left=True, left_pad_symbol=\"<s>\",\n",
    "                  pad_right=True, right_pad_symbol=\"</s>\",\n",
    "                  n=2)) # The n order of n-grams, if it's 2-grams, you pad once, 3-grams pad twice, etc. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "_uuid": "ae38644238aa46ac100381c0de046972090b093b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('<s>', 'a'), ('a', 'b'), ('b', 'c'), ('c', '</s>')]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "padded_sent = list(pad_sequence(text[0], pad_left=True, left_pad_symbol=\"<s>\", \n",
    "                                pad_right=True, right_pad_symbol=\"</s>\", n=2))\n",
    "list(ngrams(padded_sent, n=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "_uuid": "4cda3fcbd1b8ab48431240e04168011953ebc913"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<s>', '<s>', 'a', 'b', 'c', '</s>', '</s>']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(pad_sequence(text[0],\n",
    "                  pad_left=True, left_pad_symbol=\"<s>\",\n",
    "                  pad_right=True, right_pad_symbol=\"</s>\",\n",
    "                  n=3)) # The n order of n-grams, if it's 2-grams, you pad once, 3-grams pad twice, etc. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "_uuid": "bd572a57371a716471d156144e7ebc136072de01"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('<s>', '<s>', 'a'),\n",
       " ('<s>', 'a', 'b'),\n",
       " ('a', 'b', 'c'),\n",
       " ('b', 'c', '</s>'),\n",
       " ('c', '</s>', '</s>')]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "padded_sent = list(pad_sequence(text[0], pad_left=True, left_pad_symbol=\"<s>\", \n",
    "                                pad_right=True, right_pad_symbol=\"</s>\", n=3))\n",
    "list(ngrams(padded_sent, n=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e4d80e5707a2efa7ea5cee601d07bca23bc0ae87"
   },
   "source": [
    "Note the `n` argument, that tells the function we need padding for bigrams.\n",
    "\n",
    "Now, passing all these parameters every time is tedious and in most cases they can be safely assumed as defaults anyway.\n",
    "\n",
    "Thus the `nltk.lm` module provides a convenience function that has all these arguments already set while the other arguments remain the same as for `pad_sequence`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "_uuid": "eb64c3c2a205c5fc5e1fce6f63c3e9038f0d8c4a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<s>', 'a', 'b', 'c', '</s>']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.lm.preprocessing import pad_both_ends\n",
    "list(pad_both_ends(text[0], n=2))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "2d2581fae73d38d623a77b5d523c957df2dc8478"
   },
   "source": [
    "Combining the two parts discussed so far we get the following preparation steps for one sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "_uuid": "1168917cc340d400f6dba9b2703561785481d8e4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('<s>', 'a'), ('a', 'b'), ('b', 'c'), ('c', '</s>')]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(bigrams(pad_both_ends(text[0], n=2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "2778a5e5b8d95bc395ba41af491818140c7f680e"
   },
   "source": [
    "To make our model more robust we could also train it on unigrams (single words) as well as bigrams, its main source of information.\n",
    "NLTK once again helpfully provides a function called `everygrams`.\n",
    "\n",
    "While not the most efficient, it is conceptually simple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "_uuid": "dfdc9effca718a7881f1fdc4562cdfb29164b96c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('<s>',),\n",
       " ('a',),\n",
       " ('b',),\n",
       " ('c',),\n",
       " ('</s>',),\n",
       " ('<s>', 'a'),\n",
       " ('a', 'b'),\n",
       " ('b', 'c'),\n",
       " ('c', '</s>')]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.util import everygrams\n",
    "padded_bigrams = list(pad_both_ends(text[0], n=2))\n",
    "list(everygrams(padded_bigrams, max_len=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "d74b0ba683b733d5a83756361a905625a841e68d"
   },
   "source": [
    "We are almost ready to start counting ngrams, just one more step left.\n",
    "\n",
    "During training and evaluation our model will rely on a vocabulary that defines which words are \"known\" to the model.\n",
    "\n",
    "To create this vocabulary we need to pad our sentences (just like for counting ngrams) and then combine the sentences into one flat stream of words.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "_uuid": "310c937ea7cc847a26f6c3d08c7169e8cd6b354a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<s>', 'a', 'b', 'c', '</s>', '<s>', 'a', 'c', 'd', 'c', 'e', 'f', '</s>']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.lm.preprocessing import flatten\n",
    "list(flatten(pad_both_ends(sent, n=2) for sent in text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "5bab7d656c474d945669e6a86bf4beec3a44189e"
   },
   "source": [
    "In most cases we want to use the same text as the source for both vocabulary and ngram counts.\n",
    "\n",
    "Now that we understand what this means for our preprocessing, we can simply import a function that does everything for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "_uuid": "147ccdd961ac97492ee9f0eadf3dfce7325c6790"
   },
   "outputs": [],
   "source": [
    "from nltk.lm.preprocessing import padded_everygram_pipeline\n",
    "train, vocab = padded_everygram_pipeline(2, text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "b1b15504f2fb4487deec19de1c5ca786e49a42d6"
   },
   "source": [
    "So as to avoid re-creating the text in memory, both `train` and `vocab` are lazy iterators. They are evaluated on demand at training time.\n",
    "\n",
    "For the sake of understanding the output of `padded_everygram_pipeline`, we'll \"materialize\" the lazy iterators by casting them into a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "_uuid": "aa18a8a4a700926c30488c94cf519d8900de8ddf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('<s>',), ('a',), ('b',), ('c',), ('</s>',), ('<s>', 'a'), ('a', 'b'), ('b', 'c'), ('c', '</s>')]\n",
      "\n",
      "[('<s>',), ('a',), ('c',), ('d',), ('c',), ('e',), ('f',), ('</s>',), ('<s>', 'a'), ('a', 'c'), ('c', 'd'), ('d', 'c'), ('c', 'e'), ('e', 'f'), ('f', '</s>')]\n",
      "\n",
      "#############\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['<s>', 'a', 'b', 'c', '</s>', '<s>', 'a', 'c', 'd', 'c', 'e', 'f', '</s>']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_ngrams, padded_sentences = padded_everygram_pipeline(2, text)\n",
    "for ngramlize_sent in training_ngrams:\n",
    "    print(list(ngramlize_sent))\n",
    "    print()\n",
    "print('#############')\n",
    "list(padded_sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "4638a871a58694a77473a590728ebf25047cd6eb"
   },
   "source": [
    "## Lets get some real data and tokenize it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "_uuid": "c679084566e711286d861a622e21ca5c04db7bee"
   },
   "outputs": [],
   "source": [
    "try: # Use the default NLTK tokenizer.\n",
    "    from nltk import word_tokenize, sent_tokenize \n",
    "    # Testing whether it works. \n",
    "    # Sometimes it doesn't work on some machines because of setup issues.\n",
    "    word_tokenize(sent_tokenize(\"This is a foobar sentence. Yes it is.\")[0])\n",
    "except: # Use a naive sentence tokenizer and toktok.\n",
    "    import re\n",
    "    from nltk.tokenize import ToktokTokenizer\n",
    "    # See https://stackoverflow.com/a/25736515/610569\n",
    "    sent_tokenize = lambda x: re.split(r'(?<=[^A-Z].[.?]) +(?=[A-Z])', x)\n",
    "    # Use the toktok tokenizer that requires no dependencies.\n",
    "    toktok = ToktokTokenizer()\n",
    "    word_tokenize = word_tokenize = toktok.tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "_uuid": "9bb181f26fb777464b7d4e6d08beac724864cefe"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import requests\n",
    "import io #codecs\n",
    "\n",
    "\n",
    "# Text version of https://kilgarriff.co.uk/Publications/2005-K-lineer.pdf\n",
    "if os.path.isfile('language-never-random.txt'):\n",
    "    with io.open('language-never-random.txt', encoding='utf8') as fin:\n",
    "        text = fin.read()\n",
    "else:\n",
    "    url = \"https://gist.githubusercontent.com/alvations/53b01e4076573fea47c6057120bb017a/raw/b01ff96a5f76848450e648f35da6497ca9454e4a/language-never-random.txt\"\n",
    "    text = requests.get(url).content.decode('utf8')\n",
    "    with io.open('language-never-random.txt', 'w', encoding='utf8') as fout:\n",
    "        fout.write(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "_uuid": "af4102fcbf643e09319ca6394c175ca64082fd19"
   },
   "outputs": [],
   "source": [
    "# Tokenize the text.\n",
    "tokenized_text = [list(map(str.lower, word_tokenize(sent))) \n",
    "                  for sent in sent_tokenize(text)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "_uuid": "9daa634f65b63882f7192a9489fba198970f3771"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['language',\n",
       " 'is',\n",
       " 'never',\n",
       " ',',\n",
       " 'ever',\n",
       " ',',\n",
       " 'ever',\n",
       " ',',\n",
       " 'random',\n",
       " 'adam',\n",
       " 'kilgarriff',\n",
       " 'abstract',\n",
       " 'language',\n",
       " 'users',\n",
       " 'never',\n",
       " 'choose',\n",
       " 'words',\n",
       " 'randomly',\n",
       " ',',\n",
       " 'and',\n",
       " 'language',\n",
       " 'is',\n",
       " 'essentially',\n",
       " 'non-random',\n",
       " '.']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenized_text[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "_uuid": "91c93337cfea53b0c15a6e8617395dab872fb140"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                       Language is never, ever, ever, random\n",
      "\n",
      "                                                               ADAM KILGARRIFF\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Abstract\n",
      "Language users never choose words randomly, and language is essentially\n",
      "non-random. Statistical hypothesis testing uses a null hypothesis, which\n",
      "posits randomness. Hence, when we look at linguistic phenomena in cor-\n",
      "pora, the null hypothesis will never be true. Moreover, where there is enough\n",
      "data, we shall (almost) always be able to establish \n"
     ]
    }
   ],
   "source": [
    "print(text[:500])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "_uuid": "3eef00b8b1a95043e1c8bdd6912ae22060198e5c"
   },
   "outputs": [],
   "source": [
    "# Preprocess the tokenized text for 3-grams language modelling\n",
    "n = 3\n",
    "train_data, padded_sents = padded_everygram_pipeline(n, tokenized_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "32c5ad89e3e61b143e940d50a0d0ee602dadfe3f"
   },
   "source": [
    "# Training an N-gram Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "ff1ff506e3a14df8283cb93b0a45f86862d3e3c7"
   },
   "source": [
    "Having prepared our data we are ready to start training a model. As a simple example, let us train a Maximum Likelihood Estimator (MLE).\n",
    "\n",
    "We only need to specify the highest ngram order to instantiate it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "_uuid": "2635458b0f3fed618bfae6a0705ccb11555bfb0b"
   },
   "outputs": [],
   "source": [
    "from nltk.lm import MLE\n",
    "model = MLE(n) # Lets train a 3-grams model, previously we set n=3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "69cdd0e5babaf059a7b9bc87dc0a6e261cef6cb2"
   },
   "source": [
    "Initializing the MLE model, creates an empty vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "_uuid": "93b95f89f020585dabfbcd6c41273c823a91e882"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(model.vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "fb567e332a0089c86e388a8c93f32d1a31737a29"
   },
   "source": [
    "... which gets filled as we fit the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "_uuid": "a31ecc7f6df30a42df9ed6a79039b50f734299cf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Vocabulary with cutoff=1 unk_label='<UNK>' and 1391 items>\n"
     ]
    }
   ],
   "source": [
    "model.fit(train_data, padded_sents)\n",
    "print(model.vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "_uuid": "6fd7e68c2cdacbe1901fd3b8c05c440706d19100"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1391"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(model.vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "74622c58a2df2ac08c8b0c79644d1f87f4014f44"
   },
   "source": [
    "The vocabulary helps us handle words that have not occurred during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "_uuid": "e5670e34f9d1950f6a1515d57db18092d39484f2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('language', 'is', 'never', ',', 'ever', ',', 'ever', ',', 'random', 'adam', 'kilgarriff', 'abstract', 'language', 'users', 'never', 'choose', 'words', 'randomly', ',', 'and', 'language', 'is', 'essentially', 'non-random', '.')\n"
     ]
    }
   ],
   "source": [
    "print(model.vocab.lookup(tokenized_text[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "_uuid": "cfec70768e8976073dda05f702f3f210a33dc1c3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('language', 'is', 'never', 'random', '<UNK>', '.')\n"
     ]
    }
   ],
   "source": [
    "# If we lookup the vocab on unseen sentences not from the training data, \n",
    "# it automatically replace words not in the vocabulary with `<UNK>`.\n",
    "print(model.vocab.lookup('language is never random lah .'.split()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "83a71e4b25b53795dedce91c1a5686b7e22279e0"
   },
   "source": [
    "Moreover, in some cases we want to ignore words that we did see during training but that didn't occur frequently enough, to provide us useful information. \n",
    "\n",
    "You can tell the vocabulary to ignore such words using the `unk_cutoff` argument for the vocabulary lookup, To find out how that works, check out the docs for the [`nltk.lm.vocabulary.Vocabulary` class](https://github.com/nltk/nltk/blob/develop/nltk/lm/vocabulary.py)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "b0310a2f3c7d9e90f4574b44838336ca1e4ef2ef"
   },
   "source": [
    "**Note:** For more sophisticated ngram models, take a look at [these objects from `nltk.lm.models`](https://github.com/nltk/nltk/blob/develop/nltk/lm/models.py):\n",
    "\n",
    " - `Lidstone`: Provides Lidstone-smoothed scores.\n",
    " - `Laplace`: Implements Laplace (add one) smoothing.\n",
    " - `InterpolatedLanguageModel`: Logic common to all interpolated language models (Chen & Goodman 1995).\n",
    " - `WittenBellInterpolated`: Interpolated version of Witten-Bell smoothing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "af5a0851a2f8707ea2e172681342ed3ecd872328"
   },
   "source": [
    "# Using the N-gram Language Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "d098fd4686bcdc0ab9aa72dbbd4b62a17ee2c332"
   },
   "source": [
    "When it comes to ngram models the training boils down to counting up the ngrams from the training corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "_uuid": "bfc60d61539269298390044c0d3415bfd28e2b1e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<NgramCounter with 3 ngram orders and 19611 ngrams>\n"
     ]
    }
   ],
   "source": [
    "print(model.counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e9b8cac68de80aee9c7cf7b7f2faf2b199ad27cc"
   },
   "source": [
    "This provides a convenient interface to access counts for unigrams..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "_uuid": "90f4611580c41118747a42cfac4c9d729f02523f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.counts['language'] # i.e. Count('language')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "ccdf1bf83ba16f0a43f04eb96ca224a968cc81d2"
   },
   "source": [
    "...and bigrams for the phrase \"language is\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "_uuid": "3ba55a83d7c052d1f14923c4046a5f6c86d72c8c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.counts[['language']]['is'] # i.e. Count('is'|'language')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "88bceda0a01e9f2642dd30a6b341c24a606720b9"
   },
   "source": [
    "... and trigrams for the phrase \"language is never\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "_uuid": "813e30e1fb153002cfb020d430def2510bf43bbc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.counts[['language', 'is']]['never'] # i.e. Count('never'|'language is')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "2c4338d351cfc3cd785673ad9581ecab161b24c6"
   },
   "source": [
    "And so on. However, the real purpose of training a language model is to have it score how probable words are in certain contexts.\n",
    "\n",
    "This being MLE, the model returns the item's relative frequency as its score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "_uuid": "051ef5a06c004a8e8ddb6168ca318bc8c0c9abf4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.003691671588895452"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.score('language') # P('language')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "_uuid": "ab3d48919b8f074342e624a1da441621829a4dbd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.44"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.score('is', 'language'.split())  # P('is'|'language')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "_uuid": "73955908d499f459e19c2cbfaec7d94c513acb91"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6363636363636364"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.score('never', 'language is'.split())  # P('never'|'language is')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "946dde75e8d5b8d8878271359b0d91926abeb199"
   },
   "source": [
    "Items that are not seen during training are mapped to the vocabulary's \"unknown label\" token.  This is \"<UNK>\" by default.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "_uuid": "44038e8cd8d078b734a6b1a803d0795517e2fa82"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.score(\"<UNK>\") == model.score(\"lah\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "_uuid": "f2f170a0bbca4b33b5eb1a55a55f0e1b1b18a634"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.score(\"<UNK>\") == model.score(\"leh\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "_uuid": "9107d1a7c27f30a449e9f370002a6406d5a1396d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.score(\"<UNK>\") == model.score(\"lor\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "1a9ad0be90c72ece6c2a1c00314ea60f41a00f81"
   },
   "source": [
    "To avoid underflow when working with many small score values it makes sense to take their logarithm. \n",
    "\n",
    "For convenience this can be done with the `logscore` method.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "_uuid": "20ef81b7e161df3b9ab27236c33559713b3077ce"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.6520766965796932"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.logscore(\"never\", \"language is\".split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "18f9e0a8d0aba302532b8a84f342d1bf4d3e202a"
   },
   "source": [
    "# Generation using N-gram Language Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "f893159fe093aef484c07b37d922de4ac5727834"
   },
   "source": [
    "One cool feature of ngram models is that they can be used to generate text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "_uuid": "98989ec4bae592fc98332e759daf9e42bac4213e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['and', 'carroll', 'used', 'hypothesis', 'testing', 'has', 'been', 'used', ',', 'and', 'a', 'half', '.', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>']\n"
     ]
    }
   ],
   "source": [
    "print(model.generate(20, random_seed=7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "3b8d07eaf3afae978131573ae127fa0ec3f2e50d"
   },
   "source": [
    "We can do some cleaning to the generated tokens to make it human-like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "_uuid": "73d0d5e0029e64876100e0f2e368b4835a99efcd"
   },
   "outputs": [],
   "source": [
    "from nltk.tokenize.treebank import TreebankWordDetokenizer\n",
    "\n",
    "detokenize = TreebankWordDetokenizer().detokenize\n",
    "\n",
    "def generate_sent(model, num_words, random_seed=42):\n",
    "    \"\"\"\n",
    "    :param model: An ngram language model from `nltk.lm.model`.\n",
    "    :param num_words: Max no. of words to generate.\n",
    "    :param random_seed: Seed value for random.\n",
    "    \"\"\"\n",
    "    content = []\n",
    "    for token in model.generate(num_words, random_seed=random_seed):\n",
    "        if token == '<s>':\n",
    "            continue\n",
    "        if token == '</s>':\n",
    "            break\n",
    "        content.append(token)\n",
    "    return detokenize(content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "_uuid": "949378240cbc8247579b79946b113a3afd039b39"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'and carroll used hypothesis testing has been used, and a half.'"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generate_sent(model, 20, random_seed=7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "_uuid": "9041c514e0458236132fe9b3c42ac2ce651beb92"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['the', 'scf-verb', 'link', 'is', 'motivated', '.', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>', '</s>']\n"
     ]
    }
   ],
   "source": [
    "print(model.generate(28, random_seed=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "_uuid": "523e7c2373f6a4c4b6a542f1e82369fedb6cbd21"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'the scf-verb link is motivated.'"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generate_sent(model, 28, random_seed=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "_uuid": "bce2885bd22d0bb1f79c3861f10fde7df6714e40"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'237⫺246.'"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generate_sent(model, 20, random_seed=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "_uuid": "90ddb0e7d0fb52bc77d08425331944c55ee0885e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'hypothesis is ever a useful construct.'"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generate_sent(model, 20, random_seed=30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "_uuid": "ba9546c6800d9e0b0f0af4f0fd31a141af93b12f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'more (or cold) weather, or on saturday nights, or by people in (or poorer )'"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generate_sent(model, 20, random_seed=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Saving the model \n",
    "\n",
    "The native Python's pickle may not save the lambda functions in the  model, so we can use the `dill` library in place of pickle to save and load the language model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dill as pickle \n",
    "\n",
    "with open('kilgariff_ngram_model.pkl', 'wb') as fout:\n",
    "    pickle.dump(model, fout)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('kilgariff_ngram_model.pkl', 'rb') as fin:\n",
    "    model_loaded = pickle.load(fin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'more (or cold) weather, or on saturday nights, or by people in (or poorer )'"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generate_sent(model_loaded, 20, random_seed=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e4f13ed7359c87e397229104c1bcf18eb20603ad"
   },
   "source": [
    "# Lets try some generating with Donald Trump data!!!\n",
    "\n",
    "\n",
    "**Dataset:** https://www.kaggle.com/kingburrito666/better-donald-trump-tweets#Donald-Tweets!.csv\n",
    "\n",
    "\n",
    "In this part, I'll be munging that data as how I would be doing it at work. \n",
    "I've really no seen the data before but I hope this session would be helpful for you to see how to approach new datasets with the skills you have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "_uuid": "caf5fea33f84e06c3cf613136ec33f80e774fc98"
   },
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] File b'../input/Donald-Tweets!.csv' does not exist: b'../input/Donald-Tweets!.csv'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-51-a9fa6562fab4>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mpandas\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mdf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'../input/Donald-Tweets!.csv'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhead\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Library/Python/3.6/lib/python/site-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36mparser_f\u001b[0;34m(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, dialect, error_bad_lines, warn_bad_lines, delim_whitespace, low_memory, memory_map, float_precision)\u001b[0m\n\u001b[1;32m    683\u001b[0m         )\n\u001b[1;32m    684\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 685\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0m_read\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    686\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    687\u001b[0m     \u001b[0mparser_f\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Library/Python/3.6/lib/python/site-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m_read\u001b[0;34m(filepath_or_buffer, kwds)\u001b[0m\n\u001b[1;32m    455\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    456\u001b[0m     \u001b[0;31m# Create the parser.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 457\u001b[0;31m     \u001b[0mparser\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTextFileReader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfp_or_buf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    458\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    459\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mchunksize\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0miterator\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Library/Python/3.6/lib/python/site-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, f, engine, **kwds)\u001b[0m\n\u001b[1;32m    893\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"has_index_names\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"has_index_names\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    894\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 895\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_engine\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mengine\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    896\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    897\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Library/Python/3.6/lib/python/site-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m_make_engine\u001b[0;34m(self, engine)\u001b[0m\n\u001b[1;32m   1133\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_make_engine\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mengine\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"c\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1134\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mengine\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"c\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1135\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_engine\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mCParserWrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1136\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1137\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mengine\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"python\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Library/Python/3.6/lib/python/site-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, src, **kwds)\u001b[0m\n\u001b[1;32m   1915\u001b[0m         \u001b[0mkwds\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"usecols\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0musecols\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1916\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1917\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_reader\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparsers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTextReader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msrc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1918\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munnamed_cols\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_reader\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munnamed_cols\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1919\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader.__cinit__\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._setup_parser_source\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] File b'../input/Donald-Tweets!.csv' does not exist: b'../input/Donald-Tweets!.csv'"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv('Donald-Tweets!.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "a15d9d1a754ca357ae79e8390069b08b05320458"
   },
   "outputs": [],
   "source": [
    "trump_corpus = list(df['Tweet_Text'].apply(word_tokenize))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "8724b8724cc52b685205047bd1fda649545a6a24"
   },
   "outputs": [],
   "source": [
    "# Preprocess the tokenized text for 3-grams language modelling\n",
    "n = 3\n",
    "train_data, padded_sents = padded_everygram_pipeline(n, trump_corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "c5010b607062f6f2dd0ccf011bb9004e1e3dcee7"
   },
   "outputs": [],
   "source": [
    "from nltk.lm import MLE\n",
    "trump_model = MLE(n) # Lets train a 3-grams model, previously we set n=3\n",
    "trump_model.fit(train_data, padded_sents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "_uuid": "8036f1ae5e0e1438d1d2981026766cdacd4479aa"
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'trump_model' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-52-6563e0010488>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mgenerate_sent\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrump_model\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_words\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m20\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrandom_seed\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m42\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'trump_model' is not defined"
     ]
    }
   ],
   "source": [
    "generate_sent(trump_model, num_words=20, random_seed=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "_uuid": "defe4e4899cb14300710eb0c7786a6f2e894455d"
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'trump_model' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-53-96d7200ebc93>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mgenerate_sent\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrump_model\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_words\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrandom_seed\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'trump_model' is not defined"
     ]
    }
   ],
   "source": [
    "generate_sent(trump_model, num_words=10, random_seed=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "_uuid": "eba7934efa5a5e6b818e951c5daec9e9f621ca26"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\\\\" @ about_life: @ realDonaldTrump Trumps new VA plan will help the veterans like no one is even higher than anticipated in radical Islamic terrorism, as her running mate @ realDonaldTrump Trumps new VA plan will help the veterans like no one is even higher than anticipated in'"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generate_sent(trump_model, num_words=50, random_seed=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "_uuid": "3ec48d8d44b2f57b16249e12b845f43f52faadca"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "will win this thing! GET OUT TO VOTE Then``WE THE PEOPLE love you trump\"Nice\n"
     ]
    }
   ],
   "source": [
    "print(generate_sent(trump_model, num_words=100, random_seed=52))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "2567d003d69c26233b915df03509b3fd014beed4"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4rc1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}