UnsupervisedNLP.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:d487a559a2178f2ec8f9812ca4dde51d724651d8b20cef6bc5858c45359a21b8"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Unsupervised Learning in VW\n",
      "\n",
      "# DO NOT READ THIS YET, THERE IS A BUG IN VW LDA THAT WE'RE WORKING ON!\n",
      "\n",
      "Up to this point, all we've talked about has been supervised learning. For the most part, `vw` *is* a toolkit for supervised learning, but it does have a few ways to do unsupervised learning. The first is built-in support for topic models (in this case, streaming latent Dirichlet allocation). The second is a style of matrix factorization that we can use to construct autoencoders.\n",
      "\n",
      "# Topics Models in VW\n",
      "\n",
      "Before we begin, we need to make sure we have some data. We'll re-use the 20 Newsgroups data from the [Multiclass Classification](MulticlassClassification.ipynb) notebook. If you haven't downloaded it yet, please go execute the first two command blocks in that notebook. If that has succeeded, then the following should work:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!wc data/20ng.t[re]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "    7532  2932098 13603023 data/20ng.te\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "   11314  4877902 22111515 data/20ng.tr\r\n",
        "   18846  7810000 35714538 total\r\n"
       ]
      }
     ],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "First, we have to get the data into LDA-friendly format. This means (a) removing the labels (because this is unsupervised learning, after all!); (b) reducing the data to a single, un-named namespace; and (c) reducing everything to words and their counts. LDA also tends to work better after stopwords have been removed; we'll use a list of stopwords from [ranks.nl](http://www.ranks.nl/stopwords). We'll do this all below with a bit of python."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import re\n",
      "from collections import Counter\n",
      "\n",
      "def tokenize(s): return re.sub('([^A-Za-z0-9 ]+)', ' \\\\1 ', s).split()  # add space around anything not alphanum\n",
      "\n",
      "stopwords = set(tokenize(\"\"\"a about above after again against all am an and any are aren't as at be because\n",
      "                            been before being below between both but by can't cannot could couldn't did didn't\n",
      "                            do does doesn't doing don't down during each few for from further had hadn't has\n",
      "                            hasn't have haven't having he he'd he'll he's her here here's hers herself him\n",
      "                            himself his how how's i i'd i'll i'm i've if in into is isn't it it's its itself\n",
      "                            let's me more most mustn't my myself no nor not of off on once only or other ought\n",
      "                            our ours ourselves out over own same shan't she she'd she'll she's should shouldn't\n",
      "                            so some such than that that's the their theirs them themselves then there there's\n",
      "                            these they they'd they'll they're they've this those through to too under until up\n",
      "                            very was wasn't we we'd we'll we're we've were weren't what what's when when's\n",
      "                            where where's which while who who's whom why why's with won't would wouldn't you\n",
      "                            you'd you'll you're you've your yours yourself yourselves\"\"\"))\n",
      "\n",
      "def countWords(txt):\n",
      "    match = re.match('^.* \\|w ([^\\|]*)', txt)\n",
      "    if match is None: return Counter()\n",
      "    words = match.group(1).lower().split()\n",
      "    return Counter([w for w in words if w not in stopwords and w.isalpha() and len(w) >= 3 and len(w) < 20])\n",
      "\n",
      "def computeDocumentFrequencies(inputFile):\n",
      "    df = Counter()\n",
      "    numDocs = 0\n",
      "    with open(inputFile, 'r') as h:\n",
      "        for l in h.readlines():\n",
      "            count = countWords(l)\n",
      "            for w in count.iterkeys():\n",
      "                df[w] += 1\n",
      "            numDocs += 1\n",
      "    return df,numDocs\n",
      "\n",
      "def dataToLDA(inputFile, outputHandle, df, minFreq, maxFreq):\n",
      "    with open(inputFile, 'r') as h:\n",
      "        for l in h.readlines():\n",
      "            count = countWords(l)\n",
      "            if len(count) > 0:  # unfortunately, the current vw segfaults on empty lines\n",
      "                print >>outputHandle, '|', ' '.join(['%s:%d' % (w,c)\n",
      "                                                     for w,c in count.iteritems()\n",
      "                                                     if df[w] >= minFreq and df[w] <= maxFreq])\n",
      "\n",
      "df,numDocs = computeDocumentFrequencies('data/20ng.tr')\n",
      "with open('data/20ng.unlab', 'w') as h:\n",
      "    dataToLDA('data/20ng.tr', h, df, 10, numDocs/10)\n",
      "    \n",
      "!head -n2 data/20ng.unlab\n",
      "!wc -l data/20ng.unlab"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "| saying:1 money:1 attempts:1 hell:1 beliefs:1 seems:2 symptoms:1 treatment:1 covered:1 happened:1 combination:1 read:1 sarcastic:1 geb:1 every:1 accepted:2 condition:2 areas:1 school:1 countries:3 issue:1 switzerland:1 found:2 homeopathy:3 wrote:1 view:1 absolutely:1 direct:1 depends:1 helpless:1 insurance:2 living:1 lead:1 deeply:1 robert:2 scientists:1 medicine:5 reading:1 conditions:1 method:1 med:1 scientific:1 business:1 however:1 enables:1 country:1 changed:1 pitt:1 experience:1 keep:1 gordon:1 germany:1 mean:1 comment:1 pays:1 austria:2 open:1 least:1 doubt:1 taken:1 waste:1 vienna:1 life:2 offer:1 excuse:1 worked:1 case:3 holland:1 patients:1 personality:1 powerless:1 modern:4 mind:2 helped:1 britain:1 cure:2 sense:1 relatively:1 pay:1 offensive:1 note:1 answer:1 european:1 practitioner:1 intended:1 normal:2 treatments:1 paid:1 swiss:1 charm:1 oracle:3 coming:1 makes:1 severe:1 banks:1 sounded:1\r\n",
        "| sometimes:1 washington:1 pitt:2 brain:2 medication:1 throat:2 worry:2 elizabeth:1 given:1 uucp:1 carriers:1 culture:1 boot:1 vessels:1 forms:1 live:1 causing:1 assuming:1 taken:1 becoming:1 common:1 kills:1 swell:1 took:1 geb:1 course:1 blood:1 schools:1 especially:1 covering:1 camp:1 negative:1 gordon:1 carrier:1 banks:1 attacking:1\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "11297 data/20ng.unlab\r\n"
       ]
      }
     ],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now that the data is in LDA-friendly mode, we simply need to tell `vw` to run LDA.\n",
      "\n",
      "Unfortunately, unlike supervised learning, there are a number of hyperparameters that you have to tweak fairly carefully to get reasonable performance out of LDA, more or less because unsupervised learning is hard. The main parameters you have to set are:\n",
      "\n",
      "* How many topics you want! This you have to use your judgment. We'll do 5 just so that it's easy to look at the output.\n",
      "* The Dirichlet prior on p(word|topic). This is called `--lda_rho`. If you want sparsity, this should be less than 1. A starting point of 0.1 is usually reasonable.\n",
      "* The Dirichlet prior on p(topic|document). This is called `--lda_alpha` and 0.1 is also a reasonable starting point here.\n",
      "* The minibatch size: basically how many documents should LDA look at at a single time. 256 is reasonable.\n",
      "\n",
      "Here we go:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!vw -k -c -b20 -d data/20ng.unlab --passes 100 --lda 20 --lda_D 11297 --lda_alpha 0.1 --lda_rho 0.1 --minibatch 512 -f data/20ng.unlab.lda -l 0.1 -p data/20ng.unlab.pred --random_seed 1234 --random_weights on\n",
      "!tail -n2000 data/20ng.unlab.pred | sort -R | tail"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "final_regressor = data/20ng.unlab.lda\r\n",
        "predictions = data/20ng.unlab.pred\r\n",
        "Num weight bits = 20\r\n",
        "learning rate = 0.1\r\n",
        "initial_t = 0\r\n",
        "power_t = 0.5\r\n",
        "decay_learning_rate = 1\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "creating cache_file = data/20ng.unlab.cache\r\n",
        "Reading datafile = data/20ng.unlab\r\n",
        "num sources = 1\r\n",
        "average  since         example        example  current  current  current\r\n",
        "loss     last          counter         weight    label  predict features\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "15.983848 15.983848            1            1.0  unknown   0.0000       96\r\n",
        "16.033634 16.083420            2            2.0  unknown   0.0000       36\r\n",
        "15.997947 15.962260            4            4.0  unknown   0.0000       90\r\n",
        "16.024702 16.051457            8            8.0  unknown   0.0000      147\r\n",
        "17.079831 18.134961           16           16.0  unknown   0.0000      178\r\n",
        "17.608963 18.138095           32           32.0  unknown   0.0000        5\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "17.839076 18.069188           64           64.0  unknown   0.0000       54\r\n",
        "17.829170 17.819265          128          128.0  unknown   0.0000       55\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "17.824724 17.820278          256          256.0  unknown   0.0000       80\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "17.827845 17.830965          512          512.0  unknown   0.0000        8\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "17.848065 17.868286         1024         1024.0  unknown   0.0000       69\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "17.846609 17.845153         2048         2048.0  unknown   0.0000       75\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "17.850727 17.854845         4096         4096.0  unknown   0.0000      115\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "17.837450 17.824174         8192         8192.0  unknown   0.0000       16\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "17.839481 17.841512        16384        16384.0  unknown   0.0000       32\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "17.837647 17.835812        32768        32768.0  unknown   0.0000       99\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "17.837805 17.837963        65536        65536.0  unknown   0.0000       17\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "17.836958 17.836111       131072       131072.0  unknown   0.0000       68\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "17.836983 17.837008       262144       262144.0  unknown   0.0000       21\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "17.837052 17.837121       524288       524288.0  unknown   0.0000       23\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r\n",
        "finished run\r\n",
        "number of examples = 1016800\r\n",
        "weighted example sum = 1016800.000000\r\n",
        "weighted label sum = 0.000000\r\n",
        "average loss = 0.000000 h\r\n",
        "total feature number = 72343800\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "16.815300 0.100029 7.135149 0.100027 0.100026 0.100025 0.100035 0.100000 6.202374 11.961309 0.100027 0.100000 0.100000 0.100000 30.385586 0.100031 0.100000 0.100024 0.100028 0.100032 \r\n",
        "0.100026 0.100020 0.100036 0.100030 0.100028 0.100026 0.100032 0.100000 0.100035 2.770464 4.002769 0.100000 0.100000 0.100000 10.295366 0.100032 0.100000 12.651035 12.780082 0.100020 \r\n",
        "0.100035 0.100029 18.554890 0.100037 0.100032 0.100035 5.396076 0.100000 0.100035 0.100026 0.100033 0.100000 0.100000 0.100000 0.100035 11.562382 0.100000 0.100027 26.366322 9.620002 \r\n",
        "145.182770 31.085636 50.398434 99.868141 116.868889 0.108624 66.067795 0.100000 4.942534 56.463730 0.100030 0.100000 0.100000 0.100000 25.900484 88.307060 0.100000 0.100031 0.100031 99.005722 \r\n",
        "0.100034 14.342416 31.472063 9.750040 21.763355 0.100028 0.100030 0.100000 0.100025 0.100028 0.100034 0.100000 0.100000 0.100000 0.100027 0.100032 0.100000 4.723732 0.100028 11.548128 \r\n",
        "7.564924 0.100031 0.100027 0.100024 0.100033 0.100032 4.798840 0.100000 0.100024 13.499141 0.100027 0.100000 0.100000 0.100000 4.536791 0.100025 0.100000 0.100030 0.100025 0.100028 \r\n",
        "25.012398 11.817080 0.100028 0.100027 0.100032 57.715881 30.296272 0.100000 0.101200 0.100031 0.100031 0.100000 0.100000 0.100000 30.209276 42.206295 0.100000 24.220533 17.320871 0.100029 \r\n",
        "0.100036 3.238522 0.100041 0.100045 10.685641 0.100036 0.100028 0.100000 0.100040 6.746550 0.100022 0.100000 0.100000 0.100000 0.100042 0.100039 0.100000 0.100040 11.728886 0.100032 \r\n",
        "6.544247 0.100038 9.555730 0.100036 0.100022 0.100023 0.100035 0.100000 0.100021 0.100028 4.012021 0.100000 0.100000 0.100000 14.180709 10.207019 0.100000 0.100021 0.100023 0.100030 \r\n",
        "0.100037 0.100036 0.100029 0.100037 0.100032 0.100031 6.457939 0.100000 0.100036 0.100033 0.100028 0.100000 0.100000 0.100000 0.100024 24.260477 0.100000 0.100038 20.440647 20.240576 \r\n"
       ]
      }
     ],
     "prompt_number": 38
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!vw -t -d data/20ng.unlab -i data/20ng.unlab.lda -p data/20ng.unlab.pred --quiet\n",
      "!head -n20 data/20ng.unlab.pred"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "21.715076 23.202663 7.844841 0.100031 0.100033 19.938475 0.100034 0.100000 0.100027 0.100030 0.100030 0.100000 0.100000 0.100000 25.687893 0.100032 0.100000 12.902250 11.408561 0.100025 \r\n",
        "0.100030 14.869518 0.100027 14.881866 0.100025 10.548245 0.100031 0.100000 0.100036 0.100031 0.100035 0.100000 0.100000 0.100000 0.100031 0.100032 0.100000 0.100031 0.100032 0.100029 \r\n",
        "0.100028 0.100033 13.696851 23.532206 0.100026 33.708664 0.100032 0.100000 10.899662 0.100038 0.100031 0.100000 0.100000 0.100000 0.100023 0.100027 0.100000 11.662322 0.100036 0.100026 \r\n",
        "0.100025 32.965546 0.100029 0.100032 14.288982 17.240261 0.100029 0.100000 0.100029 17.485605 0.100033 0.100000 0.100000 0.100000 11.590836 0.100025 0.100000 0.100031 17.028515 0.100030 \r\n",
        "0.100022 0.100033 26.766905 0.100033 0.100030 6.657275 0.100024 0.100000 0.100023 7.312836 15.085741 0.100000 0.100000 0.100000 21.328510 18.448481 0.100000 0.100025 0.100024 0.100035 \r\n",
        "7.674386 11.294849 15.190828 10.945913 10.477317 0.100027 0.100026 0.100000 0.100025 8.016451 0.100035 0.100000 0.100000 0.100000 0.100024 0.100025 0.100000 0.100034 0.100038 0.100027 \r\n",
        "0.100027 0.100020 0.100012 0.100031 0.100030 0.100033 0.100038 0.100000 0.100021 3.301056 6.424484 0.100000 0.100000 0.100000 0.100016 0.100015 0.100000 3.574184 0.100017 0.100016 \r\n",
        "0.100029 0.100029 16.585390 0.100027 0.100030 34.432915 0.100029 0.100000 18.102285 7.212412 79.950989 0.100000 0.100000 0.100000 24.416714 0.100029 0.100000 0.100032 12.999031 0.100033 \r\n",
        "0.100040 15.319128 0.100035 0.100024 0.100036 0.100030 0.100034 0.100000 0.100022 0.100031 0.100029 0.100000 0.100000 0.100000 7.586847 0.100033 0.100000 0.100020 0.100032 14.393661 \r\n",
        "0.101697 12.081765 42.313911 0.100031 55.201290 15.307070 16.451059 0.100000 33.695316 42.657902 0.100030 0.100000 0.100000 0.100000 46.229485 95.495476 0.100000 22.323343 36.293098 27.148571 \r\n",
        "6.316563 0.100049 0.100020 0.100021 0.100014 0.100032 0.100013 0.100000 0.100015 2.477398 0.100028 0.100000 0.100000 0.100000 8.505736 0.100044 0.100000 0.100020 0.100022 0.100023 \r\n",
        "17.756857 0.100026 0.100026 11.236880 0.100032 0.100033 21.933317 0.100000 0.100035 9.471942 0.100027 0.100000 0.100000 0.100000 0.100032 0.100028 0.100000 0.100035 0.100033 10.100698 \r\n",
        "0.100034 0.100037 0.100033 0.100040 0.100046 0.100051 9.074137 0.100000 0.100037 0.100030 19.091578 0.100000 0.100000 0.100000 0.100033 0.100023 0.100000 6.133863 0.100018 0.100046 \r\n",
        "0.100031 0.100032 0.100037 5.835958 0.100023 0.100024 0.100018 0.100000 0.100034 0.100049 0.100027 0.100000 0.100000 0.100000 3.929354 0.100032 0.100000 0.100037 18.534317 0.100030 \r\n",
        "0.100029 0.100034 0.100021 0.100036 0.100027 9.389404 0.100032 0.100000 0.100037 19.837572 0.100031 0.100000 0.100000 0.100000 20.347139 20.825548 0.100000 0.100034 0.100037 0.100024 \r\n",
        "0.100017 2.735978 0.100018 0.100047 0.100021 0.100049 4.274239 0.100000 0.100026 0.100020 0.100032 0.100000 0.100000 0.100000 0.100015 0.100016 0.100000 0.100023 0.100020 2.289477 \r\n",
        "35.242641 9.909006 22.206738 31.911646 0.100343 0.100033 8.737320 0.100000 12.071489 9.889423 56.425446 0.100000 0.100000 0.100000 0.100030 38.502533 0.100000 29.103296 0.100032 0.100030 \r\n",
        "17.962646 12.813928 17.916031 25.007462 14.369626 47.378246 0.100040 0.100000 43.498878 9.396447 7.466072 0.100000 0.100000 0.100000 0.100029 19.701447 0.100000 34.633305 0.100031 50.055798 \r\n",
        "7.346448 0.100030 0.100022 2.790637 0.100033 0.100042 4.827148 0.100000 0.100038 0.100031 0.100022 0.100000 0.100000 0.100000 3.435438 0.100024 0.100000 0.100027 0.100031 0.100027 \r\n",
        "0.100034 0.100022 0.100024 0.100029 0.100035 7.024383 4.535814 0.100000 0.100028 0.100035 0.100022 0.100000 0.100000 0.100000 6.481256 0.100026 0.100000 4.769609 7.688655 0.100027 \r\n"
       ]
      }
     ],
     "prompt_number": 39
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Overall this command line basically looks like a normal `vw` command line. The only annoying thing is that you have to tell LDA how many documents there are total (via `--lda_D`).\n",
      "\n",
      "At the end, we want to get some sort of predictions out. Either we want to know, for some documents, what the topic distribution is for each document (this will be achieved with `-p` predictions, like for supervised learning) or we want to know the topic-to-word distribution (this will be achieved by bending over backwards a bit because unfortunately the current [LDA implementation doesn't support `--invert_hash`](https://groups.yahoo.com/neo/groups/vowpal_wabbit/conversations/topics/3622))."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Here, each line correponds to a document and each column corresponds to a topic. If you want probabilities, you should normalize these to sum to one. The small ones (the ones very close to 0.1) are the ones that are basically only getting their mass from the prior. So the first document is mostly about topic 3 and somewhat about topic 1. The fourth document is mostly about topic 1 and somewhat about topic 3.\n",
      "\n",
      "We can also look at the topic-to-word distribution. Unfortunately, LDA doesn't currently support `--invert_hash` so we have to go with a backdoor solution. What we'll do is create a \"vocabulary\" data set, where each example corresponds to a single word, and we remove duplicates. We can then run this data through `vw` in \"audit\" mode. In \"audit\" mode, `vw` will output some running statistics about what it is computing on each example. We can extract from this output the word-to-topic distribution. It sounds complicated, but it's not soo bad."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!cat data/20ng.unlab | cut -d' ' -f2- | sed 's/:[0-9]*//g' | tr ' ' '\\n' | sort -u | sed 's/^/| /' > data/20ng.unlab.vocab\n",
      "!echo \"Vocabulary examples:\"\n",
      "!head data/20ng.unlab.vocab\n",
      "!echo \"\"\n",
      "!echo \"Running the first few through vw with audit on:\"\n",
      "!head data/20ng.unlab.vocab | vw -t -i data/20ng.unlab.lda --audit --quiet"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Vocabulary examples:\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "| \r\n",
        "| aaa\r\n",
        "| aardvark\r\n",
        "| aaron\r\n",
        "| aau\r\n",
        "| abandon\r\n",
        "| abandoned\r\n",
        "| abbreviation\r\n",
        "| abc\r\n",
        "| aber\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Running the first few through vw with audit on:\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.000000\r\n",
        "\t ^aaa:24503:1:0.182356:0.329447:0.169864:0.182029:0.108256:0.2706:0.110772:0.1246:0.119324:0.179318:0.131611:0.185727:0.119479:0.129452:0.27818:0.138858:0.127822:0.40772:0.220733:0.199408 total of 1 features.\r\n",
        "0.000000\r\n",
        "\t ^aardvark:294333:1:0.129039:0.122834:0.119256:0.191947:0.112045:0.233309:0.113939:0.169658:0.121593:0.236195:0.24682:0.427933:0.110589:0.296139:0.141129:0.214868:0.169591:0.195062:0.152851:0.18028 total of 1 features.\r\n",
        "0.000000\r\n",
        "\t ^aaron:748696:1:0.141309:0.123212:0.146817:0.19391:0.123514:0.264397:0.299233:0.21692:0.255138:0.262913:0.199882:0.183131:0.109128:0.365715:0.180723:0.21685:0.43045:0.283646:0.182521:0.50269 total of 1 features.\r\n",
        "0.000000\r\n",
        "\t ^aau:109285:1:0.11168:0.260085:0.142467:0.289339:0.171577:0.281379:0.155165:0.16833:0.163008:0.495602:0.19089:0.253979:0.114711:0.11282:0.228666:0.114903:0.117536:0.293731:0.174034:0.294063 total of 1 features.\r\n",
        "0.000000\r\n",
        "\t ^abandon:631650:1:0.200887:0.128976:0.194837:0.370842:0.291929:0.160005:0.152399:0.306269:0.23724:0.175018:0.39876:0.111871:0.327165:0.470243:0.137895:0.265763:0.113316:0.109048:0.111025:0.356025 total of 1 features.\r\n",
        "0.000000\r\n",
        "\t ^abandoned:916826:1:0.319634:0.164488:0.626414:0.421522:0.127739:0.261759:0.146873:0.127785:0.524455:0.178173:0.323854:0.130943:0.470056:0.251227:0.166257:0.119897:0.372174:0.144705:0.246781:0.157225 total of 1 features.\r\n",
        "0.000000\r\n",
        "\t ^abbreviation:139971:1:0.197607:0.160489:0.349651:0.196141:0.266556:0.108351:0.189101:0.231393:0.228699:0.126509:0.357983:0.151137:0.147059:0.207933:0.158753:0.110079:0.486701:0.200756:0.351235:0.154029 total of 1 features.\r\n",
        "0.000000\r\n",
        "\t ^abc:889850:1:0.339874:0.169692:0.237028:0.154603:0.283697:0.181479:0.128209:0.119643:0.112896:0.436227:0.468355:0.534854:0.179209:0.278722:0.25511:0.261242:0.115069:0.262839:0.10839:0.137095 total of 1 features.\r\n",
        "0.000000\r\n",
        "\t ^aber:413121:1:0.213243:0.113454:0.172553:0.148069:0.180906:0.259975:0.258192:0.197191:0.276445:0.243005:0.126313:0.195293:0.163656:0.312354:0.185354:0.155635:0.161777:0.13528:0.112745:0.3145 total of 1 features.\r\n"
       ]
      }
     ],
     "prompt_number": 40
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Here, we can see the basic format for audit. First you get an overall prediction (for LDA this will also be 0.0). Then you get the word, followed by it's hash (eg 24503 for \"aaa\"), then it's count (the count in the vocabular file is always 1) and then it's unnormalized probability for each of the five topics. We can extract the relevant information by some shell scripting:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!vw -t -d data/20ng.unlab.vocab -i data/20ng.unlab.lda --audit --quiet 2>&1 | grep -v '^0' | cut -d' ' -f2 | tr ':' '\\t' | cut -f1,4- > data/20ng.unlab.topics\n",
      "!head data/20ng.unlab.topics"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "^aaa\t0.182356\t0.329447\t0.169864\t0.182029\t0.108256\t0.2706\t0.110772\t0.1246\t0.119324\t0.179318\t0.131611\t0.185727\t0.119479\t0.129452\t0.27818\t0.138858\t0.127822\t0.40772\t0.220733\t0.199408\r\n",
        "^aardvark\t0.129039\t0.122834\t0.119256\t0.191947\t0.112045\t0.233309\t0.113939\t0.169658\t0.121593\t0.236195\t0.24682\t0.427933\t0.110589\t0.296139\t0.141129\t0.214868\t0.169591\t0.195062\t0.152851\t0.18028\r\n",
        "^aaron\t0.141309\t0.123212\t0.146817\t0.19391\t0.123514\t0.264397\t0.299233\t0.21692\t0.255138\t0.262913\t0.199882\t0.183131\t0.109128\t0.365715\t0.180723\t0.21685\t0.43045\t0.283646\t0.182521\t0.50269\r\n",
        "^aau\t0.11168\t0.260085\t0.142467\t0.289339\t0.171577\t0.281379\t0.155165\t0.16833\t0.163008\t0.495602\t0.19089\t0.253979\t0.114711\t0.11282\t0.228666\t0.114903\t0.117536\t0.293731\t0.174034\t0.294063\r\n",
        "^abandon\t0.200887\t0.128976\t0.194837\t0.370842\t0.291929\t0.160005\t0.152399\t0.306269\t0.23724\t0.175018\t0.39876\t0.111871\t0.327165\t0.470243\t0.137895\t0.265763\t0.113316\t0.109048\t0.111025\t0.356025\r\n",
        "^abandoned\t0.319634\t0.164488\t0.626414\t0.421522\t0.127739\t0.261759\t0.146873\t0.127785\t0.524455\t0.178173\t0.323854\t0.130943\t0.470056\t0.251227\t0.166257\t0.119897\t0.372174\t0.144705\t0.246781\t0.157225\r\n",
        "^abbreviation\t0.197607\t0.160489\t0.349651\t0.196141\t0.266556\t0.108351\t0.189101\t0.231393\t0.228699\t0.126509\t0.357983\t0.151137\t0.147059\t0.207933\t0.158753\t0.110079\t0.486701\t0.200756\t0.351235\t0.154029\r\n",
        "^abc\t0.339874\t0.169692\t0.237028\t0.154603\t0.283697\t0.181479\t0.128209\t0.119643\t0.112896\t0.436227\t0.468355\t0.534854\t0.179209\t0.278722\t0.25511\t0.261242\t0.115069\t0.262839\t0.10839\t0.137095\r\n",
        "^aber\t0.213243\t0.113454\t0.172553\t0.148069\t0.180906\t0.259975\t0.258192\t0.197191\t0.276445\t0.243005\t0.126313\t0.195293\t0.163656\t0.312354\t0.185354\t0.155635\t0.161777\t0.13528\t0.112745\t0.3145\r\n",
        "^abiding\t0.19583\t0.126143\t0.13001\t0.117153\t0.372594\t0.174991\t0.201046\t0.210155\t0.141046\t0.154915\t0.112312\t0.277414\t0.167284\t0.183201\t0.122946\t0.123565\t0.20932\t0.277415\t0.2191\t0.143324\r\n"
       ]
      }
     ],
     "prompt_number": 41
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This now gives us something to work with. Let's look at the top terms for each of the topics."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%%bash --err err\n",
      "for n in `seq 1 20` ; do\n",
      "    let col=$n+1\n",
      "    echo \"top words from topic $n (from column $col)\"\n",
      "    cat data/20ng.unlab.topics | cut -f1,$col | sort -k2,2gr | head\n",
      "    echo \"\"\n",
      "done"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "top words from topic 1 (from column 2)\n",
        "^permission\t1.3347\n",
        "^corinthians\t1.18478\n",
        "^knives\t1.18478\n",
        "^worship\t0.997135\n",
        "^seize\t0.975493\n",
        "^nanosecond\t0.916404\n",
        "^voices\t0.897943\n",
        "^acker\t0.891526\n",
        "^represents\t0.884127\n",
        "^democracy\t0.859931\n",
        "\n",
        "top words from topic 2 (from column 3)\n",
        "^invited\t1.11815\n",
        "^dubious\t1.00535\n",
        "^hudson\t0.987452\n",
        "^methodology\t0.962736\n",
        "^med\t0.960603\n",
        "^full\t0.956764\n",
        "^mosaic\t0.917933\n",
        "^explicit\t0.9074\n",
        "^ibm\t0.904663\n",
        "^compact\t0.885661\n",
        "\n",
        "top words from topic 3 (from column 4)\n",
        "^brute\t1.07875\n",
        "^hansen\t0.999382\n",
        "^lengthy\t0.986911\n",
        "^launcher\t0.972936\n",
        "^lafibm\t0.93067\n",
        "^restart\t0.913159\n",
        "^binah\t0.909636\n",
        "^girl\t0.881105\n",
        "^billboard\t0.878396\n",
        "^grenade\t0.851245\n",
        "\n",
        "top words from topic 4 (from column 5)\n",
        "^punishment\t1.32391\n",
        "^informatik\t1.31411\n",
        "^interference\t1.24872\n",
        "^clearer\t1.00772\n",
        "^goaltenders\t0.955884\n",
        "^bits\t0.937808\n",
        "^permits\t0.896536\n",
        "^mccall\t0.869107\n",
        "^lemon\t0.867107\n",
        "^fluke\t0.841388\n",
        "\n",
        "top words from topic 5 (from column 6)\n",
        "^nissan\t1.10143\n",
        "^source\t1.0535\n",
        "^racism\t1.01605\n",
        "^nmsu\t1.01441\n",
        "^gif\t1.01389\n",
        "^vocal\t0.980711\n",
        "^cryptology\t0.948117\n",
        "^neglected\t0.948117\n",
        "^operating\t0.884318\n",
        "^pose\t0.854931\n",
        "\n",
        "top words from topic 6 (from column 7)\n",
        "^mature\t1.15052\n",
        "^council\t1.11209\n",
        "^haul\t1.09126\n",
        "^increases\t1.01843\n",
        "^secretary\t1.00604\n",
        "^smoke\t0.954511\n",
        "^gifs\t0.938437\n",
        "^fuer\t0.913386\n",
        "^rational\t0.889523\n",
        "^optimal\t0.881105\n",
        "\n",
        "top words from topic 7 (from column 8)\n",
        "^clever\t1.12456\n",
        "^shell\t1.12456\n",
        "^overnight\t1.08892\n",
        "^loosing\t1.05242\n",
        "^turks\t1.02999\n",
        "^authenticity\t0.972306\n",
        "^attraction\t0.960076\n",
        "^approximation\t0.922112\n",
        "^talon\t0.917178\n",
        "^ends\t0.911875\n",
        "\n",
        "top words from topic 8 (from column 9)\n",
        "^defining\t1.02342\n",
        "^humanity\t0.981563\n",
        "^grown\t0.965316\n",
        "^expert\t0.902774\n",
        "^troy\t0.869741\n",
        "^hypothesis\t0.869091\n",
        "^belongs\t0.861842\n",
        "^newman\t0.861085\n",
        "^player\t0.853319\n",
        "^iscp\t0.848127\n",
        "\n",
        "top words from topic 9 (from column 10)\n",
        "^bps\t1.11052\n",
        "^expects\t1.10091\n",
        "^naming\t1.02374\n",
        "^blackhawks\t0.994692\n",
        "^dunno\t0.961593\n",
        "^cryptographically\t0.920888\n",
        "^organizers\t0.908545\n",
        "^yankee\t0.901851\n",
        "^gripe\t0.894153\n",
        "^outrageous\t0.845494\n",
        "\n",
        "top words from topic 10 (from column 11)\n",
        "^past\t1.25183\n",
        "^weighed\t1.04362\n",
        "^texts\t1.02087\n",
        "^constellation\t0.979739\n",
        "^playing\t0.962413\n",
        "^johns\t0.960041\n",
        "^illegal\t0.956764\n",
        "^bios\t0.951493\n",
        "^nth\t0.932608\n",
        "^greatly\t0.927932\n",
        "\n",
        "top words from topic 11 (from column 12)\n",
        "^severely\t1.21558\n",
        "^indian\t1.00876\n",
        "^cancer\t0.988724\n",
        "^utkvx\t0.976508\n",
        "^radiator\t0.942288\n",
        "^noone\t0.940316\n",
        "^smb\t0.930803\n",
        "^backs\t0.917295\n",
        "^shaky\t0.887557\n",
        "^clinical\t0.882631\n",
        "\n",
        "top words from topic 12 (from column 13)\n",
        "^jebright\t1.23132\n",
        "^gifts\t1.19482\n",
        "^genes\t1.16153\n",
        "^lucky\t1.08294\n",
        "^sprite\t0.983331\n",
        "^stone\t0.959971\n",
        "^sincerely\t0.956019\n",
        "^ida\t0.926768\n",
        "^leftover\t0.921376\n",
        "^misc\t0.916708\n",
        "\n",
        "top words from topic 13 (from column 14)\n",
        "^einstien\t1.10078\n",
        "^nelson\t1.08973\n",
        "^description\t1.07261\n",
        "^focus\t0.98535\n",
        "^unnecessarily\t0.977988\n",
        "^handgun\t0.974567\n",
        "^evolve\t0.97357\n",
        "^needless\t0.953881\n",
        "^combining\t0.946622\n",
        "^semitism\t0.946622\n",
        "\n",
        "top words from topic 14 (from column 15)\n",
        "^apartment\t1.20185\n",
        "^brooks\t1.04593\n",
        "^transmission\t1.03086\n",
        "^digress\t1.02198\n",
        "^pressure\t0.984336\n",
        "^impress\t0.978943\n",
        "^berg\t0.977286\n",
        "^consequences\t0.954511\n",
        "^sucked\t0.940054\n",
        "^expressions\t0.928688\n",
        "\n",
        "top words from topic 15 (from column 16)\n",
        "^louray\t1.27392\n",
        "^diverted\t1.0006\n",
        "^beginner\t0.949887\n",
        "^nosc\t0.908155\n",
        "^foods\t0.905631\n",
        "^castle\t0.857253\n",
        "^attach\t0.838165\n",
        "^comprehensive\t0.83462\n",
        "^readable\t0.824822\n",
        "^bos\t0.820325\n",
        "\n",
        "top words from topic 16 (from column 17)\n",
        "^deepak\t1.10663\n",
        "^phils\t1.06101\n",
        "^lose\t1.05792\n",
        "^mentions\t1.03952\n",
        "^camel\t1.02986\n",
        "^uga\t1.01355\n",
        "^tuba\t1.0054\n",
        "^opt\t0.968073\n",
        "^amoco\t0.952893\n",
        "^hoc\t0.939706\n",
        "\n",
        "top words from topic 17 (from column 18)\n",
        "^domino\t1.01314\n",
        "^equipped\t1.00992\n",
        "^del\t0.97266\n",
        "^recovered\t0.969326\n",
        "^win\t0.953452\n",
        "^shadows\t0.942616\n",
        "^billboards\t0.914505\n",
        "^mostly\t0.90597\n",
        "^chaos\t0.903746\n",
        "^ists\t0.89816\n",
        "\n",
        "top words from topic 18 (from column 19)\n",
        "^wdstarr\t1.15196\n",
        "^warner\t1.06726\n",
        "^wicked\t1.05827\n",
        "^saving\t1.01777\n",
        "^crucified\t1.01138\n",
        "^hackers\t0.99763\n",
        "^udel\t0.994837\n",
        "^safety\t0.948242\n",
        "^buttons\t0.908328\n",
        "^hsh\t0.903704\n",
        "\n",
        "top words from topic 19 (from column 20)\n",
        "^versions\t1.24085\n",
        "^planet\t1.11267\n",
        "^turk\t1.09499\n",
        "^murphy\t1.08549\n",
        "^gregg\t1.01623\n",
        "^hard\t1.00953\n",
        "^mozumder\t1.00178\n",
        "^unaware\t0.92718\n",
        "^differing\t0.915845\n",
        "^vehicle\t0.90695\n",
        "\n",
        "top words from topic 20 (from column 21)\n",
        "^plea\t1.13435\n",
        "^chocolate\t1.04523\n",
        "^cruiser\t1.01048\n",
        "^mirror\t1.00019\n",
        "^whitmore\t0.985972\n",
        "^reward\t0.979236\n",
        "^depend\t0.976917\n",
        "^taxation\t0.969785\n",
        "^kotb\t0.953386\n",
        "^avenue\t0.9235\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 43
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "These are not super meaningful, largely because we didn't run enough topics."
     ]
    }
   ],
   "metadata": {}
  }
 ]
}