RareCategory.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:91769ee8fccda583b214132ef11e1e99e61df0bd952f499a034720471f78cfa1"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Rare Category Detection and Related Tasks\n",
      "\n",
      "By default, `vw` optimizes (a surrogate to) zero-one loss for binary classification. This is great for \"balanced\" or approximately balanced problems, in which there are (roughly) an equal number of positive and negative examples.\n",
      "\n",
      "Some tasks, however, are more like \"needle in a haystack\" type problems, where there are very few positives (needles) to find amongst a large set of negatives (hay). For such problems, zero-one binary loss is just not a good measure to optimize. If 99% of the data is negative, then you can simply label everything as negative and get 1% error. This is not what we want.\n",
      "\n",
      "Often in NLP-land, we evaluate the performance of rare category problems using precision, recall and F. Precision is: of all the needles I detected, how many were correct? Recall is: of all the needles that I should have found, how many did I find? and F is the harmonic mean of P and R: $F = \\left[\\frac 1 P + \\frac 1 R\\right]^{-1} = \\frac {2PR} {P+R}$.\n",
      "\n",
      "The question is: how to optimize something like $F$ rather than binary loss. The current solution is example weighting.\n",
      "\n",
      "# <a id=\"weighting\"></a> Example Weighting\n",
      "\n",
      "In example weighting, we simply say that some examples (the needles) are much more important than other examples (the hay).\n",
      "\n",
      "As a prototypical example, suppose that the positive class accounts for about 1% of the data, so that for every one positive example there are 99 negative examples. A standard heuristic is to give the negative examples a smaller weight than the positive examples to balance this out. In this case, if we give each negative example a weight of $1/99 \\approx 0.0101$, then the total **weight** of positive examples in the training data will match the total weight of the negative examples.\n",
      "\n",
      "When `vw` is given example weights, it optimizes a weighted zero-one loss. Now, predicting always negative will incur a 50% weighted error because we've downweighted the negatives to account for only half the mass of the training data.\n",
      "\n",
      "# <a id=\"example\"></a> A Running Example\n",
      "\n",
      "The sentiment data set from the previous tutorials is not a good place to start because it was constructed to be balanced. We'll use a slightly different problem: identifying **word-level sentiment** in reviews from Rotten Tomatoes. The data we'll use is the [Stanford Sentiment Dataset](http://nlp.stanford.edu/sentiment/).\n",
      "\n",
      "First, we need to download the data. We can get it directly (it's about 771k):"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!curl -o data/trainDevTestTrees_PTB.zip http://nlp.stanford.edu/sentiment/trainDevTestTrees_PTB.zip\n",
      "!rm -rf data/trees\n",
      "!unzip -d data data/trainDevTestTrees_PTB.zip\n",
      "!head -n1 data/trees/train.txt"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\r\n",
        "                                 Dload  Upload   Total   Spent    Left  Speed\r\n",
        "\r",
        "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        " 55  771k   55  428k    0     0   641k      0  0:00:01 --:--:--  0:00:01  641k"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        "100  771k  100  771k    0     0   949k      0 --:--:-- --:--:-- --:--:--  949k\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Archive:  data/trainDevTestTrees_PTB.zip\r\n",
        "   creating: data/trees/\r\n",
        "  inflating: data/trees/dev.txt      \r\n",
        "  inflating: data/trees/test.txt     \r\n",
        "  inflating: data/trees/train.txt    \r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "(3 (2 (2 The) (2 Rock)) (4 (3 (2 is) (4 (2 destined) (2 (2 (2 (2 (2 to) (2 (2 be) (2 (2 the) (2 (2 21st) (2 (2 (2 Century) (2 's)) (2 (3 new) (2 (2 ``) (2 Conan)))))))) (2 '')) (2 and)) (3 (2 that) (3 (2 he) (3 (2 's) (3 (2 going) (3 (2 to) (4 (3 (2 make) (3 (3 (2 a) (3 splash)) (2 (2 even) (3 greater)))) (2 (2 than) (2 (2 (2 (2 (1 (2 Arnold) (2 Schwarzenegger)) (2 ,)) (2 (2 Jean-Claud) (2 (2 Van) (2 Damme)))) (2 or)) (2 (2 Steven) (2 Segal))))))))))))) (2 .)))\r\n"
       ]
      }
     ],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This data is in Penn Treebank format. Each word has a sentiment label (0=strongly negative, 1=negative, 2=neutral, 3=positive, 4=strongly positive). Each phrase also has a sentiment label. We're going to ignore the phrase-level sentiment labels and just work with the words. To make this a binary problem, we're going to predict: strong-opinion-versus-not, namely \"label=1,2,3\" is going to be the negative class (neutral) and \"label=0,4\" are the positive classes (strongly opinionated). The training data is roughly 97.5% negative examples and 2.5% positive examples, which makes this a somewhat-rare category detection problem.\n",
      "\n",
      "First, we'll write some small functions to load this data into Python:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import re\n",
      "def readSentimentTree(s):\n",
      "    # return a list of (word, sentimentLabel) for all words in the tree\n",
      "    return [(w[3:-1].lower(), int(w[1])) for w in re.findall('\\([0-4] [^\\(\\)]+\\)', s)]\n",
      "\n",
      "def readSentimentFile(filename):\n",
      "    return map(readSentimentTree, open(filename,'r').readlines())\n",
      "\n",
      "train = readSentimentFile('data/trees/train.txt')\n",
      "dev   = readSentimentFile('data/trees/dev.txt')\n",
      "test  = readSentimentFile('data/trees/test.txt')\n",
      "\n",
      "print 'example: ', train[0]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "example:  [('the', 2), ('rock', 2), ('is', 2), ('destined', 2), ('to', 2), ('be', 2), ('the', 2), ('21st', 2), ('century', 2), (\"'s\", 2), ('new', 3), ('``', 2), ('conan', 2), (\"''\", 2), ('and', 2), ('that', 2), ('he', 2), (\"'s\", 2), ('going', 2), ('to', 2), ('make', 2), ('a', 2), ('splash', 3), ('even', 2), ('greater', 3), ('than', 2), ('arnold', 2), ('schwarzenegger', 2), (',', 2), ('jean-claud', 2), ('van', 2), ('damme', 2), ('or', 2), ('steven', 2), ('segal', 2), ('.', 2)]\n"
       ]
      }
     ],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We now need to generate appropriate `vw` input files for this. The way that you specify example weights for `vw` is to put the weight right next to the label, but before the pipe. For example:\n",
      "\n",
      "    +1 | ...\n",
      "    +1 | ...\n",
      "    -1 0.01 | ...\n",
      "    +1 | ...\n",
      "    -1 0.01 | ...\n",
      "\n",
      "And so on. In this case, the two negative examples have a weight of 0.01 and the positive examples have an implicit weight of one. You can put whatever weight you want on whatever examples you want: they don't have to be consistent in any way. Examples with higher weight just \"matter more\" from the perspective of the loss being optimized.\n",
      "\n",
      "The other question is what features will we use? We'll make significant use of namespaces here. Obviously we'll include the word to be labeled; this will go in it's own namespace (called `w`). We'll also use a context window of 5 words to the left (into a `l` namespace) and 5 words to the right (into an `r`) namespace.\n",
      "\n",
      "The label itself will be -1 if sentLabel is 1,2 or 3; and +1 otherwise.\n",
      "\n",
      "Given this, we're ready to construct some data to be directly written to a file:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def sanify(s): return s.replace(':','COLON').replace('|','PIPE')\n",
      "\n",
      "def sentimentToVW(sentence, negativeWeight, outFile):\n",
      "    N = len(sentence)\n",
      "    for n,(word,sentLabel) in enumerate(sentence):   # loop over each word, and get it's position n\n",
      "        label = '-1 %g' % negativeWeight if sentLabel in [1,2,3] else '+1'\n",
      "        leftBoundary = max(0,n-5)\n",
      "        leftContext  = ' '.join([sanify(x[0]) for x in sentence[leftBoundary:n]])\n",
      "        rightContext = ' '.join([sanify(x[0]) for x in sentence[n+1:n+6]])\n",
      "        print >>outFile, '%s |w %s |l %s |r %s' % (label, sanify(word), leftContext, rightContext)\n",
      "\n",
      "# test it\n",
      "import sys\n",
      "sentimentToVW(train[0], 2.5/97.5, sys.stdout)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "-1 0.025641 |w the |l  |r rock is destined to be\n",
        "-1 0.025641 |w rock |l the |r is destined to be the\n",
        "-1 0.025641 |w is |l the rock |r destined to be the 21st\n",
        "-1 0.025641 |w destined |l the rock is |r to be the 21st century\n",
        "-1 0.025641 |w to |l the rock is destined |r be the 21st century 's\n",
        "-1 0.025641 |w be |l the rock is destined to |r the 21st century 's new\n",
        "-1 0.025641 |w the |l rock is destined to be |r 21st century 's new ``\n",
        "-1 0.025641 |w 21st |l is destined to be the |r century 's new `` conan\n",
        "-1 0.025641 |w century |l destined to be the 21st |r 's new `` conan ''\n",
        "-1 0.025641 |w 's |l to be the 21st century |r new `` conan '' and\n",
        "-1 0.025641 |w new |l be the 21st century 's |r `` conan '' and that\n",
        "-1 0.025641 |w `` |l the 21st century 's new |r conan '' and that he\n",
        "-1 0.025641 |w conan |l 21st century 's new `` |r '' and that he 's\n",
        "-1 0.025641 |w '' |l century 's new `` conan |r and that he 's going\n",
        "-1 0.025641 |w and |l 's new `` conan '' |r that he 's going to\n",
        "-1 0.025641 |w that |l new `` conan '' and |r he 's going to make\n",
        "-1 0.025641 |w he |l `` conan '' and that |r 's going to make a\n",
        "-1 0.025641 |w 's |l conan '' and that he |r going to make a splash\n",
        "-1 0.025641 |w going |l '' and that he 's |r to make a splash even\n",
        "-1 0.025641 |w to |l and that he 's going |r make a splash even greater\n",
        "-1 0.025641 |w make |l that he 's going to |r a splash even greater than\n",
        "-1 0.025641 |w a |l he 's going to make |r splash even greater than arnold\n",
        "-1 0.025641 |w splash |l 's going to make a |r even greater than arnold schwarzenegger\n",
        "-1 0.025641 |w even |l going to make a splash |r greater than arnold schwarzenegger ,\n",
        "-1 0.025641 |w greater |l to make a splash even |r than arnold schwarzenegger , jean-claud\n",
        "-1 0.025641 |w than |l make a splash even greater |r arnold schwarzenegger , jean-claud van\n",
        "-1 0.025641 |w arnold |l a splash even greater than |r schwarzenegger , jean-claud van damme\n",
        "-1 0.025641 |w schwarzenegger |l splash even greater than arnold |r , jean-claud van damme or\n",
        "-1 0.025641 |w , |l even greater than arnold schwarzenegger |r jean-claud van damme or steven\n",
        "-1 0.025641 |w jean-claud |l greater than arnold schwarzenegger , |r van damme or steven segal\n",
        "-1 0.025641 |w van |l than arnold schwarzenegger , jean-claud |r damme or steven segal .\n",
        "-1 0.025641 |w damme |l arnold schwarzenegger , jean-claud van |r or steven segal .\n",
        "-1 0.025641 |w or |l schwarzenegger , jean-claud van damme |r steven segal .\n",
        "-1 0.025641 |w steven |l , jean-claud van damme or |r segal .\n",
        "-1 0.025641 |w segal |l jean-claud van damme or steven |r .\n",
        "-1 0.025641 |w . |l van damme or steven segal |r \n"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we just need to construct training, development and test files.\n",
      "\n",
      "What we're *actually* going to do is put the training and development data together into one file, and use `--holdout_after` to let `vw` handle the development data. Of course, first we also need to shuffle the data. (Note: here, we're just going to shuffle the sentences; in real life, we'd probably want to actually shuffle the VW examples. But we're lazy.)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import random\n",
      "random.shuffle(train)\n",
      "random.shuffle(dev)\n",
      "random.shuffle(test)\n",
      "\n",
      "def sentimentToVWFile(filename, data, negativeWeight):\n",
      "    with open(filename, 'w') as h:\n",
      "        for sentence in data:\n",
      "            sentimentToVW(sentence, negativeWeight, h)\n",
      "\n",
      "# remove old data\n",
      "!rm -f data/sentiword.*\n",
      "\n",
      "# generate new data\n",
      "negWeight = 2.5/97.5\n",
      "sentimentToVWFile('data/sentiword.tr', train, negWeight)\n",
      "sentimentToVWFile('data/sentiword.de', dev  , negWeight)\n",
      "sentimentToVWFile('data/sentiword.te', test , negWeight)\n",
      "\n",
      "# combine train and dev into one\n",
      "!cat data/sentiword.tr data/sentiword.de > data/sentiword.trde\n",
      "\n",
      "!wc -l data/sentiword.*"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "   21274 data/sentiword.de\r\n",
        "   42405 data/sentiword.te\r\n",
        "  163563 data/sentiword.tr\r\n",
        "  184837 data/sentiword.trde\r\n",
        "  412079 total\r\n"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "So, we have about 163563 training examples, 21k dev examples and 42k test examples. The combined train/dev set has about 185k examples.\n",
      "\n",
      "Now we just need to train `vw`!"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!vw -k -c -b 27 --binary data/sentiword.trde --passes 100 -f data/sentiword.model --loss_function logistic --holdout_after 163564"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "final_regressor = data/sentiword.model\r\n",
        "Num weight bits = 27\r\n",
        "learning rate = 0.5\r\n",
        "initial_t = 0\r\n",
        "power_t = 0.5\r\n",
        "decay_learning_rate = 1\r\n",
        "creating cache_file = data/sentiword.trde.cache\r\n",
        "Reading datafile = data/sentiword.trde\r\n",
        "num sources = 1\r\n",
        "average  since         example        example  current  current  current\r\n",
        "loss     last          counter         weight    label  predict features\r\n",
        "0.000000 0.000000           40            1.0  -1.0000  -1.0000       12\r\n",
        "0.000000 0.000000           81            2.1  -1.0000  -1.0000        8\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.240741 0.481482          124            4.2  -1.0000  -1.0000       12\r\n",
        "0.361111 0.481482          210            8.3  -1.0000  -1.0000        7\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.283843 0.214876          497           17.6   1.0000  -1.0000        9\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.369724 0.455604          880           35.2  -1.0000  -1.0000       12\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.404986 0.439742         1552           71.0   1.0000  -1.0000       12\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.396135 0.387283         2952          141.9  -1.0000   1.0000       11\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.392593 0.389069         5778          284.6   1.0000   1.0000       11\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.319142 0.245698        11519          569.2  -1.0000  -1.0000       12\r\n",
        "0.286698 0.254297        23485         1139.1   1.0000  -1.0000       12\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.242487 0.198276        46818         2278.1  -1.0000  -1.0000       12\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.195484 0.148482        94396         4556.2  -1.0000  -1.0000        8\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.101881 0.101881       188108         9112.4  -1.0000  -1.0000       12 h\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.091317 0.080753       376097        18225.7   1.0000   1.0000        8 h\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.082237 0.073156       752536        36451.3  -1.0000  -1.0000       12 h\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.074883 0.069000      1504525        72903.2   1.0000   1.0000        8 h\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r\n",
        "finished run\r\n",
        "number of examples per pass = 163563\r\n",
        "passes used = 12\r\n",
        "weighted example sum = 95096.873733\r\n",
        "weighted label sum = -3200.873733\r\n",
        "average loss = 0.066373 h\r\n",
        "best constant = -0.067344\r\n",
        "best constant's loss = 0.692581\r\n",
        "total feature number = 20493552\r\n"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# <a id=\"eval\"></a> Making Predictions and Evaluating Performance\n",
      "\n",
      "And now we will make raw predictions (we need raw predictions because of how we will evaluate performance). This is exactly the same as before. However, because we're going to be choosing weights as hyperparameters, we are going to do all evaluations on the development data."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!vw --binary -t -r data/sentiword.de.raw -i data/sentiword.model data/sentiword.de --quiet\n",
      "!head data/sentiword.de.raw"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "-4.507044\r\n",
        "-0.372609\r\n",
        "-1.805572\r\n",
        "-3.560715\r\n",
        "-2.972163\r\n",
        "-1.174249\r\n",
        "-0.556056\r\n",
        "-0.684190\r\n",
        "-2.711704\r\n",
        "-1.565568\r\n"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In order to evaluate the model, we'll use use the [\"perf\" evaluation script](http://osmot.cs.cornell.edu/kddcup/software.html) from the KDD 2004 challenge. This script computes basically every measure of performance you could possibly want. In order to follow along here, you'll need to download it, build it, and put it in your path. You can test that it works with:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!perf --help | head -n4"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r\n",
        "Error: Unrecognized program option --help\r\n",
        "Version 5.12 [KDDCUP-2004 July 12, 2004]\r\n",
        "\r\n"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The eval script needs a single input that has two columns: (1) the truth and (2) scored predictions. It needs scores because it needs to think of the predictions as a ranked list. We can get the true labels by extracting the first column from the test data, and then the raw predictions work directly as scored predictions:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!echo \"The first few lines...\"\n",
      "!cut -d' ' -f1 data/sentiword.de | paste - data/sentiword.de.raw | head\n",
      "!echo \"\"\n",
      "!echo \"Running perf...\"\n",
      "!cut -d' ' -f1 data/sentiword.de | paste - data/sentiword.de.raw | perf -t 0 -PRE -REC -PRF -PRB"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "The first few lines...\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "-1\t-4.507044\r\n",
        "-1\t-0.372609\r\n",
        "-1\t-1.805572\r\n",
        "-1\t-3.560715\r\n",
        "-1\t-2.972163\r\n",
        "-1\t-1.174249\r\n",
        "-1\t-0.556056\r\n",
        "-1\t-0.684190\r\n",
        "-1\t-2.711704\r\n",
        "-1\t-1.565568\r\n",
        "paste: write error: Broken pipe\r\n",
        "paste: write error\r\n",
        "cut: write error: Broken pipe\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Running perf...\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "PRE    0.66527   pred_thresh  0.000000\r\n",
        "REC    0.87963   pred_thresh  0.000000\r\n",
        "PRF    0.75758   pred_thresh  0.000000\r\n",
        "PRB    0.82593\r\n"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We asked `perf` just to give us precision, recall and F score with a threshold of zero. If you run it without those arguments, you get a *lot* more statistics.\n",
      "\n",
      "In this case, we get a precision of 76.3% a recall of 86.9% and an F score of 81.2%. We've also asked for the precision-recall break-even point (that's the point where P=R=F); here it is 84.6%. (PRB is often an upper-bound, optimistic estimate of how good your precision/recall could be if you magically chose the best threshold.)\n",
      "\n",
      "For comparison, let's see what happens if we don't do example weighting:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "negWeight = 1.0\n",
      "sentimentToVWFile('data/sentiword-unw.tr', train, negWeight)\n",
      "sentimentToVWFile('data/sentiword-unw.de', dev  , negWeight)\n",
      "sentimentToVWFile('data/sentiword-unw.te', test , negWeight)\n",
      "\n",
      "# combine train and dev into one\n",
      "!cat data/sentiword-unw.tr data/sentiword-unw.de > data/sentiword-unw.trde\n",
      "!vw -k -c -b 27 --binary data/sentiword-unw.trde --passes 100 -f data/sentiword-unw.model --loss_function logistic --holdout_after 163564\n",
      "!vw --binary -t -r data/sentiword-unw.de.raw -i data/sentiword-unw.model data/sentiword-unw.de --quiet\n",
      "!echo \"\"\n",
      "!echo \"Running perf...\"\n",
      "!cut -d' ' -f1 data/sentiword-unw.de | paste - data/sentiword-unw.de.raw | perf -t 0.0 -PRE -REC -PRF -PRB"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "final_regressor = data/sentiword-unw.model\r\n",
        "Num weight bits = 27\r\n",
        "learning rate = 0.5\r\n",
        "initial_t = 0\r\n",
        "power_t = 0.5\r\n",
        "decay_learning_rate = 1\r\n",
        "creating cache_file = data/sentiword-unw.trde.cache\r\n",
        "Reading datafile = data/sentiword-unw.trde\r\n",
        "num sources = 1\r\n",
        "average  since         example        example  current  current  current\r\n",
        "loss     last          counter         weight    label  predict features\r\n",
        "0.000000 0.000000            1            1.0  -1.0000  -1.0000        7\r\n",
        "0.000000 0.000000            2            2.0  -1.0000  -1.0000        8\r\n",
        "0.000000 0.000000            4            4.0  -1.0000  -1.0000       10\r\n",
        "0.000000 0.000000            8            8.0  -1.0000  -1.0000       12\r\n",
        "0.000000 0.000000           16           16.0  -1.0000  -1.0000       12\r\n",
        "0.000000 0.000000           32           32.0  -1.0000  -1.0000       11\r\n",
        "0.000000 0.000000           64           64.0  -1.0000  -1.0000        7\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.007812 0.015625          128          128.0  -1.0000  -1.0000       12\r\n",
        "0.011719 0.015625          256          256.0  -1.0000  -1.0000       11\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.009766 0.007812          512          512.0  -1.0000  -1.0000       10\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.015625 0.021484         1024         1024.0  -1.0000  -1.0000       12\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.020996 0.026367         2048         2048.0  -1.0000  -1.0000        8\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.022705 0.024414         4096         4096.0   1.0000  -1.0000        7\r\n",
        "0.025146 0.027588         8192         8192.0  -1.0000  -1.0000       12\r\n",
        "0.024170 0.023193        16384        16384.0  -1.0000  -1.0000       12\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.023590 0.023010        32768        32768.0  -1.0000  -1.0000       12\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.022614 0.021637        65536        65536.0  -1.0000  -1.0000       12\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.021667 0.020721       131072       131072.0  -1.0000  -1.0000       12\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.020542 0.020542       262144       262144.0  -1.0000  -1.0000       12 h\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.017611 0.016146       524288       524288.0  -1.0000  -1.0000       12 h\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.015473 0.013334      1048576      1048576.0  -1.0000  -1.0000        7 h\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.013506 0.011540      2097152      2097152.0  -1.0000  -1.0000       12 h\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.011819 0.010262      4194304      4194304.0  -1.0000  -1.0000        9 h\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r\n",
        "finished run\r\n",
        "number of examples per pass = 163563\r\n",
        "passes used = 36\r\n",
        "weighted example sum = 5888268.000000\r\n",
        "weighted label sum = -5612580.000000\r\n",
        "average loss = 0.009166 h\r\n",
        "best constant = -3.730906\r\n",
        "best constant's loss = 0.111029\r\n",
        "total feature number = 61480656\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Running perf...\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "PRE    0.97784   pred_thresh  0.000000\r\n",
        "REC    0.65370   pred_thresh  0.000000\r\n",
        "PRF    0.78357   pred_thresh  0.000000\r\n",
        "PRB    0.81667\r\n"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "As expected, this is worse than before. The F score is 75.7% (< 81.2%) and the break-even point is 79.8% (< 84.6%). The gap is definitely enough to care about.\n",
      "\n",
      "# <a id=\"opt\"></a> Optimal Example Weights?\n",
      "\n",
      "A question that immediately comes up is: what is the *best* weight to give the positive and negative examples? We used a heuristic setting above, but is this optimal?\n",
      "\n",
      "Unfortunately, we don't know. What we [do know](http://papers.nips.cc/paper/5454-consistent-binary-classification-with-generalized-performance-metrics) is that if the end performance measure is F score, then there *exists* some weighting that is guaranteed to optimize this, but we don't *constructively* know what it is. The weighting then becomes a new hyperparameter that we can tune to get the best possible F.\n",
      "\n",
      "This is somewhat cumbersome, but we can try a few different values as follows. We'll switch to hinge loss too."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "baseRatio = 2.5/97.5\n",
      "for multiplier in [2.0**k for k in range(-5,6)]:\n",
      "    negWeight = baseRatio * multiplier\n",
      "    sentimentToVWFile('data/sentiword.tr', train, negWeight)\n",
      "    sentimentToVWFile('data/sentiword.de', dev  , negWeight)\n",
      "    sentimentToVWFile('data/sentiword.te', test , negWeight)\n",
      "\n",
      "    # combine train and dev into one\n",
      "    print 'weight = %g * %g = %g' % (baseRatio, multiplier, negWeight)\n",
      "    !cat data/sentiword.tr data/sentiword.de > data/sentiword.trde\n",
      "    !vw -k -c -b 27 --binary data/sentiword.trde --passes 100 -f data/sentiword.model --loss_function logistic --quiet --holdout_after 163564\n",
      "    !vw --binary -t -r data/sentiword.de.raw -i data/sentiword.model data/sentiword.de --quiet\n",
      "    !cut -d' ' -f1 data/sentiword.de | paste - data/sentiword.de.raw | perf -PRF -t 0\n",
      "    print ''"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "weight = 0.025641 * 0.03125 = 0.000801282\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "PRF    0.09226   pred_thresh  0.000000\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "weight = 0.025641 * 0.0625 = 0.00160256"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "PRF    0.14089   pred_thresh  0.000000\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "weight = 0.025641 * 0.125 = 0.00320513"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "PRF    0.16000   pred_thresh  0.000000\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "weight = 0.025641 * 0.25 = 0.00641026"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "PRF    0.33624   pred_thresh  0.000000\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "weight = 0.025641 * 0.5 = 0.0128205"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "PRF    0.72214   pred_thresh  0.000000\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "weight = 0.025641 * 1 = 0.025641"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "PRF    0.75758   pred_thresh  0.000000\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "weight = 0.025641 * 2 = 0.0512821"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "PRF    0.86267   pred_thresh  0.000000\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "weight = 0.025641 * 4 = 0.102564"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "PRF    0.85249   pred_thresh  0.000000\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "weight = 0.025641 * 8 = 0.205128"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "PRF    0.83633   pred_thresh  0.000000\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "weight = 0.025641 * 16 = 0.410256"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "PRF    0.81553   pred_thresh  0.000000\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "weight = 0.025641 * 32 = 0.820513"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "PRF    0.79473   pred_thresh  0.000000\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "From this, we can see a fairly wide range of performance. The optimal performance here is with a weight of around 0.05, yielding an F score of 86.3%.\n",
      "\n",
      "# <a id=\"pred\"></a> Making Final Predictions\n",
      "\n",
      "Now that we know a good negative example weighting (0.0512821), we're going to train a final model and predict and evaluate on test data:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "negWeight = 0.0512821\n",
      "sentimentToVWFile('data/sentiword.tr', train, negWeight)\n",
      "sentimentToVWFile('data/sentiword.de', dev  , negWeight)\n",
      "sentimentToVWFile('data/sentiword.te', test , negWeight)\n",
      "\n",
      "# combine train and dev into one\n",
      "!cat data/sentiword.tr data/sentiword.de > data/sentiword.trde\n",
      "# train\n",
      "!vw -k -c -b 27 --binary data/sentiword.trde --passes 100 -f data/sentiword.model --loss_function logistic --quiet --holdout_after 163564\n",
      "# now, predict on TEST\n",
      "!vw --binary -t -r data/sentiword.te.raw -i data/sentiword.model data/sentiword.te --quiet\n",
      "!cut -d' ' -f1 data/sentiword.te | paste - data/sentiword.te.raw | perf -PRF -t 0"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "PRF    0.84909   pred_thresh  0.000000\r\n"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "And voila, we have a system that gets an F score of 84.9% on test data.\n",
      "\n",
      "How does this compare to anything else? I have no idea. This was kind of a made-up task :)."
     ]
    }
   ],
   "metadata": {}
  }
 ]
}