CLA05-NB01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Identifying safe loans with decision trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [LendingClub](https://www.lendingclub.com/) is a peer-to-peer leading company that directly connects borrowers and potential lenders/investors. In this notebook, you will build a classification model to predict whether or not a loan provided by LendingClub is likely to [default](https://en.wikipedia.org/wiki/Default_%28finance%29).\n",
    "\n",
    "In this notebook you will use data from the LendingClub to predict whether a loan will be paid off in full or the loan will be [charged off](https://en.wikipedia.org/wiki/Charge-off) and possibly go into default. In this assignment you will:\n",
    "\n",
    "* Use SFrames to do some feature engineering.\n",
    "* Train a decision-tree on the LendingClub dataset.\n",
    "* Predict whether a loan will default along with prediction probabilities (on a validation set).\n",
    "* Train a complex tree model and compare it to simple tree model.\n",
    "\n",
    "Let's get started!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fire up Turi Create"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure you have the latest version of Turi Create. If you don't find the decision tree module, then you would need to upgrade Turi Create using\n",
    "\n",
    "```\n",
    "   pip install turicreate --upgrade\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import turicreate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load LendingClub dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using a dataset from the [LendingClub](https://www.lendingclub.com/). A parsed and cleaned form of the dataset is availiable [here](https://github.com/learnml/machine-learning-specialization-private). Make sure you **download the dataset** before running the following command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "loans = turicreate.SFrame('lending-club-data.sframe/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring some features\n",
    "\n",
    "Let's quickly explore what the dataset looks like. First, let's print out the column names to see what features we have in this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['id',\n",
       " 'member_id',\n",
       " 'loan_amnt',\n",
       " 'funded_amnt',\n",
       " 'funded_amnt_inv',\n",
       " 'term',\n",
       " 'int_rate',\n",
       " 'installment',\n",
       " 'grade',\n",
       " 'sub_grade',\n",
       " 'emp_title',\n",
       " 'emp_length',\n",
       " 'home_ownership',\n",
       " 'annual_inc',\n",
       " 'is_inc_v',\n",
       " 'issue_d',\n",
       " 'loan_status',\n",
       " 'pymnt_plan',\n",
       " 'url',\n",
       " 'desc',\n",
       " 'purpose',\n",
       " 'title',\n",
       " 'zip_code',\n",
       " 'addr_state',\n",
       " 'dti',\n",
       " 'delinq_2yrs',\n",
       " 'earliest_cr_line',\n",
       " 'inq_last_6mths',\n",
       " 'mths_since_last_delinq',\n",
       " 'mths_since_last_record',\n",
       " 'open_acc',\n",
       " 'pub_rec',\n",
       " 'revol_bal',\n",
       " 'revol_util',\n",
       " 'total_acc',\n",
       " 'initial_list_status',\n",
       " 'out_prncp',\n",
       " 'out_prncp_inv',\n",
       " 'total_pymnt',\n",
       " 'total_pymnt_inv',\n",
       " 'total_rec_prncp',\n",
       " 'total_rec_int',\n",
       " 'total_rec_late_fee',\n",
       " 'recoveries',\n",
       " 'collection_recovery_fee',\n",
       " 'last_pymnt_d',\n",
       " 'last_pymnt_amnt',\n",
       " 'next_pymnt_d',\n",
       " 'last_credit_pull_d',\n",
       " 'collections_12_mths_ex_med',\n",
       " 'mths_since_last_major_derog',\n",
       " 'policy_code',\n",
       " 'not_compliant',\n",
       " 'status',\n",
       " 'inactive_loans',\n",
       " 'bad_loans',\n",
       " 'emp_length_num',\n",
       " 'grade_num',\n",
       " 'sub_grade_num',\n",
       " 'delinq_2yrs_zero',\n",
       " 'pub_rec_zero',\n",
       " 'collections_12_mths_zero',\n",
       " 'short_emp',\n",
       " 'payment_inc_ratio',\n",
       " 'final_d',\n",
       " 'last_delinq_none',\n",
       " 'last_record_none',\n",
       " 'last_major_derog_none']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loans.column_names()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we see that we have some feature columns that have to do with grade of the loan, annual income, home ownership status, etc. Let's take a look at the distribution of loan grades in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>Materializing SArray</pre>"
      ],
      "text/plain": [
       "Materializing SArray"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<html>                 <body>                     <iframe style=\"border:0;margin:0\" width=\"920\" height=\"770\" srcdoc='<html lang=\"en\">                         <head>                             <script src=\"https://cdnjs.cloudflare.com/ajax/libs/vega/5.4.0/vega.js\"></script>                             <script src=\"https://cdnjs.cloudflare.com/ajax/libs/vega-embed/4.0.0/vega-embed.js\"></script>                             <script src=\"https://cdnjs.cloudflare.com/ajax/libs/vega-tooltip/0.5.1/vega-tooltip.min.js\"></script>                             <link rel=\"stylesheet\" type=\"text/css\" href=\"https://cdnjs.cloudflare.com/ajax/libs/vega-tooltip/0.5.1/vega-tooltip.min.css\">                             <style>                             .vega-actions > a{                                 color:white;                                 text-decoration: none;                                 font-family: \"Arial\";                                 cursor:pointer;                                 padding:5px;                                 background:#AAAAAA;                                 border-radius:4px;                                 padding-left:10px;                                 padding-right:10px;                                 margin-right:5px;                             }                             .vega-actions{                                 margin-top:20px;                                 text-align:center                             }                            .vega-actions > a{                                 background:#999999;                            }                             </style>                         </head>                         <body>                             <div id=\"vis\">                             </div>                             <script>                                 var vega_json = \"{\\\"$schema\\\": \\\"https://vega.github.io/schema/vega/v4.json\\\", \\\"autosize\\\": {\\\"type\\\": \\\"fit\\\", \\\"resize\\\": false, \\\"contains\\\": \\\"padding\\\"}, \\\"padding\\\": 8, \\\"metadata\\\": {\\\"bubbleOpts\\\": {\\\"showAllFields\\\": false, \\\"fields\\\": [{\\\"field\\\": \\\"count\\\"}, {\\\"field\\\": \\\"label\\\"}, {\\\"field\\\": \\\"percentage\\\"}]}}, \\\"width\\\": 720, \\\"height\\\": 550, \\\"title\\\": \\\"Distribution of Values [string]\\\", \\\"style\\\": \\\"cell\\\", \\\"data\\\": [{\\\"name\\\": \\\"pts_store_store\\\"}, {\\\"name\\\": \\\"source_2\\\", \\\"values\\\": [{\\\"label\\\": \\\"B\\\", \\\"label_idx\\\": 0, \\\"count\\\": 37172, \\\"percentage\\\": \\\"30.318%\\\"}, {\\\"label\\\": \\\"C\\\", \\\"label_idx\\\": 1, \\\"count\\\": 29950, \\\"percentage\\\": \\\"24.4276%\\\"}, {\\\"label\\\": \\\"A\\\", \\\"label_idx\\\": 2, \\\"count\\\": 22314, \\\"percentage\\\": \\\"18.1996%\\\"}, {\\\"label\\\": \\\"D\\\", \\\"label_idx\\\": 3, \\\"count\\\": 19175, \\\"percentage\\\": \\\"15.6394%\\\"}, {\\\"label\\\": \\\"E\\\", \\\"label_idx\\\": 4, \\\"count\\\": 8990, \\\"percentage\\\": \\\"7.33237%\\\"}, {\\\"label\\\": \\\"F\\\", \\\"label_idx\\\": 5, \\\"count\\\": 3932, \\\"percentage\\\": \\\"3.20699%\\\"}, {\\\"label\\\": \\\"G\\\", \\\"label_idx\\\": 6, \\\"count\\\": 1074, \\\"percentage\\\": \\\"0.87597%\\\"}]}, {\\\"name\\\": \\\"data_0\\\", \\\"source\\\": \\\"source_2\\\", \\\"transform\\\": [{\\\"type\\\": \\\"formula\\\", \\\"expr\\\": \\\"toNumber(datum[\\\\\\\"count\\\\\\\"])\\\", \\\"as\\\": \\\"count\\\"}, {\\\"type\\\": \\\"filter\\\", \\\"expr\\\": \\\"datum[\\\\\\\"count\\\\\\\"] !== null &amp;&amp; !isNaN(datum[\\\\\\\"count\\\\\\\"])\\\"}]}], \\\"signals\\\": [{\\\"name\\\": \\\"unit\\\", \\\"value\\\": {}, \\\"on\\\": [{\\\"events\\\": \\\"mousemove\\\", \\\"update\\\": \\\"isTuple(group()) ? group() : unit\\\"}]}, {\\\"name\\\": \\\"pts_store\\\", \\\"update\\\": \\\"data(\\\\\\\"pts_store_store\\\\\\\").length &amp;&amp; {count: data(\\\\\\\"pts_store_store\\\\\\\")[0].values[0]}\\\"}, {\\\"name\\\": \\\"pts_store_tuple\\\", \\\"value\\\": {}, \\\"on\\\": [{\\\"events\\\": [{\\\"source\\\": \\\"scope\\\", \\\"type\\\": \\\"click\\\"}], \\\"update\\\": \\\"datum &amp;&amp; item().mark.marktype !== &apos;group&apos; ? {unit: \\\\\\\"\\\\\\\", encodings: [\\\\\\\"x\\\\\\\"], fields: [\\\\\\\"count\\\\\\\"], values: [datum[\\\\\\\"count\\\\\\\"]]} : null\\\", \\\"force\\\": true}]}, {\\\"name\\\": \\\"pts_store_modify\\\", \\\"on\\\": [{\\\"events\\\": {\\\"signal\\\": \\\"pts_store_tuple\\\"}, \\\"update\\\": \\\"modify(\\\\\\\"pts_store_store\\\\\\\", pts_store_tuple, true)\\\"}]}], \\\"marks\\\": [{\\\"name\\\": \\\"marks\\\", \\\"type\\\": \\\"rect\\\", \\\"style\\\": [\\\"bar\\\"], \\\"from\\\": {\\\"data\\\": \\\"data_0\\\"}, \\\"encode\\\": {\\\"hover\\\": {\\\"fill\\\": {\\\"value\\\": \\\"#7EC2F3\\\"}}, \\\"update\\\": {\\\"x\\\": {\\\"scale\\\": \\\"x\\\", \\\"field\\\": \\\"count\\\"}, \\\"x2\\\": {\\\"scale\\\": \\\"x\\\", \\\"value\\\": 0}, \\\"y\\\": {\\\"scale\\\": \\\"y\\\", \\\"field\\\": \\\"label\\\"}, \\\"height\\\": {\\\"scale\\\": \\\"y\\\", \\\"band\\\": true}, \\\"fill\\\": {\\\"value\\\": \\\"#108EE9\\\"}}}}], \\\"scales\\\": [{\\\"name\\\": \\\"x\\\", \\\"type\\\": \\\"linear\\\", \\\"domain\\\": {\\\"data\\\": \\\"data_0\\\", \\\"field\\\": \\\"count\\\"}, \\\"range\\\": [0, {\\\"signal\\\": \\\"width\\\"}], \\\"nice\\\": true, \\\"zero\\\": true}, {\\\"name\\\": \\\"y\\\", \\\"type\\\": \\\"band\\\", \\\"domain\\\": {\\\"data\\\": \\\"data_0\\\", \\\"field\\\": \\\"label\\\", \\\"sort\\\": {\\\"op\\\": \\\"mean\\\", \\\"field\\\": \\\"label_idx\\\", \\\"order\\\": \\\"descending\\\"}}, \\\"range\\\": [{\\\"signal\\\": \\\"height\\\"}, 0], \\\"paddingInner\\\": 0.1, \\\"paddingOuter\\\": 0.05}], \\\"axes\\\": [{\\\"orient\\\": \\\"top\\\", \\\"scale\\\": \\\"x\\\", \\\"labelOverlap\\\": true, \\\"tickCount\\\": {\\\"signal\\\": \\\"ceil(width/40)\\\"}, \\\"title\\\": \\\"Count\\\", \\\"zindex\\\": 1}, {\\\"orient\\\": \\\"top\\\", \\\"scale\\\": \\\"x\\\", \\\"domain\\\": false, \\\"grid\\\": true, \\\"labels\\\": false, \\\"maxExtent\\\": 0, \\\"minExtent\\\": 0, \\\"tickCount\\\": {\\\"signal\\\": \\\"ceil(width/40)\\\"}, \\\"ticks\\\": false, \\\"zindex\\\": 0, \\\"gridScale\\\": \\\"y\\\"}, {\\\"scale\\\": \\\"y\\\", \\\"labelOverlap\\\": true, \\\"orient\\\": \\\"left\\\", \\\"title\\\": \\\"Values\\\", \\\"zindex\\\": 1}], \\\"config\\\": {\\\"axis\\\": {\\\"gridColor\\\": \\\"rgba(204,204,204,1.0)\\\", \\\"labelFont\\\": \\\"\\\\\\\"San Francisco\\\\\\\", HelveticaNeue, Arial\\\", \\\"labelFontSize\\\": 12, \\\"labelPadding\\\": 10, \\\"labelColor\\\": \\\"rgba(0,0,0,0.847)\\\", \\\"tickColor\\\": \\\"rgb(136,136,136)\\\", \\\"titleFont\\\": \\\"\\\\\\\"San Francisco\\\\\\\", HelveticaNeue, Arial\\\", \\\"titleFontWeight\\\": \\\"normal\\\", \\\"titlePadding\\\": 20, \\\"titleFontSize\\\": 14, \\\"titleColor\\\": \\\"rgba(0,0,0,0.847)\\\"}, \\\"axisY\\\": {\\\"minExtent\\\": 30}, \\\"legend\\\": {\\\"labelFont\\\": \\\"\\\\\\\"San Francisco\\\\\\\", HelveticaNeue, Arial\\\", \\\"labelColor\\\": \\\"rgba(0,0,0,0.847)\\\", \\\"titleFont\\\": \\\"\\\\\\\"San Francisco\\\\\\\", HelveticaNeue, Arial\\\", \\\"cornerRadius\\\": 30, \\\"gradientLength\\\": 608, \\\"titleColor\\\": \\\"rgba(0,0,0,0.847)\\\"}, \\\"range\\\": {\\\"heatmap\\\": {\\\"scheme\\\": \\\"greenblue\\\"}}, \\\"style\\\": {\\\"rect\\\": {\\\"stroke\\\": \\\"rgba(200, 200, 200, 0.5)\\\"}, \\\"cell\\\": {\\\"stroke\\\": \\\"transparent\\\"}, \\\"group-title\\\": {\\\"fontSize\\\": 29, \\\"font\\\": \\\"HelveticaNeue, Arial\\\", \\\"fontWeight\\\": \\\"normal\\\", \\\"fill\\\": \\\"rgba(0,0,0,0.65)\\\"}}, \\\"title\\\": {\\\"color\\\": \\\"rgba(0,0,0,0.847)\\\", \\\"font\\\": \\\"\\\\\\\"San Francisco\\\\\\\", HelveticaNeue, Arial\\\", \\\"fontSize\\\": 18, \\\"fontWeight\\\": \\\"normal\\\", \\\"offset\\\": 30}}}\";                                 var vega_json_parsed = JSON.parse(vega_json);                                 var toolTipOpts = {                                     showAllFields: true                                 };                                 if(vega_json_parsed[\"metadata\"] != null){                                     if(vega_json_parsed[\"metadata\"][\"bubbleOpts\"] != null){                                         toolTipOpts = vega_json_parsed[\"metadata\"][\"bubbleOpts\"];                                     };                                 };                                 vegaEmbed(\"#vis\", vega_json_parsed).then(function (result) {                                     vegaTooltip.vega(result.view, toolTipOpts);                                  });                             </script>                         </body>                     </html>' src=\"demo_iframe_srcdoc.htm\">                         <p>Your browser does not support iframes.</p>                     </iframe>                 </body>             </html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "loans['grade'].show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">member_id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">loan_amnt</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">funded_amnt</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">funded_amnt_inv</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">term</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">int_rate</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">installment</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">grade</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">sub_grade</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1077501</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1296599</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4975</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"> 36 months</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10.65</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">162.87</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">B</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">B2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1077430</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1314167</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2500</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2500</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2500</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"> 60 months</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15.27</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">59.83</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">C4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1077175</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1313524</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2400</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2400</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2400</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"> 36 months</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15.96</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">84.33</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">C5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1076863</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1277178</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"> 36 months</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">13.49</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">339.31</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">C1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1075269</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1311441</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"> 36 months</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7.9</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">156.46</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1072053</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1288686</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"> 36 months</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">18.64</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">109.43</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">E</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">E1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1071795</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1306957</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5600</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5600</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5600</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"> 60 months</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">21.28</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">152.39</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">F</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">F2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1071570</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1306721</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5375</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5375</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5350</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"> 60 months</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">12.69</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">121.45</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">B</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">B5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1070078</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1305201</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6500</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6500</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6500</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"> 60 months</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">14.65</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">153.45</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">C3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1069908</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1305008</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">12000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">12000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">12000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"> 36 months</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">12.69</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">402.54</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">B</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">B5</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">emp_title</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">emp_length</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">home_ownership</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">annual_inc</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">is_inc_v</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">issue_d</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">loan_status</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">pymnt_plan</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"></td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10+ years</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">24000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Verified</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">20111201T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Fully Paid</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Ryder</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">&lt; 1 year</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">30000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Source Verified</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">20111201T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Charged Off</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"></td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10+ years</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">12252</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Not Verified</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">20111201T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Fully Paid</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">AIR RESOURCES BOARD</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10+ years</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">49200</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Source Verified</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">20111201T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Fully Paid</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Veolia Transportaton</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3 years</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">36000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Source Verified</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">20111201T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Fully Paid</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MKC Accounting </td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">9 years</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">48000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Source Verified</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">20111201T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Fully Paid</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"></td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4 years</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">OWN</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">40000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Source Verified</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">20111201T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Charged Off</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Starbucks</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">&lt; 1 year</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Verified</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">20111201T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Charged Off</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Southwest Rural metro</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5 years</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">OWN</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">72000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Not Verified</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">20111201T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Fully Paid</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">UCLA</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10+ years</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">OWN</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">75000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Source Verified</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">20111201T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Fully Paid</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">n</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">url</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">desc</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">purpose</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">zip_code</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">https://www.lendingclub.c<br>om/browse/loanDetail. ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">  Borrower added on<br>12/22/11 &gt; I need to ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">credit_card</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Computer</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">860xx</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">https://www.lendingclub.c<br>om/browse/loanDetail. ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">  Borrower added on<br>12/22/11 &gt; I plan to use ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">car</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">bike</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">309xx</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">https://www.lendingclub.c<br>om/browse/loanDetail. ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"></td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">small_business</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">real estate business</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">606xx</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">https://www.lendingclub.c<br>om/browse/loanDetail. ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">  Borrower added on<br>12/21/11 &gt; to pay for ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">other</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">personel</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">917xx</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">https://www.lendingclub.c<br>om/browse/loanDetail. ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"></td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">wedding</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">My wedding loan I promise<br>to pay back ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">852xx</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">https://www.lendingclub.c<br>om/browse/loanDetail. ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">  Borrower added on<br>12/16/11 &gt; Downpayment ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">car</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Car Downpayment</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">900xx</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">https://www.lendingclub.c<br>om/browse/loanDetail. ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">  Borrower added on<br>12/21/11 &gt; I own a small ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">small_business</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Expand Business &amp; Buy<br>Debt Portfolio ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">958xx</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">https://www.lendingclub.c<br>om/browse/loanDetail. ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">  Borrower added on<br>12/16/11 &gt; I&#x27;m trying to ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">other</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Building my credit<br>history. ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">774xx</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">https://www.lendingclub.c<br>om/browse/loanDetail. ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">  Borrower added on<br>12/15/11 &gt; I had recived ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">debt_consolidation</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">High intrest<br>Consolidation ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">853xx</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">https://www.lendingclub.c<br>om/browse/loanDetail. ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"></td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">debt_consolidation</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Consolidation</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">913xx</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">addr_state</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">dti</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">delinq_2yrs</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">earliest_cr_line</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">inq_last_6mths</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">mths_since_last_delinq</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">mths_since_last_record</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">AZ</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">27.65</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">19850101T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">GA</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">19990401T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">IL</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8.72</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">20011101T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">CA</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">20.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">19960201T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">35</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">AZ</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11.2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">20041101T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">CA</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.35</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">20070101T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">CA</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.55</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">20040401T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">TX</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">18.08</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">20040901T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">AZ</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16.12</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">19980101T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">CA</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10.78</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">19891001T000000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">open_acc</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">pub_rec</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">revol_bal</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">revol_util</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">total_acc</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">initial_list_status</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">out_prncp</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">out_prncp_inv</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">total_pymnt</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">13648</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">83.7</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">9</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">f</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5861.07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1687</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">9.4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">f</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1008.71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2956</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">98.5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">f</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3003.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5598</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">21.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">37</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">f</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">12226.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">9</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7963</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">28.3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">12</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">f</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5631.38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8221</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">87.5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">f</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3938.14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5210</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">32.6</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">13</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">f</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">646.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">9279</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">36.5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">f</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1476.19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">14</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4032</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">20.6</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">23</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">f</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7677.52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">12</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">23336</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">67.1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">34</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">f</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">13943.1</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">total_pymnt_inv</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">...</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5831.78</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1008.71</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3003.65</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">12226.3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5631.38</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3938.14</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">646.02</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1469.34</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7677.52</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">13943.1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">...</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[10 rows x 68 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tid\tint\n",
       "\tmember_id\tint\n",
       "\tloan_amnt\tint\n",
       "\tfunded_amnt\tint\n",
       "\tfunded_amnt_inv\tint\n",
       "\tterm\tstr\n",
       "\tint_rate\tfloat\n",
       "\tinstallment\tfloat\n",
       "\tgrade\tstr\n",
       "\tsub_grade\tstr\n",
       "\temp_title\tstr\n",
       "\temp_length\tstr\n",
       "\thome_ownership\tstr\n",
       "\tannual_inc\tint\n",
       "\tis_inc_v\tstr\n",
       "\tissue_d\tstr\n",
       "\tloan_status\tstr\n",
       "\tpymnt_plan\tstr\n",
       "\turl\tstr\n",
       "\tdesc\tstr\n",
       "\tpurpose\tstr\n",
       "\ttitle\tstr\n",
       "\tzip_code\tstr\n",
       "\taddr_state\tstr\n",
       "\tdti\tfloat\n",
       "\tdelinq_2yrs\tint\n",
       "\tearliest_cr_line\tstr\n",
       "\tinq_last_6mths\tint\n",
       "\tmths_since_last_delinq\tint\n",
       "\tmths_since_last_record\tint\n",
       "\topen_acc\tint\n",
       "\tpub_rec\tint\n",
       "\trevol_bal\tint\n",
       "\trevol_util\tfloat\n",
       "\ttotal_acc\tint\n",
       "\tinitial_list_status\tstr\n",
       "\tout_prncp\tfloat\n",
       "\tout_prncp_inv\tfloat\n",
       "\ttotal_pymnt\tfloat\n",
       "\ttotal_pymnt_inv\tfloat\n",
       "\ttotal_rec_prncp\tfloat\n",
       "\ttotal_rec_int\tfloat\n",
       "\ttotal_rec_late_fee\tfloat\n",
       "\trecoveries\tfloat\n",
       "\tcollection_recovery_fee\tfloat\n",
       "\tlast_pymnt_d\tstr\n",
       "\tlast_pymnt_amnt\tfloat\n",
       "\tnext_pymnt_d\tstr\n",
       "\tlast_credit_pull_d\tstr\n",
       "\tcollections_12_mths_ex_med\tint\n",
       "\tmths_since_last_major_derog\tstr\n",
       "\tpolicy_code\tint\n",
       "\tnot_compliant\tint\n",
       "\tstatus\tstr\n",
       "\tinactive_loans\tint\n",
       "\tbad_loans\tint\n",
       "\temp_length_num\tint\n",
       "\tgrade_num\tint\n",
       "\tsub_grade_num\tfloat\n",
       "\tdelinq_2yrs_zero\tint\n",
       "\tpub_rec_zero\tint\n",
       "\tcollections_12_mths_zero\tint\n",
       "\tshort_emp\tint\n",
       "\tpayment_inc_ratio\tfloat\n",
       "\tfinal_d\tstr\n",
       "\tlast_delinq_none\tint\n",
       "\tlast_record_none\tint\n",
       "\tlast_major_derog_none\tint\n",
       "\n",
       "Rows: 10\n",
       "\n",
       "Data:\n",
       "+---------+-----------+-----------+-------------+-----------------+------------+\n",
       "|    id   | member_id | loan_amnt | funded_amnt | funded_amnt_inv |    term    |\n",
       "+---------+-----------+-----------+-------------+-----------------+------------+\n",
       "| 1077501 |  1296599  |    5000   |     5000    |       4975      |  36 months |\n",
       "| 1077430 |  1314167  |    2500   |     2500    |       2500      |  60 months |\n",
       "| 1077175 |  1313524  |    2400   |     2400    |       2400      |  36 months |\n",
       "| 1076863 |  1277178  |   10000   |    10000    |      10000      |  36 months |\n",
       "| 1075269 |  1311441  |    5000   |     5000    |       5000      |  36 months |\n",
       "| 1072053 |  1288686  |    3000   |     3000    |       3000      |  36 months |\n",
       "| 1071795 |  1306957  |    5600   |     5600    |       5600      |  60 months |\n",
       "| 1071570 |  1306721  |    5375   |     5375    |       5350      |  60 months |\n",
       "| 1070078 |  1305201  |    6500   |     6500    |       6500      |  60 months |\n",
       "| 1069908 |  1305008  |   12000   |    12000    |      12000      |  36 months |\n",
       "+---------+-----------+-----------+-------------+-----------------+------------+\n",
       "+----------+-------------+-------+-----------+-----------------------+------------+\n",
       "| int_rate | installment | grade | sub_grade |       emp_title       | emp_length |\n",
       "+----------+-------------+-------+-----------+-----------------------+------------+\n",
       "|  10.65   |    162.87   |   B   |     B2    |                       | 10+ years  |\n",
       "|  15.27   |    59.83    |   C   |     C4    |         Ryder         |  < 1 year  |\n",
       "|  15.96   |    84.33    |   C   |     C5    |                       | 10+ years  |\n",
       "|  13.49   |    339.31   |   C   |     C1    |  AIR RESOURCES BOARD  | 10+ years  |\n",
       "|   7.9    |    156.46   |   A   |     A4    |  Veolia Transportaton |  3 years   |\n",
       "|  18.64   |    109.43   |   E   |     E1    |    MKC Accounting     |  9 years   |\n",
       "|  21.28   |    152.39   |   F   |     F2    |                       |  4 years   |\n",
       "|  12.69   |    121.45   |   B   |     B5    |       Starbucks       |  < 1 year  |\n",
       "|  14.65   |    153.45   |   C   |     C3    | Southwest Rural metro |  5 years   |\n",
       "|  12.69   |    402.54   |   B   |     B5    |          UCLA         | 10+ years  |\n",
       "+----------+-------------+-------+-----------+-----------------------+------------+\n",
       "+----------------+------------+-----------------+-----------------+-------------+\n",
       "| home_ownership | annual_inc |     is_inc_v    |     issue_d     | loan_status |\n",
       "+----------------+------------+-----------------+-----------------+-------------+\n",
       "|      RENT      |   24000    |     Verified    | 20111201T000000 |  Fully Paid |\n",
       "|      RENT      |   30000    | Source Verified | 20111201T000000 | Charged Off |\n",
       "|      RENT      |   12252    |   Not Verified  | 20111201T000000 |  Fully Paid |\n",
       "|      RENT      |   49200    | Source Verified | 20111201T000000 |  Fully Paid |\n",
       "|      RENT      |   36000    | Source Verified | 20111201T000000 |  Fully Paid |\n",
       "|      RENT      |   48000    | Source Verified | 20111201T000000 |  Fully Paid |\n",
       "|      OWN       |   40000    | Source Verified | 20111201T000000 | Charged Off |\n",
       "|      RENT      |   15000    |     Verified    | 20111201T000000 | Charged Off |\n",
       "|      OWN       |   72000    |   Not Verified  | 20111201T000000 |  Fully Paid |\n",
       "|      OWN       |   75000    | Source Verified | 20111201T000000 |  Fully Paid |\n",
       "+----------------+------------+-----------------+-----------------+-------------+\n",
       "+------------+-------------------------------+-------------------------------+-----+\n",
       "| pymnt_plan |              url              |              desc             | ... |\n",
       "+------------+-------------------------------+-------------------------------+-----+\n",
       "|     n      | https://www.lendingclub.co... |   Borrower added on 12/22/... | ... |\n",
       "|     n      | https://www.lendingclub.co... |   Borrower added on 12/22/... | ... |\n",
       "|     n      | https://www.lendingclub.co... |                               | ... |\n",
       "|     n      | https://www.lendingclub.co... |   Borrower added on 12/21/... | ... |\n",
       "|     n      | https://www.lendingclub.co... |                               | ... |\n",
       "|     n      | https://www.lendingclub.co... |   Borrower added on 12/16/... | ... |\n",
       "|     n      | https://www.lendingclub.co... |   Borrower added on 12/21/... | ... |\n",
       "|     n      | https://www.lendingclub.co... |   Borrower added on 12/16/... | ... |\n",
       "|     n      | https://www.lendingclub.co... |   Borrower added on 12/15/... | ... |\n",
       "|     n      | https://www.lendingclub.co... |                               | ... |\n",
       "+------------+-------------------------------+-------------------------------+-----+\n",
       "[10 rows x 68 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loans.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that over half of the loan grades are assigned values `B` or `C`. Each loan is assigned one of these grades, along with a more finely discretized feature called `sub_grade` (feel free to explore that feature column as well!). These values depend on the loan application and credit report, and determine the interest rate of the loan. More information can be found [here](https://www.lendingclub.com/public/rates-and-fees.action).\n",
    "\n",
    "Now, let's look at a different feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>Materializing SArray</pre>"
      ],
      "text/plain": [
       "Materializing SArray"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<html>                 <body>                     <iframe style=\"border:0;margin:0\" width=\"920\" height=\"770\" srcdoc='<html lang=\"en\">                         <head>                             <script src=\"https://cdnjs.cloudflare.com/ajax/libs/vega/5.4.0/vega.js\"></script>                             <script src=\"https://cdnjs.cloudflare.com/ajax/libs/vega-embed/4.0.0/vega-embed.js\"></script>                             <script src=\"https://cdnjs.cloudflare.com/ajax/libs/vega-tooltip/0.5.1/vega-tooltip.min.js\"></script>                             <link rel=\"stylesheet\" type=\"text/css\" href=\"https://cdnjs.cloudflare.com/ajax/libs/vega-tooltip/0.5.1/vega-tooltip.min.css\">                             <style>                             .vega-actions > a{                                 color:white;                                 text-decoration: none;                                 font-family: \"Arial\";                                 cursor:pointer;                                 padding:5px;                                 background:#AAAAAA;                                 border-radius:4px;                                 padding-left:10px;                                 padding-right:10px;                                 margin-right:5px;                             }                             .vega-actions{                                 margin-top:20px;                                 text-align:center                             }                            .vega-actions > a{                                 background:#999999;                            }                             </style>                         </head>                         <body>                             <div id=\"vis\">                             </div>                             <script>                                 var vega_json = \"{\\\"$schema\\\": \\\"https://vega.github.io/schema/vega/v4.json\\\", \\\"autosize\\\": {\\\"type\\\": \\\"fit\\\", \\\"resize\\\": false, \\\"contains\\\": \\\"padding\\\"}, \\\"padding\\\": 8, \\\"metadata\\\": {\\\"bubbleOpts\\\": {\\\"showAllFields\\\": false, \\\"fields\\\": [{\\\"field\\\": \\\"count\\\"}, {\\\"field\\\": \\\"label\\\"}, {\\\"field\\\": \\\"percentage\\\"}]}}, \\\"width\\\": 720, \\\"height\\\": 550, \\\"title\\\": \\\"Distribution of Values [string]\\\", \\\"style\\\": \\\"cell\\\", \\\"data\\\": [{\\\"name\\\": \\\"pts_store_store\\\"}, {\\\"name\\\": \\\"source_2\\\", \\\"values\\\": [{\\\"label\\\": \\\"MORTGAGE\\\", \\\"label_idx\\\": 0, \\\"count\\\": 59240, \\\"percentage\\\": \\\"48.317%\\\"}, {\\\"label\\\": \\\"RENT\\\", \\\"label_idx\\\": 1, \\\"count\\\": 53245, \\\"percentage\\\": \\\"43.4274%\\\"}, {\\\"label\\\": \\\"OWN\\\", \\\"label_idx\\\": 2, \\\"count\\\": 9943, \\\"percentage\\\": \\\"8.10965%\\\"}, {\\\"label\\\": \\\"OTHER\\\", \\\"label_idx\\\": 3, \\\"count\\\": 179, \\\"percentage\\\": \\\"0.145995%\\\"}]}, {\\\"name\\\": \\\"data_0\\\", \\\"source\\\": \\\"source_2\\\", \\\"transform\\\": [{\\\"type\\\": \\\"formula\\\", \\\"expr\\\": \\\"toNumber(datum[\\\\\\\"count\\\\\\\"])\\\", \\\"as\\\": \\\"count\\\"}, {\\\"type\\\": \\\"filter\\\", \\\"expr\\\": \\\"datum[\\\\\\\"count\\\\\\\"] !== null &amp;&amp; !isNaN(datum[\\\\\\\"count\\\\\\\"])\\\"}]}], \\\"signals\\\": [{\\\"name\\\": \\\"unit\\\", \\\"value\\\": {}, \\\"on\\\": [{\\\"events\\\": \\\"mousemove\\\", \\\"update\\\": \\\"isTuple(group()) ? group() : unit\\\"}]}, {\\\"name\\\": \\\"pts_store\\\", \\\"update\\\": \\\"data(\\\\\\\"pts_store_store\\\\\\\").length &amp;&amp; {count: data(\\\\\\\"pts_store_store\\\\\\\")[0].values[0]}\\\"}, {\\\"name\\\": \\\"pts_store_tuple\\\", \\\"value\\\": {}, \\\"on\\\": [{\\\"events\\\": [{\\\"source\\\": \\\"scope\\\", \\\"type\\\": \\\"click\\\"}], \\\"update\\\": \\\"datum &amp;&amp; item().mark.marktype !== &apos;group&apos; ? {unit: \\\\\\\"\\\\\\\", encodings: [\\\\\\\"x\\\\\\\"], fields: [\\\\\\\"count\\\\\\\"], values: [datum[\\\\\\\"count\\\\\\\"]]} : null\\\", \\\"force\\\": true}]}, {\\\"name\\\": \\\"pts_store_modify\\\", \\\"on\\\": [{\\\"events\\\": {\\\"signal\\\": \\\"pts_store_tuple\\\"}, \\\"update\\\": \\\"modify(\\\\\\\"pts_store_store\\\\\\\", pts_store_tuple, true)\\\"}]}], \\\"marks\\\": [{\\\"name\\\": \\\"marks\\\", \\\"type\\\": \\\"rect\\\", \\\"style\\\": [\\\"bar\\\"], \\\"from\\\": {\\\"data\\\": \\\"data_0\\\"}, \\\"encode\\\": {\\\"hover\\\": {\\\"fill\\\": {\\\"value\\\": \\\"#7EC2F3\\\"}}, \\\"update\\\": {\\\"x\\\": {\\\"scale\\\": \\\"x\\\", \\\"field\\\": \\\"count\\\"}, \\\"x2\\\": {\\\"scale\\\": \\\"x\\\", \\\"value\\\": 0}, \\\"y\\\": {\\\"scale\\\": \\\"y\\\", \\\"field\\\": \\\"label\\\"}, \\\"height\\\": {\\\"scale\\\": \\\"y\\\", \\\"band\\\": true}, \\\"fill\\\": {\\\"value\\\": \\\"#108EE9\\\"}}}}], \\\"scales\\\": [{\\\"name\\\": \\\"x\\\", \\\"type\\\": \\\"linear\\\", \\\"domain\\\": {\\\"data\\\": \\\"data_0\\\", \\\"field\\\": \\\"count\\\"}, \\\"range\\\": [0, {\\\"signal\\\": \\\"width\\\"}], \\\"nice\\\": true, \\\"zero\\\": true}, {\\\"name\\\": \\\"y\\\", \\\"type\\\": \\\"band\\\", \\\"domain\\\": {\\\"data\\\": \\\"data_0\\\", \\\"field\\\": \\\"label\\\", \\\"sort\\\": {\\\"op\\\": \\\"mean\\\", \\\"field\\\": \\\"label_idx\\\", \\\"order\\\": \\\"descending\\\"}}, \\\"range\\\": [{\\\"signal\\\": \\\"height\\\"}, 0], \\\"paddingInner\\\": 0.1, \\\"paddingOuter\\\": 0.05}], \\\"axes\\\": [{\\\"orient\\\": \\\"top\\\", \\\"scale\\\": \\\"x\\\", \\\"labelOverlap\\\": true, \\\"tickCount\\\": {\\\"signal\\\": \\\"ceil(width/40)\\\"}, \\\"title\\\": \\\"Count\\\", \\\"zindex\\\": 1}, {\\\"orient\\\": \\\"top\\\", \\\"scale\\\": \\\"x\\\", \\\"domain\\\": false, \\\"grid\\\": true, \\\"labels\\\": false, \\\"maxExtent\\\": 0, \\\"minExtent\\\": 0, \\\"tickCount\\\": {\\\"signal\\\": \\\"ceil(width/40)\\\"}, \\\"ticks\\\": false, \\\"zindex\\\": 0, \\\"gridScale\\\": \\\"y\\\"}, {\\\"scale\\\": \\\"y\\\", \\\"labelOverlap\\\": true, \\\"orient\\\": \\\"left\\\", \\\"title\\\": \\\"Values\\\", \\\"zindex\\\": 1}], \\\"config\\\": {\\\"axis\\\": {\\\"gridColor\\\": \\\"rgba(204,204,204,1.0)\\\", \\\"labelFont\\\": \\\"\\\\\\\"San Francisco\\\\\\\", HelveticaNeue, Arial\\\", \\\"labelFontSize\\\": 12, \\\"labelPadding\\\": 10, \\\"labelColor\\\": \\\"rgba(0,0,0,0.847)\\\", \\\"tickColor\\\": \\\"rgb(136,136,136)\\\", \\\"titleFont\\\": \\\"\\\\\\\"San Francisco\\\\\\\", HelveticaNeue, Arial\\\", \\\"titleFontWeight\\\": \\\"normal\\\", \\\"titlePadding\\\": 20, \\\"titleFontSize\\\": 14, \\\"titleColor\\\": \\\"rgba(0,0,0,0.847)\\\"}, \\\"axisY\\\": {\\\"minExtent\\\": 30}, \\\"legend\\\": {\\\"labelFont\\\": \\\"\\\\\\\"San Francisco\\\\\\\", HelveticaNeue, Arial\\\", \\\"labelColor\\\": \\\"rgba(0,0,0,0.847)\\\", \\\"titleFont\\\": \\\"\\\\\\\"San Francisco\\\\\\\", HelveticaNeue, Arial\\\", \\\"cornerRadius\\\": 30, \\\"gradientLength\\\": 608, \\\"titleColor\\\": \\\"rgba(0,0,0,0.847)\\\"}, \\\"range\\\": {\\\"heatmap\\\": {\\\"scheme\\\": \\\"greenblue\\\"}}, \\\"style\\\": {\\\"rect\\\": {\\\"stroke\\\": \\\"rgba(200, 200, 200, 0.5)\\\"}, \\\"cell\\\": {\\\"stroke\\\": \\\"transparent\\\"}, \\\"group-title\\\": {\\\"fontSize\\\": 29, \\\"font\\\": \\\"HelveticaNeue, Arial\\\", \\\"fontWeight\\\": \\\"normal\\\", \\\"fill\\\": \\\"rgba(0,0,0,0.65)\\\"}}, \\\"title\\\": {\\\"color\\\": \\\"rgba(0,0,0,0.847)\\\", \\\"font\\\": \\\"\\\\\\\"San Francisco\\\\\\\", HelveticaNeue, Arial\\\", \\\"fontSize\\\": 18, \\\"fontWeight\\\": \\\"normal\\\", \\\"offset\\\": 30}}}\";                                 var vega_json_parsed = JSON.parse(vega_json);                                 var toolTipOpts = {                                     showAllFields: true                                 };                                 if(vega_json_parsed[\"metadata\"] != null){                                     if(vega_json_parsed[\"metadata\"][\"bubbleOpts\"] != null){                                         toolTipOpts = vega_json_parsed[\"metadata\"][\"bubbleOpts\"];                                     };                                 };                                 vegaEmbed(\"#vis\", vega_json_parsed).then(function (result) {                                     vegaTooltip.vega(result.view, toolTipOpts);                                  });                             </script>                         </body>                     </html>' src=\"demo_iframe_srcdoc.htm\">                         <p>Your browser does not support iframes.</p>                     </iframe>                 </body>             </html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "loans['home_ownership'].show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This feature describes whether the loanee is mortaging, renting, or owns a home. We can see that a small percentage of the loanees own a home."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the target column\n",
    "\n",
    "The target column (label column) of the dataset that we are interested in is called `bad_loans`. In this column **1** means a risky (bad) loan **0** means a safe  loan.\n",
    "\n",
    "In order to make this more intuitive and consistent with the lectures, we reassign the target to be:\n",
    "* **+1** as a safe  loan, \n",
    "* **-1** as a risky (bad) loan. \n",
    "\n",
    "We put this in a new column called `safe_loans`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# safe_loans =  1 => safe\n",
    "# safe_loans = -1 => risky\n",
    "loans['safe_loans'] = loans['bad_loans'].apply(lambda x : +1 if x==0 else -1)\n",
    "loans = loans.remove_column('bad_loans')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us explore the distribution of the column `safe_loans`. This gives us a sense of how many safe and risky loans are present in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>Materializing SArray</pre>"
      ],
      "text/plain": [
       "Materializing SArray"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<html>                 <body>                     <iframe style=\"border:0;margin:0\" width=\"920\" height=\"770\" srcdoc='<html lang=\"en\">                         <head>                             <script src=\"https://cdnjs.cloudflare.com/ajax/libs/vega/5.4.0/vega.js\"></script>                             <script src=\"https://cdnjs.cloudflare.com/ajax/libs/vega-embed/4.0.0/vega-embed.js\"></script>                             <script src=\"https://cdnjs.cloudflare.com/ajax/libs/vega-tooltip/0.5.1/vega-tooltip.min.js\"></script>                             <link rel=\"stylesheet\" type=\"text/css\" href=\"https://cdnjs.cloudflare.com/ajax/libs/vega-tooltip/0.5.1/vega-tooltip.min.css\">                             <style>                             .vega-actions > a{                                 color:white;                                 text-decoration: none;                                 font-family: \"Arial\";                                 cursor:pointer;                                 padding:5px;                                 background:#AAAAAA;                                 border-radius:4px;                                 padding-left:10px;                                 padding-right:10px;                                 margin-right:5px;                             }                             .vega-actions{                                 margin-top:20px;                                 text-align:center                             }                            .vega-actions > a{                                 background:#999999;                            }                             </style>                         </head>                         <body>                             <div id=\"vis\">                             </div>                             <script>                                 var vega_json = \"{\\\"$schema\\\": \\\"https://vega.github.io/schema/vega/v4.json\\\", \\\"description\\\": \\\"A simple bar chart with embedded data.\\\", \\\"autosize\\\": {\\\"type\\\": \\\"fit\\\", \\\"resize\\\": false, \\\"contains\\\": \\\"padding\\\"}, \\\"width\\\": 720, \\\"height\\\": 550, \\\"padding\\\": 8, \\\"title\\\": \\\"Distribution of Values [integer]\\\", \\\"style\\\": \\\"cell\\\", \\\"signals\\\": [{\\\"name\\\": \\\"bins\\\", \\\"update\\\": \\\"data(\\\\\\\"bins_data\\\\\\\")[0]\\\"}, {\\\"name\\\": \\\"binCount\\\", \\\"update\\\": \\\"(bins.stop - bins.start) / bins.step\\\"}, {\\\"name\\\": \\\"nullGap\\\", \\\"update\\\": \\\"data(\\\\\\\"nulls\\\\\\\").length ? 10 : 0\\\"}, {\\\"name\\\": \\\"barStep\\\", \\\"update\\\": \\\"(width - nullGap) / (1 + binCount)\\\"}], \\\"data\\\": [{\\\"name\\\": \\\"source_2\\\", \\\"values\\\": [{\\\"left\\\": -9, \\\"right\\\": -8, \\\"count\\\": 0}, {\\\"left\\\": -8, \\\"right\\\": -7, \\\"count\\\": 0}, {\\\"left\\\": -7, \\\"right\\\": -6, \\\"count\\\": 0}, {\\\"left\\\": -6, \\\"right\\\": -5, \\\"count\\\": 0}, {\\\"left\\\": -5, \\\"right\\\": -4, \\\"count\\\": 0}, {\\\"left\\\": -4, \\\"right\\\": -3, \\\"count\\\": 0}, {\\\"left\\\": -3, \\\"right\\\": -2, \\\"count\\\": 0}, {\\\"left\\\": -2, \\\"right\\\": -1, \\\"count\\\": 0}, {\\\"left\\\": -1, \\\"right\\\": 0, \\\"count\\\": 23150}, {\\\"left\\\": 0, \\\"right\\\": 1, \\\"count\\\": 0}, {\\\"left\\\": 1, \\\"right\\\": 2, \\\"count\\\": 99457}, {\\\"left\\\": 2, \\\"right\\\": 3, \\\"count\\\": 0}, {\\\"left\\\": 3, \\\"right\\\": 4, \\\"count\\\": 0}, {\\\"left\\\": 4, \\\"right\\\": 5, \\\"count\\\": 0}, {\\\"left\\\": 5, \\\"right\\\": 6, \\\"count\\\": 0}, {\\\"left\\\": 6, \\\"right\\\": 7, \\\"count\\\": 0}, {\\\"left\\\": 7, \\\"right\\\": 8, \\\"count\\\": 0}, {\\\"left\\\": 8, \\\"right\\\": 9, \\\"count\\\": 0}, {\\\"left\\\": 9, \\\"right\\\": 10, \\\"count\\\": 0}, {\\\"left\\\": 10, \\\"right\\\": 11, \\\"count\\\": 0}, {\\\"start\\\": -9, \\\"stop\\\": 11, \\\"step\\\": 1}]}, {\\\"name\\\": \\\"counts\\\", \\\"source\\\": \\\"source_2\\\", \\\"transform\\\": [{\\\"type\\\": \\\"filter\\\", \\\"expr\\\": \\\"datum[\\\\\\\"missing\\\\\\\"] !== true &amp;&amp; datum[\\\\\\\"count\\\\\\\"] != null\\\"}]}, {\\\"name\\\": \\\"nulls\\\", \\\"source\\\": \\\"source_2\\\", \\\"transform\\\": [{\\\"expr\\\": \\\"datum[\\\\\\\"missing\\\\\\\"] === true &amp;&amp; datum[\\\\\\\"count\\\\\\\"] != null\\\", \\\"type\\\": \\\"filter\\\"}]}, {\\\"name\\\": \\\"bins_data\\\", \\\"source\\\": \\\"source_2\\\", \\\"transform\\\": [{\\\"expr\\\": \\\"datum[\\\\\\\"start\\\\\\\"] != null &amp;&amp; datum[\\\\\\\"stop\\\\\\\"] != null &amp;&amp; datum[\\\\\\\"step\\\\\\\"] != null\\\", \\\"type\\\": \\\"filter\\\"}]}], \\\"marks\\\": [{\\\"type\\\": \\\"rect\\\", \\\"from\\\": {\\\"data\\\": \\\"counts\\\"}, \\\"encode\\\": {\\\"update\\\": {\\\"x\\\": {\\\"scale\\\": \\\"xscale\\\", \\\"field\\\": \\\"left\\\", \\\"offset\\\": 1}, \\\"x2\\\": {\\\"scale\\\": \\\"xscale\\\", \\\"field\\\": \\\"right\\\"}, \\\"y\\\": {\\\"scale\\\": \\\"yscale\\\", \\\"field\\\": \\\"count\\\"}, \\\"y2\\\": {\\\"scale\\\": \\\"yscale\\\", \\\"value\\\": 0}, \\\"fill\\\": {\\\"value\\\": \\\"#108EE9\\\"}}, \\\"hover\\\": {\\\"fill\\\": {\\\"value\\\": \\\"#7EC2F3\\\"}}}}, {\\\"type\\\": \\\"rect\\\", \\\"from\\\": {\\\"data\\\": \\\"nulls\\\"}, \\\"encode\\\": {\\\"update\\\": {\\\"x\\\": {\\\"scale\\\": \\\"xscale-null\\\", \\\"value\\\": null, \\\"offset\\\": 1}, \\\"x2\\\": {\\\"scale\\\": \\\"xscale-null\\\", \\\"band\\\": 1}, \\\"y\\\": {\\\"scale\\\": \\\"yscale\\\", \\\"field\\\": \\\"count\\\"}, \\\"y2\\\": {\\\"scale\\\": \\\"yscale\\\", \\\"value\\\": 0}, \\\"fill\\\": {\\\"value\\\": \\\"#108EE9\\\"}}, \\\"hover\\\": {\\\"fill\\\": {\\\"value\\\": \\\"#7EC2F3\\\"}}}}], \\\"scales\\\": [{\\\"name\\\": \\\"yscale\\\", \\\"type\\\": \\\"linear\\\", \\\"range\\\": \\\"height\\\", \\\"round\\\": true, \\\"nice\\\": true, \\\"domain\\\": {\\\"fields\\\": [{\\\"data\\\": \\\"counts\\\", \\\"field\\\": \\\"count\\\"}, {\\\"data\\\": \\\"nulls\\\", \\\"field\\\": \\\"count\\\"}]}}, {\\\"name\\\": \\\"xscale\\\", \\\"type\\\": \\\"linear\\\", \\\"range\\\": [{\\\"signal\\\": \\\"nullGap ? barStep + nullGap : 0\\\"}, {\\\"signal\\\": \\\"width\\\"}], \\\"round\\\": true, \\\"domain\\\": {\\\"signal\\\": \\\"[bins.start, bins.stop]\\\"}, \\\"bins\\\": {\\\"signal\\\": \\\"bins\\\"}}, {\\\"name\\\": \\\"xscale-null\\\", \\\"type\\\": \\\"band\\\", \\\"range\\\": [{\\\"signal\\\": \\\"nullGap ? 0 : 1\\\"}, {\\\"signal\\\": \\\"nullGap ? barStep : 0\\\"}], \\\"round\\\": true, \\\"domain\\\": [{\\\"signal\\\": \\\"nullGap ? null : &apos;&apos;\\\"}]}], \\\"axes\\\": [{\\\"title\\\": \\\"Values\\\", \\\"orient\\\": \\\"bottom\\\", \\\"scale\\\": \\\"xscale\\\", \\\"tickMinStep\\\": 1, \\\"grid\\\": true}, {\\\"orient\\\": \\\"bottom\\\", \\\"scale\\\": \\\"xscale-null\\\"}, {\\\"title\\\": \\\"Count\\\", \\\"orient\\\": \\\"left\\\", \\\"scale\\\": \\\"yscale\\\", \\\"tickCount\\\": 5, \\\"offset\\\": {\\\"signal\\\": \\\"nullGap ? 5 : 0\\\"}, \\\"grid\\\": true}], \\\"config\\\": {\\\"axis\\\": {\\\"gridColor\\\": \\\"rgba(204,204,204,1.0)\\\", \\\"labelFont\\\": \\\"\\\\\\\"San Francisco\\\\\\\", HelveticaNeue, Arial\\\", \\\"labelFontSize\\\": 12, \\\"labelPadding\\\": 10, \\\"labelColor\\\": \\\"rgba(0,0,0,0.847)\\\", \\\"tickColor\\\": \\\"rgb(136,136,136)\\\", \\\"titleFont\\\": \\\"\\\\\\\"San Francisco\\\\\\\", HelveticaNeue, Arial\\\", \\\"titleFontWeight\\\": \\\"normal\\\", \\\"titlePadding\\\": 20, \\\"titleFontSize\\\": 14, \\\"titleColor\\\": \\\"rgba(0,0,0,0.847)\\\"}, \\\"axisY\\\": {\\\"minExtent\\\": 30}, \\\"legend\\\": {\\\"labelFont\\\": \\\"\\\\\\\"San Francisco\\\\\\\", HelveticaNeue, Arial\\\", \\\"labelColor\\\": \\\"rgba(0,0,0,0.847)\\\", \\\"titleFont\\\": \\\"\\\\\\\"San Francisco\\\\\\\", HelveticaNeue, Arial\\\", \\\"cornerRadius\\\": 30, \\\"gradientLength\\\": 608, \\\"titleColor\\\": \\\"rgba(0,0,0,0.847)\\\"}, \\\"range\\\": {\\\"heatmap\\\": {\\\"scheme\\\": \\\"greenblue\\\"}}, \\\"style\\\": {\\\"rect\\\": {\\\"stroke\\\": \\\"rgba(200, 200, 200, 0.5)\\\"}, \\\"cell\\\": {\\\"stroke\\\": \\\"transparent\\\"}, \\\"group-title\\\": {\\\"fontSize\\\": 29, \\\"font\\\": \\\"HelveticaNeue, Arial\\\", \\\"fontWeight\\\": \\\"normal\\\", \\\"fill\\\": \\\"rgba(0,0,0,0.65)\\\"}}, \\\"title\\\": {\\\"color\\\": \\\"rgba(0,0,0,0.847)\\\", \\\"font\\\": \\\"\\\\\\\"San Francisco\\\\\\\", HelveticaNeue, Arial\\\", \\\"fontSize\\\": 18, \\\"fontWeight\\\": \\\"normal\\\", \\\"offset\\\": 30}}}\";                                 var vega_json_parsed = JSON.parse(vega_json);                                 var toolTipOpts = {                                     showAllFields: true                                 };                                 if(vega_json_parsed[\"metadata\"] != null){                                     if(vega_json_parsed[\"metadata\"][\"bubbleOpts\"] != null){                                         toolTipOpts = vega_json_parsed[\"metadata\"][\"bubbleOpts\"];                                     };                                 };                                 vegaEmbed(\"#vis\", vega_json_parsed).then(function (result) {                                     vegaTooltip.vega(result.view, toolTipOpts);                                  });                             </script>                         </body>                     </html>' src=\"demo_iframe_srcdoc.htm\">                         <p>Your browser does not support iframes.</p>                     </iframe>                 </body>             </html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "loans['safe_loans'].show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should have:\n",
    "* Around 81% safe loans\n",
    "* Around 19% risky loans\n",
    "\n",
    "It looks like most of these loans are safe loans (thankfully). But this does make our problem of identifying risky loans challenging."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Features for the classification algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this assignment, we will be using a subset of features (categorical and numeric). The features we will be using are **described in the code comments** below. If you are a finance geek, the [LendingClub](https://www.lendingclub.com/) website has a lot more details about these features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = ['grade',                     # grade of the loan\n",
    "            'sub_grade',                 # sub-grade of the loan\n",
    "            'short_emp',                 # one year or less of employment\n",
    "            'emp_length_num',            # number of years of employment\n",
    "            'home_ownership',            # home_ownership status: own, mortgage or rent\n",
    "            'dti',                       # debt to income ratio\n",
    "            'purpose',                   # the purpose of the loan\n",
    "            'term',                      # the term of the loan\n",
    "            'last_delinq_none',          # has borrower had a delinquincy\n",
    "            'last_major_derog_none',     # has borrower had 90 day or worse rating\n",
    "            'revol_util',                # percent of available credit being used\n",
    "            'total_rec_late_fee',        # total late fees received to day\n",
    "           ]\n",
    "\n",
    "target = 'safe_loans'                   # prediction target (y) (+1 means safe, -1 is risky)\n",
    "\n",
    "# Extract the feature columns and target column\n",
    "loans = loans[features + [target]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What remains now is a **subset of features** and the **target** that we will use for the rest of this notebook. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample data to balance classes\n",
    "\n",
    "As we explored above, our data is disproportionally full of safe loans.  Let's create two datasets: one with just the safe loans (`safe_loans_raw`) and one with just the risky loans (`risky_loans_raw`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of safe loans  : 99457\n",
      "Number of risky loans : 23150\n"
     ]
    }
   ],
   "source": [
    "safe_loans_raw = loans[loans[target] == +1]\n",
    "risky_loans_raw = loans[loans[target] == -1]\n",
    "print(\"Number of safe loans  : %s\" % len(safe_loans_raw))\n",
    "print(\"Number of risky loans : %s\" % len(risky_loans_raw))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, write some code to compute below the percentage of safe and risky loans in the dataset and validate these numbers against what was given using `.show` earlier in the assignment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percentage of safe loans  : 0.8111853319957262\n",
      "Percentage of risky loans : 0.18881466800427382\n"
     ]
    }
   ],
   "source": [
    "print(\"Percentage of safe loans  :\", len(safe_loans_raw) / (len(safe_loans_raw) + len(risky_loans_raw))) \n",
    "print(\"Percentage of risky loans :\", len(risky_loans_raw) / (len(safe_loans_raw) + len(risky_loans_raw)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One way to combat class imbalance is to undersample the larger class until the class distribution is approximately half and half. Here, we will undersample the larger class (safe loans) in order to balance out our dataset. This means we are throwing away many data points. We used `seed=1` so everyone gets the same results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Since there are fewer risky loans than safe loans, find the ratio of the sizes\n",
    "# and use that percentage to undersample the safe loans.\n",
    "percentage = len(risky_loans_raw)/float(len(safe_loans_raw))\n",
    "\n",
    "risky_loans = risky_loans_raw\n",
    "safe_loans = safe_loans_raw.sample(percentage, seed=1)\n",
    "\n",
    "# Append the risky_loans with the downsampled version of safe_loans\n",
    "loans_data = risky_loans.append(safe_loans)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's verify that the resulting percentage of safe and risky loans are each nearly 50%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percentage of safe loans                 : 0.5022361744216048\n",
      "Percentage of risky loans                : 0.4977638255783951\n",
      "Total number of loans in our new dataset : 46508\n"
     ]
    }
   ],
   "source": [
    "print(\"Percentage of safe loans                 :\", len(safe_loans) / float(len(loans_data)))\n",
    "print(\"Percentage of risky loans                :\", len(risky_loans) / float(len(loans_data)))\n",
    "print(\"Total number of loans in our new dataset :\", len(loans_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** There are many approaches for dealing with imbalanced data, including some where we modify the learning algorithm. These approaches are beyond the scope of this course, but some of them are reviewed in this [paper](http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=5128907&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F69%2F5173046%2F05128907.pdf%3Farnumber%3D5128907 ). For this assignment, we use the simplest possible approach, where we subsample the overly represented class to get a more balanced dataset. In general, and especially when the data is highly imbalanced, we recommend using more advanced methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split data into training and validation sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We split the data into training and validation sets using an 80/20 split and specifying `seed=1` so everyone gets the same results.\n",
    "\n",
    "**Note**: In previous assignments, we have called this a **train-test split**. However, the portion of data that we don't train on will be used to help **select model parameters** (this is known as model selection). Thus, this portion of data should be called a **validation set**. Recall that examining performance of various potential models (i.e. models with different parameters) should be on validation set, while evaluation of the final selected model should always be on test data. Typically, we would also save a portion of the data (a real test set) to test our final model on or use cross-validation on the training set to select our final model. But for the learning purposes of this assignment, we won't do that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data, validation_data = loans_data.random_split(.8, seed=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use decision tree to build a classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's use the built-in Turi Create decision tree learner to create a loan prediction model on the training data. (In the next assignment, you will implement your own decision tree learning algorithm.)  Our feature columns and target column have already been decided above. Use `validation_set=None` to get the same results as everyone else."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>Decision tree classifier:</pre>"
      ],
      "text/plain": [
       "Decision tree classifier:"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>--------------------------------------------------------</pre>"
      ],
      "text/plain": [
       "--------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of examples          : 37224</pre>"
      ],
      "text/plain": [
       "Number of examples          : 37224"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of classes           : 2</pre>"
      ],
      "text/plain": [
       "Number of classes           : 2"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of feature columns   : 12</pre>"
      ],
      "text/plain": [
       "Number of feature columns   : 12"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of unpacked features : 12</pre>"
      ],
      "text/plain": [
       "Number of unpacked features : 12"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+--------------+-------------------+-------------------+</pre>"
      ],
      "text/plain": [
       "+-----------+--------------+-------------------+-------------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| Iteration | Elapsed Time | Training Accuracy | Training Log Loss |</pre>"
      ],
      "text/plain": [
       "| Iteration | Elapsed Time | Training Accuracy | Training Log Loss |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+--------------+-------------------+-------------------+</pre>"
      ],
      "text/plain": [
       "+-----------+--------------+-------------------+-------------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 1         | 0.030152     | 0.640581          | 0.631396          |</pre>"
      ],
      "text/plain": [
       "| 1         | 0.030152     | 0.640581          | 0.631396          |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+--------------+-------------------+-------------------+</pre>"
      ],
      "text/plain": [
       "+-----------+--------------+-------------------+-------------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "decision_tree_model = turicreate.decision_tree_classifier.create(train_data,\n",
    "                                                                 validation_set=None,\n",
    "                                                                 target = target,\n",
    "                                                                 features = features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a smaller tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Typically the max depth of the tree is capped at 6. However, such a tree can be hard to visualize graphically, and moreover, it may overfit..  Here, we instead learn a smaller model with **max depth of 2** to gain some intuition and to understand the learned tree more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>Decision tree classifier:</pre>"
      ],
      "text/plain": [
       "Decision tree classifier:"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>--------------------------------------------------------</pre>"
      ],
      "text/plain": [
       "--------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of examples          : 37224</pre>"
      ],
      "text/plain": [
       "Number of examples          : 37224"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of classes           : 2</pre>"
      ],
      "text/plain": [
       "Number of classes           : 2"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of feature columns   : 12</pre>"
      ],
      "text/plain": [
       "Number of feature columns   : 12"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of unpacked features : 12</pre>"
      ],
      "text/plain": [
       "Number of unpacked features : 12"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+--------------+-------------------+-------------------+</pre>"
      ],
      "text/plain": [
       "+-----------+--------------+-------------------+-------------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| Iteration | Elapsed Time | Training Accuracy | Training Log Loss |</pre>"
      ],
      "text/plain": [
       "| Iteration | Elapsed Time | Training Accuracy | Training Log Loss |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+--------------+-------------------+-------------------+</pre>"
      ],
      "text/plain": [
       "+-----------+--------------+-------------------+-------------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 1         | 0.020431     | 0.613502          | 0.658759          |</pre>"
      ],
      "text/plain": [
       "| 1         | 0.020431     | 0.613502          | 0.658759          |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+--------------+-------------------+-------------------+</pre>"
      ],
      "text/plain": [
       "+-----------+--------------+-------------------+-------------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "small_model = turicreate.decision_tree_classifier.create(train_data,\n",
    "                                                         validation_set=None,\n",
    "                                                         target = target,\n",
    "                                                         features = features,\n",
    "                                                         max_depth = 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making predictions\n",
    "\n",
    "Let's consider two positive and two negative examples **from the validation set** and see what the model predicts. We will do the following:\n",
    "* Predict whether or not a loan is safe.\n",
    "* Predict the probability that a loan is safe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">grade</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">sub_grade</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">short_emp</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">emp_length_num</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">home_ownership</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">dti</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">purpose</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">term</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">last_delinq_none</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">B</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">B3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">OWN</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11.18</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">credit_card</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"> 36 months</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">D</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">D1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16.85</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">debt_consolidation</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"> 36 months</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">D</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">D2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">13.97</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">other</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"> 60 months</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MORTGAGE</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16.33</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">debt_consolidation</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"> 36 months</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">last_major_derog_none</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">revol_util</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">total_rec_late_fee</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">safe_loans</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">82.4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">96.4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">59.5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">62.1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[4 rows x 13 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tgrade\tstr\n",
       "\tsub_grade\tstr\n",
       "\tshort_emp\tint\n",
       "\temp_length_num\tint\n",
       "\thome_ownership\tstr\n",
       "\tdti\tfloat\n",
       "\tpurpose\tstr\n",
       "\tterm\tstr\n",
       "\tlast_delinq_none\tint\n",
       "\tlast_major_derog_none\tint\n",
       "\trevol_util\tfloat\n",
       "\ttotal_rec_late_fee\tfloat\n",
       "\tsafe_loans\tint\n",
       "\n",
       "Rows: 4\n",
       "\n",
       "Data:\n",
       "+-------+-----------+-----------+----------------+----------------+-------+\n",
       "| grade | sub_grade | short_emp | emp_length_num | home_ownership |  dti  |\n",
       "+-------+-----------+-----------+----------------+----------------+-------+\n",
       "|   B   |     B3    |     0     |       11       |      OWN       | 11.18 |\n",
       "|   D   |     D1    |     0     |       10       |      RENT      | 16.85 |\n",
       "|   D   |     D2    |     0     |       3        |      RENT      | 13.97 |\n",
       "|   A   |     A5    |     0     |       11       |    MORTGAGE    | 16.33 |\n",
       "+-------+-----------+-----------+----------------+----------------+-------+\n",
       "+--------------------+------------+------------------+-----------------------+\n",
       "|      purpose       |    term    | last_delinq_none | last_major_derog_none |\n",
       "+--------------------+------------+------------------+-----------------------+\n",
       "|    credit_card     |  36 months |        1         |           1           |\n",
       "| debt_consolidation |  36 months |        1         |           1           |\n",
       "|       other        |  60 months |        0         |           1           |\n",
       "| debt_consolidation |  36 months |        1         |           1           |\n",
       "+--------------------+------------+------------------+-----------------------+\n",
       "+------------+--------------------+------------+\n",
       "| revol_util | total_rec_late_fee | safe_loans |\n",
       "+------------+--------------------+------------+\n",
       "|    82.4    |        0.0         |     1      |\n",
       "|    96.4    |        0.0         |     1      |\n",
       "|    59.5    |        0.0         |     -1     |\n",
       "|    62.1    |        0.0         |     -1     |\n",
       "+------------+--------------------+------------+\n",
       "[4 rows x 13 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "validation_safe_loans = validation_data[validation_data[target] == 1]\n",
    "validation_risky_loans = validation_data[validation_data[target] == -1]\n",
    "\n",
    "sample_validation_data_risky = validation_risky_loans[0:2]\n",
    "sample_validation_data_safe = validation_safe_loans[0:2]\n",
    "\n",
    "sample_validation_data = sample_validation_data_safe.append(sample_validation_data_risky)\n",
    "sample_validation_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore label predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will use our model  to predict whether or not a loan is likely to default. For each row in the **sample_validation_data**, use the **decision_tree_model** to predict whether or not the loan is classified as a **safe loan**. \n",
    "\n",
    "**Hint:** Be sure to use the `.predict()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, -1, -1, 1]\n",
      "[1, 1, -1, -1]\n"
     ]
    }
   ],
   "source": [
    "print(decision_tree_model.predict(sample_validation_data))\n",
    "print(sample_validation_data['safe_loans'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** What percentage of the predictions on `sample_validation_data` did `decision_tree_model` get correct?\n",
    "\n",
    "**A:** 0.25"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore probability predictions\n",
    "\n",
    "For each row in the **sample_validation_data**, what is the probability (according **decision_tree_model**) of a loan being classified as **safe**? \n",
    "\n",
    "\n",
    "**Hint:** Set `output_type='probability'` to make **probability** predictions using **decision_tree_model** on `sample_validation_data`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype: float\n",
       "Rows: 4\n",
       "[0.6532223224639893, 0.463798463344574, 0.356814444065094, 0.7621196508407593]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decision_tree_model.predict(sample_validation_data, output_type='probability')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** Which loan has the highest probability of being classified as a **safe loan**?\n",
    "\n",
    "**A:** Last one\n",
    "\n",
    "**Checkpoint:** Can you verify that for all the predictions with `probability >= 0.5`, the model predicted the label **+1**?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tricky predictions!\n",
    "\n",
    "Now, we will explore something pretty interesting. For each row in the **sample_validation_data**, what is the probability (according to **small_model**) of a loan being classified as **safe**?\n",
    "\n",
    "**Hint:** Set `output_type='probability'` to make **probability** predictions using **small_model** on `sample_validation_data`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype: float\n",
       "Rows: 4\n",
       "[0.5803016424179077, 0.4085058867931366, 0.4085058867931366, 0.7454202175140381]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "small_model.predict(sample_validation_data, output_type='probability')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** Notice that the probability preditions are the **exact same** for the 2nd and 3rd loans. Why would this happen?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "they are similar for the small model. They split with the same features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating accuracy of the decision tree model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that the accuracy is defined as follows:\n",
    "$$\n",
    "\\mbox{accuracy} = \\frac{\\mbox{# correctly classified examples}}{\\mbox{# total examples}}\n",
    "$$\n",
    "\n",
    "Let us start by evaluating the accuracy of the `small_model` and `decision_tree_model` on the training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6135020416935311\n",
      "0.6405813453685794\n"
     ]
    }
   ],
   "source": [
    "print(small_model.evaluate(train_data)['accuracy'])\n",
    "print(decision_tree_model.evaluate(train_data)['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Checkpoint:** You should see that the **small_model** performs worse than the **decision_tree_model** on the training data.\n",
    "\n",
    "\n",
    "Now, let us evaluate the accuracy of the **small_model** and **decision_tree_model** on the entire **validation_data**, not just the subsample considered above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6193451098664369\n",
      "0.6367944851357173\n"
     ]
    }
   ],
   "source": [
    "print(small_model.evaluate(validation_data)['accuracy'])\n",
    "print(decision_tree_model.evaluate(validation_data)['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** What is the accuracy of `decision_tree_model` on the validation set, rounded to the nearest .01?\n",
    "\n",
    "**A:** 0.64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating accuracy of a complex decision tree model\n",
    "\n",
    "Here, we will train a large decision tree with `max_depth=10`. This will allow the learned tree to become very deep, and result in a very complex model. Recall that in lecture, we prefer simpler models with similar predictive power. This will be an example of a more complicated model which has similar predictive power, i.e. something we don't want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>Decision tree classifier:</pre>"
      ],
      "text/plain": [
       "Decision tree classifier:"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>--------------------------------------------------------</pre>"
      ],
      "text/plain": [
       "--------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of examples          : 37224</pre>"
      ],
      "text/plain": [
       "Number of examples          : 37224"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of classes           : 2</pre>"
      ],
      "text/plain": [
       "Number of classes           : 2"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of feature columns   : 12</pre>"
      ],
      "text/plain": [
       "Number of feature columns   : 12"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of unpacked features : 12</pre>"
      ],
      "text/plain": [
       "Number of unpacked features : 12"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+--------------+-------------------+-------------------+</pre>"
      ],
      "text/plain": [
       "+-----------+--------------+-------------------+-------------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| Iteration | Elapsed Time | Training Accuracy | Training Log Loss |</pre>"
      ],
      "text/plain": [
       "| Iteration | Elapsed Time | Training Accuracy | Training Log Loss |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+--------------+-------------------+-------------------+</pre>"
      ],
      "text/plain": [
       "+-----------+--------------+-------------------+-------------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 1         | 0.068331     | 0.665538          | 0.606828          |</pre>"
      ],
      "text/plain": [
       "| 1         | 0.068331     | 0.665538          | 0.606828          |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+--------------+-------------------+-------------------+</pre>"
      ],
      "text/plain": [
       "+-----------+--------------+-------------------+-------------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "big_model = turicreate.decision_tree_classifier.create(train_data, validation_set=None,\n",
    "                                                       target = target, features = features, max_depth = 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us evaluate **big_model** on the training set and validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.665538362346873\n",
      "0.6274235243429557\n"
     ]
    }
   ],
   "source": [
    "print(big_model.evaluate(train_data)['accuracy'])\n",
    "print(big_model.evaluate(validation_data)['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Checkpoint:** We should see that **big_model** has even better performance on the training set than **decision_tree_model** did on the training set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** How does the performance of **big_model** on the validation set compare to **decision_tree_model** on the validation set? Is this a sign of overfitting?\n",
    "\n",
    "**A:** it performs worse, it's overfitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quantifying the cost of mistakes\n",
    "\n",
    "Every mistake the model makes costs money. In this section, we will try and quantify the cost of each mistake made by the model.\n",
    "\n",
    "Assume the following:\n",
    "\n",
    "* **False negatives**: Loans that were actually safe but were predicted to be risky. This results in an oppurtunity cost of losing a loan that would have otherwise been accepted. \n",
    "* **False positives**: Loans that were actually risky but were predicted to be safe. These are much more expensive because it results in a risky loan being given. \n",
    "* **Correct predictions**: All correct predictions don't typically incur any cost.\n",
    "\n",
    "\n",
    "Let's write code that can compute the cost of mistakes made by the model. Complete the following 4 steps:\n",
    "1. First, let us compute the predictions made by the model.\n",
    "1. Second, compute the number of false positives.\n",
    "2. Third, compute the number of false negatives.\n",
    "3. Finally, compute the cost of mistakes made by the model by adding up the costs of true positives and false positives.\n",
    "\n",
    "First, let us make predictions on `validation_data` using the `decision_tree_model`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = decision_tree_model.predict(validation_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**False positives** are predictions where the model predicts +1 but the true label is -1. Complete the following code block for the number of false positives:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "false_pos:  1656\n",
      "false_neg:  1716\n"
     ]
    }
   ],
   "source": [
    "#predictions[predictions[target] != predictions[target]]\n",
    "\n",
    "false_negs = 0\n",
    "false_pos = 0\n",
    "for i, it in enumerate(predictions):\n",
    "    if validation_data[i][target] == -1 and it == +1:\n",
    "        false_pos +=1\n",
    "    if validation_data[i][target] == +1 and it == -1:\n",
    "        false_negs +=1\n",
    "\n",
    "print(\"false_pos: \", false_pos)\n",
    "print(\"false_neg: \", false_negs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**False negatives** are predictions where the model predicts -1 but the true label is +1. Complete the following code block for the number of false negatives:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** Let us assume that each mistake costs money:\n",
    "* Assume a cost of \\$10,000 per false negative.\n",
    "* Assume a cost of \\$20,000 per false positive.\n",
    "\n",
    "What is the total cost of mistakes made by `decision_tree_model` on `validation_data`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cost_false_pos:  33120000\n",
      "cost_false_neg:  17160000\n",
      "cost_false_neg:  50280000\n"
     ]
    }
   ],
   "source": [
    "print(\"cost_false_pos: \", false_pos * 20000)\n",
    "print(\"cost_false_neg: \", false_negs * 10000)\n",
    "print(\"cost_false_neg: \", (false_pos * 20000) + (false_negs * 10000))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}