09.3_Engineering_rare_values_3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Engineering Rare Labels / Rare Categories\n",
    "\n",
    "Rare values are labels/ categories within a categorical variable that are only present for a small percentage of the observations.\n",
    "\n",
    "There is no rule of thumb to determine how small is a small percentage, but typically, any value below 5% may cause over-fitting in trees.\n",
    "\n",
    "Rare labels may exist in variables that have intrinsically a huge number of labels, or they can be present in variables with few labels (e.g., 2-10). There is no rule of thumb to determine how many different labels is big (and therefore represent high cardinality) and it will depend as well on how many observations there are in the dataset. In a dataset with 1,000 observations, 100 labels may seem a lot, whereas in a dataset with 100,000 observations it may not be so high.\n",
    "\n",
    "In situations where rare labels are present in variables with only a few categories, the rare label may be adding some information. On the other hand, in variables with a high number of categories, likely there will be very many labels with a low frequency, which will quite likely add noise instead of information. \n",
    "\n",
    "Whether rare labels should be processed before training a machine learning algorithm will depend on the dataset and problem at hand. Ideally, if there are not too many variables, you could try and explore the variables and their categories one at a time, and determine whether the rare labels add noise or information.\n",
    "\n",
    "If, on the other hand, the dataset has very many categorical variables, and exploring one at a time is not an option you may choose to sacrifice the ideal / optimal performance for a higher delivery speed.\n",
    "\n",
    "### Engineering rare labels\n",
    "\n",
    "There are multiple ways of accounting for rare labels. Some of them handle rare labels at the same time of converting labels into numbers. I will explain those in the section \"Engineer labels of categorical variables\".\n",
    "\n",
    "In this section of the course, I will expand on how to handle rare labels by re-categorising the observation that show rare labels for a certain variable. These observations can be re-categorised by:\n",
    "\n",
    "- Replacing the rare label by most frequent label\n",
    "- Grouping the observations that show rare labels into a unique category (with a new label like 'Rare', or 'Other')\n",
    "\n",
    "In this and the comming lectures I will explain when it is convenient to use one or the other way of replacing rare values, and evaluate the consequences of replacing rare labels in variables with:\n",
    "\n",
    "- One predominant category\n",
    "- A small number of different categories\n",
    "- High cardinality\n",
    "\n",
    "Specifically, in this lecture I will demonstrate how to work with rare labels in variables with one predominant category using the House Sale dataset from Kaggle.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "===============================================================================\n",
    "\n",
    "## Real Life example: \n",
    "\n",
    "### Predicting Sale Price of Houses\n",
    "\n",
    "The problem at hand aims to predict the final sale price of homes based on different explanatory variables describing aspects of residential homes. Predicting house prices is useful to identify fruitful investments, or to determine whether the price advertised for a house is over or underestimated, before making a buying judgment.\n",
    "\n",
    "============================================================================="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "% matplotlib inline\n",
    "\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "#from sklearn.ensemble import GradientBoostingRegressor\n",
    "\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "pd.set_option('display.max_columns', None) # to display the total number columns present in the dataset\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## House Sale Price dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>LotConfig</th>\n",
       "      <th>LandSlope</th>\n",
       "      <th>Neighborhood</th>\n",
       "      <th>Condition1</th>\n",
       "      <th>Condition2</th>\n",
       "      <th>BldgType</th>\n",
       "      <th>HouseStyle</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>RoofStyle</th>\n",
       "      <th>RoofMatl</th>\n",
       "      <th>Exterior1st</th>\n",
       "      <th>Exterior2nd</th>\n",
       "      <th>MasVnrType</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>ExterQual</th>\n",
       "      <th>ExterCond</th>\n",
       "      <th>Foundation</th>\n",
       "      <th>BsmtQual</th>\n",
       "      <th>BsmtCond</th>\n",
       "      <th>BsmtExposure</th>\n",
       "      <th>BsmtFinType1</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtFinType2</th>\n",
       "      <th>BsmtFinSF2</th>\n",
       "      <th>BsmtUnfSF</th>\n",
       "      <th>TotalBsmtSF</th>\n",
       "      <th>Heating</th>\n",
       "      <th>HeatingQC</th>\n",
       "      <th>CentralAir</th>\n",
       "      <th>Electrical</th>\n",
       "      <th>1stFlrSF</th>\n",
       "      <th>2ndFlrSF</th>\n",
       "      <th>LowQualFinSF</th>\n",
       "      <th>GrLivArea</th>\n",
       "      <th>BsmtFullBath</th>\n",
       "      <th>BsmtHalfBath</th>\n",
       "      <th>FullBath</th>\n",
       "      <th>HalfBath</th>\n",
       "      <th>BedroomAbvGr</th>\n",
       "      <th>KitchenAbvGr</th>\n",
       "      <th>KitchenQual</th>\n",
       "      <th>TotRmsAbvGrd</th>\n",
       "      <th>Functional</th>\n",
       "      <th>Fireplaces</th>\n",
       "      <th>FireplaceQu</th>\n",
       "      <th>GarageType</th>\n",
       "      <th>GarageYrBlt</th>\n",
       "      <th>GarageFinish</th>\n",
       "      <th>GarageCars</th>\n",
       "      <th>GarageArea</th>\n",
       "      <th>GarageQual</th>\n",
       "      <th>GarageCond</th>\n",
       "      <th>PavedDrive</th>\n",
       "      <th>WoodDeckSF</th>\n",
       "      <th>OpenPorchSF</th>\n",
       "      <th>EnclosedPorch</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>CollgCr</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>2Story</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>2003</td>\n",
       "      <td>2003</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>196.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>PConc</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>No</td>\n",
       "      <td>GLQ</td>\n",
       "      <td>706</td>\n",
       "      <td>Unf</td>\n",
       "      <td>0</td>\n",
       "      <td>150</td>\n",
       "      <td>856</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Ex</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>856</td>\n",
       "      <td>854</td>\n",
       "      <td>0</td>\n",
       "      <td>1710</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>8</td>\n",
       "      <td>Typ</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>RFn</td>\n",
       "      <td>2</td>\n",
       "      <td>548</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>61</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>FR2</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>Veenker</td>\n",
       "      <td>Feedr</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>1Story</td>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>1976</td>\n",
       "      <td>1976</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>MetalSd</td>\n",
       "      <td>MetalSd</td>\n",
       "      <td>None</td>\n",
       "      <td>0.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>CBlock</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>Gd</td>\n",
       "      <td>ALQ</td>\n",
       "      <td>978</td>\n",
       "      <td>Unf</td>\n",
       "      <td>0</td>\n",
       "      <td>284</td>\n",
       "      <td>1262</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Ex</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>1262</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1262</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>TA</td>\n",
       "      <td>6</td>\n",
       "      <td>Typ</td>\n",
       "      <td>1</td>\n",
       "      <td>TA</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>1976.0</td>\n",
       "      <td>RFn</td>\n",
       "      <td>2</td>\n",
       "      <td>460</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>298</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>CollgCr</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>2Story</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>2001</td>\n",
       "      <td>2002</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>162.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>PConc</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>Mn</td>\n",
       "      <td>GLQ</td>\n",
       "      <td>486</td>\n",
       "      <td>Unf</td>\n",
       "      <td>0</td>\n",
       "      <td>434</td>\n",
       "      <td>920</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Ex</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>920</td>\n",
       "      <td>866</td>\n",
       "      <td>0</td>\n",
       "      <td>1786</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>6</td>\n",
       "      <td>Typ</td>\n",
       "      <td>1</td>\n",
       "      <td>TA</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>2001.0</td>\n",
       "      <td>RFn</td>\n",
       "      <td>2</td>\n",
       "      <td>608</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>42</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>70</td>\n",
       "      <td>RL</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9550</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Corner</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>Crawfor</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>2Story</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>1915</td>\n",
       "      <td>1970</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>Wd Sdng</td>\n",
       "      <td>Wd Shng</td>\n",
       "      <td>None</td>\n",
       "      <td>0.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>BrkTil</td>\n",
       "      <td>TA</td>\n",
       "      <td>Gd</td>\n",
       "      <td>No</td>\n",
       "      <td>ALQ</td>\n",
       "      <td>216</td>\n",
       "      <td>Unf</td>\n",
       "      <td>0</td>\n",
       "      <td>540</td>\n",
       "      <td>756</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Gd</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>961</td>\n",
       "      <td>756</td>\n",
       "      <td>0</td>\n",
       "      <td>1717</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>7</td>\n",
       "      <td>Typ</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>Detchd</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>Unf</td>\n",
       "      <td>3</td>\n",
       "      <td>642</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>272</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2006</td>\n",
       "      <td>WD</td>\n",
       "      <td>Abnorml</td>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>84.0</td>\n",
       "      <td>14260</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>FR2</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>NoRidge</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>2Story</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>2000</td>\n",
       "      <td>2000</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>350.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>PConc</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>Av</td>\n",
       "      <td>GLQ</td>\n",
       "      <td>655</td>\n",
       "      <td>Unf</td>\n",
       "      <td>0</td>\n",
       "      <td>490</td>\n",
       "      <td>1145</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Ex</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>1145</td>\n",
       "      <td>1053</td>\n",
       "      <td>0</td>\n",
       "      <td>2198</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>9</td>\n",
       "      <td>Typ</td>\n",
       "      <td>1</td>\n",
       "      <td>TA</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>RFn</td>\n",
       "      <td>3</td>\n",
       "      <td>836</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>192</td>\n",
       "      <td>84</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  MSSubClass MSZoning  LotFrontage  LotArea Street Alley LotShape  \\\n",
       "0   1          60       RL         65.0     8450   Pave   NaN      Reg   \n",
       "1   2          20       RL         80.0     9600   Pave   NaN      Reg   \n",
       "2   3          60       RL         68.0    11250   Pave   NaN      IR1   \n",
       "3   4          70       RL         60.0     9550   Pave   NaN      IR1   \n",
       "4   5          60       RL         84.0    14260   Pave   NaN      IR1   \n",
       "\n",
       "  LandContour Utilities LotConfig LandSlope Neighborhood Condition1  \\\n",
       "0         Lvl    AllPub    Inside       Gtl      CollgCr       Norm   \n",
       "1         Lvl    AllPub       FR2       Gtl      Veenker      Feedr   \n",
       "2         Lvl    AllPub    Inside       Gtl      CollgCr       Norm   \n",
       "3         Lvl    AllPub    Corner       Gtl      Crawfor       Norm   \n",
       "4         Lvl    AllPub       FR2       Gtl      NoRidge       Norm   \n",
       "\n",
       "  Condition2 BldgType HouseStyle  OverallQual  OverallCond  YearBuilt  \\\n",
       "0       Norm     1Fam     2Story            7            5       2003   \n",
       "1       Norm     1Fam     1Story            6            8       1976   \n",
       "2       Norm     1Fam     2Story            7            5       2001   \n",
       "3       Norm     1Fam     2Story            7            5       1915   \n",
       "4       Norm     1Fam     2Story            8            5       2000   \n",
       "\n",
       "   YearRemodAdd RoofStyle RoofMatl Exterior1st Exterior2nd MasVnrType  \\\n",
       "0          2003     Gable  CompShg     VinylSd     VinylSd    BrkFace   \n",
       "1          1976     Gable  CompShg     MetalSd     MetalSd       None   \n",
       "2          2002     Gable  CompShg     VinylSd     VinylSd    BrkFace   \n",
       "3          1970     Gable  CompShg     Wd Sdng     Wd Shng       None   \n",
       "4          2000     Gable  CompShg     VinylSd     VinylSd    BrkFace   \n",
       "\n",
       "   MasVnrArea ExterQual ExterCond Foundation BsmtQual BsmtCond BsmtExposure  \\\n",
       "0       196.0        Gd        TA      PConc       Gd       TA           No   \n",
       "1         0.0        TA        TA     CBlock       Gd       TA           Gd   \n",
       "2       162.0        Gd        TA      PConc       Gd       TA           Mn   \n",
       "3         0.0        TA        TA     BrkTil       TA       Gd           No   \n",
       "4       350.0        Gd        TA      PConc       Gd       TA           Av   \n",
       "\n",
       "  BsmtFinType1  BsmtFinSF1 BsmtFinType2  BsmtFinSF2  BsmtUnfSF  TotalBsmtSF  \\\n",
       "0          GLQ         706          Unf           0        150          856   \n",
       "1          ALQ         978          Unf           0        284         1262   \n",
       "2          GLQ         486          Unf           0        434          920   \n",
       "3          ALQ         216          Unf           0        540          756   \n",
       "4          GLQ         655          Unf           0        490         1145   \n",
       "\n",
       "  Heating HeatingQC CentralAir Electrical  1stFlrSF  2ndFlrSF  LowQualFinSF  \\\n",
       "0    GasA        Ex          Y      SBrkr       856       854             0   \n",
       "1    GasA        Ex          Y      SBrkr      1262         0             0   \n",
       "2    GasA        Ex          Y      SBrkr       920       866             0   \n",
       "3    GasA        Gd          Y      SBrkr       961       756             0   \n",
       "4    GasA        Ex          Y      SBrkr      1145      1053             0   \n",
       "\n",
       "   GrLivArea  BsmtFullBath  BsmtHalfBath  FullBath  HalfBath  BedroomAbvGr  \\\n",
       "0       1710             1             0         2         1             3   \n",
       "1       1262             0             1         2         0             3   \n",
       "2       1786             1             0         2         1             3   \n",
       "3       1717             1             0         1         0             3   \n",
       "4       2198             1             0         2         1             4   \n",
       "\n",
       "   KitchenAbvGr KitchenQual  TotRmsAbvGrd Functional  Fireplaces FireplaceQu  \\\n",
       "0             1          Gd             8        Typ           0         NaN   \n",
       "1             1          TA             6        Typ           1          TA   \n",
       "2             1          Gd             6        Typ           1          TA   \n",
       "3             1          Gd             7        Typ           1          Gd   \n",
       "4             1          Gd             9        Typ           1          TA   \n",
       "\n",
       "  GarageType  GarageYrBlt GarageFinish  GarageCars  GarageArea GarageQual  \\\n",
       "0     Attchd       2003.0          RFn           2         548         TA   \n",
       "1     Attchd       1976.0          RFn           2         460         TA   \n",
       "2     Attchd       2001.0          RFn           2         608         TA   \n",
       "3     Detchd       1998.0          Unf           3         642         TA   \n",
       "4     Attchd       2000.0          RFn           3         836         TA   \n",
       "\n",
       "  GarageCond PavedDrive  WoodDeckSF  OpenPorchSF  EnclosedPorch  3SsnPorch  \\\n",
       "0         TA          Y           0           61              0          0   \n",
       "1         TA          Y         298            0              0          0   \n",
       "2         TA          Y           0           42              0          0   \n",
       "3         TA          Y           0           35            272          0   \n",
       "4         TA          Y         192           84              0          0   \n",
       "\n",
       "   ScreenPorch  PoolArea PoolQC Fence MiscFeature  MiscVal  MoSold  YrSold  \\\n",
       "0            0         0    NaN   NaN         NaN        0       2    2008   \n",
       "1            0         0    NaN   NaN         NaN        0       5    2007   \n",
       "2            0         0    NaN   NaN         NaN        0       9    2008   \n",
       "3            0         0    NaN   NaN         NaN        0       2    2006   \n",
       "4            0         0    NaN   NaN         NaN        0      12    2008   \n",
       "\n",
       "  SaleType SaleCondition  SalePrice  \n",
       "0       WD        Normal     208500  \n",
       "1       WD        Normal     181500  \n",
       "2       WD        Normal     223500  \n",
       "3       WD       Abnorml     140000  \n",
       "4       WD        Normal     250000  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's load the house price dataset from Kaggle\n",
    "\n",
    "data = pd.read_csv('houseprice.csv')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rare value imputation - important\n",
    "\n",
    "The identification of rare labels should be done considering only the presence of rare labels in the training set, and then propagated to the test set. This means, rare labels should be identified in the training set. And then, when those are present in the test set as well, they should be replaced, regardless of whether in the test set they are rare or not (i.e., regardless of whether in the test set they are also present in a tiny percentage of the observations or in a high percentage of observations)\n",
    "\n",
    "In addition, there may be in the test set labels that were not present in the train set. They should be considered rare and preprocessed using the method that was selected to replace rare labels in the training set.\n",
    "\n",
    "For example, let's imagine that we have in the training set the variable 'city' with the labels 'London', 'Manchester' and 'Yorkshire'. 'Yorkshire' is present in less than 5% of the observations so we decide to replace it by 'London', the most frequent city in the training dataset. In the test set, we should also replace 'Yorkshire' by 'London', regardless of the percentage of observations for 'Yorkshire' or whether 'London' is still the most represented city in the test set.\n",
    "\n",
    "In addition, if in the test set we find the category 'Milton Keynes', that was not present in the training set, we should also replace that category by London. This is, all categories present in test set, not present in training set, should be treated as rare values and imputed accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1022, 81), (438, 81))"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's go ahead and divide dataset into train and test set\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(data, data.SalePrice, test_size=0.3,\n",
    "                                                    random_state=0)\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Functions\n",
    "\n",
    "Below I will write a few functions to convert categories in categorical variables into numbers so we can use them in sklearn and then to quickly test these variables in a random forest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train_rf(X_train, y_train, X_test, y_test, columns):\n",
    "    # function to train the random forest\n",
    "    # and test it on train and test sets\n",
    "    \n",
    "    rf = RandomForestRegressor(n_estimators=800, random_state=39)\n",
    "    \n",
    "    if type(columns)==str: # if we train using only 1 variable (pass a string instead of list in the \"columns\" argument of the function)\n",
    "        rf.fit(X_train[columns].to_frame(), y_train.values)\n",
    "        pred_train = rf.predict(X_train[columns].to_frame())\n",
    "        pred_test = rf.predict(X_test[columns].to_frame())\n",
    "        \n",
    "    else: # if we train using multiple variables (pass a list in the argument \"columns\")\n",
    "        rf.fit(X_train[columns], y_train.values)\n",
    "        pred_train = rf.predict(X_train[columns])\n",
    "        pred_test = rf.predict(X_test[columns])\n",
    "        \n",
    "    print('Train set')\n",
    "    print('Random Forests mse: {}'.format(mean_squared_error(y_train, pred_train)))\n",
    "    print('Test set')\n",
    "    print('Random Forests mse: {}'.format(mean_squared_error(y_test, pred_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def labels_to_numbers(X_train, X_test, columns):\n",
    "    # function to encode labels into numbers\n",
    "    # each label will be assigned an ordinal number from 0 onwards\n",
    "    \n",
    "    for col in columns:\n",
    "        labels_dict = {k:i for i, k in enumerate(X_train[col].unique(), 0)}\n",
    "        X_train.loc[:, col] = X_train.loc[:, col].map(labels_dict )\n",
    "        X_test.loc[:, col] = X_test.loc[:, col].map(labels_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variables with high cardinality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Neighborhood\n",
      "Blmngtn    0.011644\n",
      "Blueste    0.001370\n",
      "BrDale     0.010959\n",
      "BrkSide    0.039726\n",
      "ClearCr    0.019178\n",
      "CollgCr    0.102740\n",
      "Crawfor    0.034932\n",
      "Edwards    0.068493\n",
      "Gilbert    0.054110\n",
      "IDOTRR     0.025342\n",
      "MeadowV    0.011644\n",
      "Mitchel    0.033562\n",
      "NAmes      0.154110\n",
      "NPkVill    0.006164\n",
      "NWAmes     0.050000\n",
      "NoRidge    0.028082\n",
      "NridgHt    0.052740\n",
      "OldTown    0.077397\n",
      "SWISU      0.017123\n",
      "Sawyer     0.050685\n",
      "SawyerW    0.040411\n",
      "Somerst    0.058904\n",
      "StoneBr    0.017123\n",
      "Timber     0.026027\n",
      "Veenker    0.007534\n",
      "Name: Neighborhood, dtype: float64\n",
      "\n",
      "Exterior1st\n",
      "AsbShng    0.013699\n",
      "AsphShn    0.000685\n",
      "BrkComm    0.001370\n",
      "BrkFace    0.034247\n",
      "CBlock     0.000685\n",
      "CemntBd    0.041781\n",
      "HdBoard    0.152055\n",
      "ImStucc    0.000685\n",
      "MetalSd    0.150685\n",
      "Plywood    0.073973\n",
      "Stone      0.001370\n",
      "Stucco     0.017123\n",
      "VinylSd    0.352740\n",
      "Wd Sdng    0.141096\n",
      "WdShing    0.017808\n",
      "Name: Exterior1st, dtype: float64\n",
      "\n",
      "Exterior2nd\n",
      "AsbShng    0.013699\n",
      "AsphShn    0.002055\n",
      "Brk Cmn    0.004795\n",
      "BrkFace    0.017123\n",
      "CBlock     0.000685\n",
      "CmentBd    0.041096\n",
      "HdBoard    0.141781\n",
      "ImStucc    0.006849\n",
      "MetalSd    0.146575\n",
      "Other      0.000685\n",
      "Plywood    0.097260\n",
      "Stone      0.003425\n",
      "Stucco     0.017808\n",
      "VinylSd    0.345205\n",
      "Wd Sdng    0.134932\n",
      "Wd Shng    0.026027\n",
      "Name: Exterior2nd, dtype: float64\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# let's explore examples in which variables have several categories, say more than 10\n",
    "# let's add highly cardinal variables into a list\n",
    "\n",
    "multi_cat_cols = []\n",
    "for col in data.columns:\n",
    "    if data[col].dtypes =='O': # if variable  is categorical\n",
    "        if len(data[col].unique())>10: # and has more than 10 categories\n",
    "            multi_cat_cols.append(col)  # add to the list\n",
    "            print(data.groupby(col)[col].count()/np.float(len(data))) # and print the percentage of observations within each category\n",
    "            print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Neighborhood', 'Exterior1st', 'Exterior2nd']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's inspect our highly cardinal variable names\n",
    "multi_cat_cols"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the above frequency distributions we observe that for each of the three variables, there are many categories that are rare."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# for comparison, I will replace rare values by both the most frequent category\n",
    "# or by re-categorising them under a new label \"Rare\"\n",
    "# I will create a function to make the 2 rare value imputations at once\n",
    "\n",
    "def rare_imputation(X_train, X_test, variable):\n",
    "    \n",
    "    # find the most frequent category\n",
    "    frequent_cat = X_train.groupby(variable)[variable].count().sort_values().tail(1).index.values[0]\n",
    "    \n",
    "    # find rare labels\n",
    "    temp = X_train.groupby([variable])[variable].count()/np.float(len(X_train))\n",
    "    rare_cat = [x for x in temp.loc[temp<0.05].index.values]\n",
    "    \n",
    "    # create new variables, with Rare labels imputed\n",
    "    \n",
    "    # by the most frequent category\n",
    "    X_train[variable+'_freq_imp'] = np.where(X_train[variable].isin(rare_cat), frequent_cat, X_train[variable])\n",
    "    X_test[variable+'_freq_imp'] = np.where(X_test[variable].isin(rare_cat), frequent_cat, X_test[variable])\n",
    "    \n",
    "    # by adding a new label 'Rare'\n",
    "    X_train[variable+'_rare_imp'] = np.where(X_train[variable].isin(rare_cat), 'Rare', X_train[variable])\n",
    "    X_test[variable+'_rare_imp'] = np.where(X_test[variable].isin(rare_cat), 'Rare', X_test[variable])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# let's go ahead and impute rare categories\n",
    "\n",
    "for col in multi_cat_cols:\n",
    "    rare_imputation(X_train, X_test, col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Neighborhood\n",
       "Blmngtn    0.011742\n",
       "Blueste    0.001957\n",
       "BrDale     0.009785\n",
       "BrkSide    0.040117\n",
       "ClearCr    0.023483\n",
       "CollgCr    0.102740\n",
       "Crawfor    0.034247\n",
       "Edwards    0.069472\n",
       "Gilbert    0.053816\n",
       "IDOTRR     0.023483\n",
       "MeadowV    0.011742\n",
       "Mitchel    0.035225\n",
       "NAmes      0.147750\n",
       "NPkVill    0.006849\n",
       "NWAmes     0.049902\n",
       "NoRidge    0.029354\n",
       "NridgHt    0.049902\n",
       "OldTown    0.071429\n",
       "SWISU      0.017613\n",
       "Sawyer     0.059687\n",
       "SawyerW    0.044031\n",
       "Somerst    0.054795\n",
       "StoneBr    0.015656\n",
       "Timber     0.029354\n",
       "Veenker    0.005871\n",
       "Name: Neighborhood, dtype: float64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's inspect the original distribution for the variable Neighborhood\n",
    "X_train.groupby('Neighborhood')['Neighborhood'].count()/np.float(len(X_train))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Neighborhood_rare_imp\n",
       "CollgCr    0.071918\n",
       "Edwards    0.048630\n",
       "Gilbert    0.037671\n",
       "NAmes      0.103425\n",
       "OldTown    0.050000\n",
       "Rare       0.308219\n",
       "Sawyer     0.041781\n",
       "Somerst    0.038356\n",
       "Name: Neighborhood_rare_imp, dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# and now the modified distribution after rare imputation into rare category\n",
    "X_train.groupby('Neighborhood_rare_imp')['Neighborhood_rare_imp'].count()/np.float(len(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the number of different labels has decreased substantially."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Neighborhood_freq_imp\n",
       "CollgCr    0.071918\n",
       "Edwards    0.048630\n",
       "Gilbert    0.037671\n",
       "NAmes      0.411644\n",
       "OldTown    0.050000\n",
       "Sawyer     0.041781\n",
       "Somerst    0.038356\n",
       "Name: Neighborhood_freq_imp, dtype: float64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's inspect the modified distribution after rare imputation into most frequent category\n",
    "\n",
    "X_train.groupby('Neighborhood_freq_imp')['Neighborhood_freq_imp'].count()/np.float(len(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, imputation reduced the number of labels dramatically for this variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Neighborhood_rare_imp', 'Exterior1st_rare_imp', 'Exterior2nd_rare_imp']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's create different variable lists for training random forests with the different imputation methods\n",
    "\n",
    "cols_freq = [x+'_freq_imp' for x in multi_cat_cols]\n",
    "cols_rare = [x+'_rare_imp' for x in multi_cat_cols]\n",
    "\n",
    "cols_rare"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Random Forests mse: 1984889003.84204\n",
      "Test set\n",
      "Random Forests mse: 3198856428.390624\n"
     ]
    }
   ],
   "source": [
    "# model built on data with infrequent categories\n",
    "# # let's first encode the categories into numbers\n",
    "labels_to_numbers(X_train, X_test, multi_cat_cols)\n",
    "\n",
    "# and now train random forests with the original variables\n",
    "train_rf(X_train, y_train, X_test, y_test, multi_cat_cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Random Forests mse: 4185463008.435918\n",
      "Test set\n",
      "Random Forests mse: 5288100968.253248\n"
     ]
    }
   ],
   "source": [
    "# or with the rare into rare imputation variables\n",
    "labels_to_numbers(X_train, X_test, cols_rare)\n",
    "train_rf(X_train, y_train, X_test, y_test, cols_rare)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Random Forests mse: 4743496587.188402\n",
      "Test set\n",
      "Random Forests mse: 5582333261.023977\n"
     ]
    }
   ],
   "source": [
    "# or with the rare into frequent imputation\n",
    "labels_to_numbers(X_train, X_test, cols_freq)\n",
    "train_rf(X_train, y_train, X_test, y_test, cols_freq)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the house price predictions, we see that actually, replacing those rare values by re-categorisation does not improve the performance of random forests, on the contrary, it affects it quite dramatically (the mse are higher). This indicates that actually, those rarities, those infrequent labels, have quite a dramatic impact on price."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**That is all for this demonstration. I hope you enjoyed the notebook, and see you in the next one.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}