03.3_Labels.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Labels\n",
    "\n",
    "The values of a categorical variable are selected from a group of categories, also called labels. Thus, in the variable _gender_ the categories or labels are male and female, whereas in the variable _city_ the labels can be London, Manchester, Brighton and so on.\n",
    "\n",
    "We can see from the above examples, that different categorical variables can contain different amount of labels (or categories). The variable gender contains only 2 labels, but a variable like city or postcode, can contain a huge number of different labels.\n",
    "\n",
    "The number of labels within a categorical variable is known as cardinality. A high number of labels within a variable is known as __high cardinality__.\n",
    "\n",
    "### Are multiple labels in a categorical variable a problem?\n",
    "\n",
    "If highly cardinal categorical variables are used to train machine learning models, the following problems may appear: \n",
    "\n",
    "- Variables with too many labels tend to dominate over those with only a few labels, particularly in **Tree based** algorithms.\n",
    "\n",
    "- A big number of labels within a variable may introduce noise with little if any information, therefore making the machine learning models prone to over-fit.\n",
    "\n",
    "- Some of the labels may only be present in the training data set, but not in the test set, therefore causing the machine learning algorithms to over-fit the training set.\n",
    "\n",
    "- Contrarily, new labels may appear in the test set that were not present in the training set, therefore leaving the machine learning algorithm unable to perform a calculation over the new observation.\n",
    "\n",
    "\n",
    "In particular, **tree methods are biased towards variables with lots of labels**, so their performance may be affected by these type of variables.\n",
    "\n",
    "Below, I will show the effect of high cardinality of variables on the performance of different machine learning algorithms, and how a quick fix to reduce the number of labels, without any sort of data insight, already helps to boost performance.\n",
    "\n",
    "\n",
    "\n",
    "## Real Life example: \n",
    "\n",
    "### Predicting Survival on the Titanic: understanding society behaviour and beliefs\n",
    "\n",
    "Perhaps one of the most infamous shipwrecks in history, the Titanic sank after colliding with an iceberg, killing 1502 out of 2224 people on board. Interestingly, by analysing the probability of survival based on few attributes like gender, age, and social status, we can make very accurate predictions on which passengers would survive. Some groups of people were more likely to survive than others, such as women, children, and the upper-class. Therefore, we can learn about the society priorities and privileges at the time.\n",
    "\n",
    "====================================================================================================\n",
    "\n",
    "To download the Titanic data, go ahead to this [website](https://www.kaggle.com/c/titanic/data)\n",
    "\n",
    "Click on the link 'train.csv', and then click the 'download' blue button towards the right of the screen, to download the dataset. Save it in a folder of your choice.\n",
    "\n",
    "**Note that you need to be logged in to Kaggle in order to download the datasets**.\n",
    "\n",
    "If you save it in the same directory from which you are running this notebook, and you rename the file to 'titanic.csv' then you can load it the same way I will load it below.\n",
    "\n",
    "===================================================================================================="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "% matplotlib inline\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "4                           Allen, Mr. William Henry    male  35.0      0   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  \n",
       "0      0         A/5 21171   7.2500   NaN        S  \n",
       "1      0          PC 17599  71.2833   C85        C  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3      0            113803  53.1000  C123        S  \n",
       "4      0            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's load the titanic dataset\n",
    "\n",
    "data = pd.read_csv('titanic.csv')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The categorical variables in this dataset are Name, Sex, Ticket, Cabin and Embarked."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of categories in the variable Name: 891\n",
      "Number of categories in the variable Gender: 2\n",
      "Number of categories in the variable Ticket: 681\n",
      "Number of categories in the variable Cabin: 148\n",
      "Number of categories in the variable Embarked: 4\n",
      "Total number of passengers in the Titanic: 891\n"
     ]
    }
   ],
   "source": [
    "# let's inspect at the number of labels for the different categorical variables\n",
    "\n",
    "print('Number of categories in the variable Name: {}'.format(\n",
    "    len(data.Name.unique())))\n",
    "\n",
    "print('Number of categories in the variable Gender: {}'.format(\n",
    "    len(data.Sex.unique())))\n",
    "\n",
    "print('Number of categories in the variable Ticket: {}'.format(\n",
    "    len(data.Ticket.unique())))\n",
    "\n",
    "print('Number of categories in the variable Cabin: {}'.format(\n",
    "    len(data.Cabin.unique())))\n",
    "\n",
    "print('Number of categories in the variable Embarked: {}'.format(\n",
    "    len(data.Embarked.unique())))\n",
    "\n",
    "print('Total number of passengers in the Titanic: {}'.format(len(data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the variable Sex contains only 2 categories and Embarked 4, the variables Ticket and Cabin, as expected, contain a huge number of different labels. They show high cardinality. And so does the variable Name (different for each passenger).\n",
    "\n",
    "To demonstrate the effect of high cardinality in machine learning performance, I will work with the variable Cabin."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([nan, 'C85', 'C123', 'E46', 'G6', 'C103', 'D56', 'A6', 'C23 C25 C27',\n",
       "       'B78', 'D33', 'B30', 'C52', 'B28', 'C83', 'F33', 'F G73', 'E31',\n",
       "       'A5', 'D10 D12', 'D26', 'C110', 'B58 B60', 'E101', 'F E69', 'D47',\n",
       "       'B86', 'F2', 'C2', 'E33', 'B19', 'A7', 'C49', 'F4', 'A32', 'B4',\n",
       "       'B80', 'A31', 'D36', 'D15', 'C93', 'C78', 'D35', 'C87', 'B77',\n",
       "       'E67', 'B94', 'C125', 'C99', 'C118', 'D7', 'A19', 'B49', 'D',\n",
       "       'C22 C26', 'C106', 'C65', 'E36', 'C54', 'B57 B59 B63 B66', 'C7',\n",
       "       'E34', 'C32', 'B18', 'C124', 'C91', 'E40', 'T', 'C128', 'D37',\n",
       "       'B35', 'E50', 'C82', 'B96 B98', 'E10', 'E44', 'A34', 'C104', 'C111',\n",
       "       'C92', 'E38', 'D21', 'E12', 'E63', 'A14', 'B37', 'C30', 'D20',\n",
       "       'B79', 'E25', 'D46', 'B73', 'C95', 'B38', 'B39', 'B22', 'C86',\n",
       "       'C70', 'A16', 'C101', 'C68', 'A10', 'E68', 'B41', 'A20', 'D19',\n",
       "       'D50', 'D9', 'A23', 'B50', 'A26', 'D48', 'E58', 'C126', 'B71',\n",
       "       'B51 B53 B55', 'D49', 'B5', 'B20', 'F G63', 'C62 C64', 'E24', 'C90',\n",
       "       'C45', 'E8', 'B101', 'D45', 'C46', 'D30', 'E121', 'D11', 'E77',\n",
       "       'F38', 'B3', 'D6', 'B82 B84', 'D17', 'A36', 'B102', 'B69', 'E49',\n",
       "       'C47', 'D28', 'E17', 'A24', 'C50', 'B42', 'C148'], dtype=object)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's explore the values / categories of Cabin\n",
    "\n",
    "# we know from the previous cell that there are 148 different cabins\n",
    "# therefore the variable is highly cardinal\n",
    "\n",
    "data.Cabin.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now reduce the cardinality of the variable. How? instead of using the entire Cabin value, I will capture only the \n",
    "first letter.\n",
    "\n",
    "***Rationale***: the first letter indicates the deck on which the cabin was located, and is therefore an indication of both social class status and proximity to the surface of the Titanic. Both are known to help or improve the probability of survival."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Cabin_reduced</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C123</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Cabin Cabin_reduced\n",
       "0   NaN             n\n",
       "1   C85             C\n",
       "2   NaN             n\n",
       "3  C123             C\n",
       "4   NaN             n"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's capture the first letter\n",
    "data['Cabin_reduced'] = data['Cabin'].astype(str).str[0]\n",
    "\n",
    "data[['Cabin', 'Cabin_reduced']].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of categories in the variable Cabin: 148\n",
      "Number of categories in the variable Cabin reduced: 9\n"
     ]
    }
   ],
   "source": [
    "print('Number of categories in the variable Cabin: {}'.format(\n",
    "    len(data.Cabin.unique())))\n",
    "\n",
    "print('Number of categories in the variable Cabin reduced: {}'.format(\n",
    "    len(data.Cabin_reduced.unique())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We reduced the number of different labels from 148 to 9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((623, 3), (268, 3))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's separate into training and testing set\n",
    "# in order to build machine learning models\n",
    "\n",
    "use_cols = ['Cabin', 'Cabin_reduced', 'Sex']\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    data[use_cols], \n",
    "    data.Survived,  \n",
    "    test_size=0.3,\n",
    "    random_state=0)\n",
    "\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### High cardinality leads to uneven distribution of categories in train and test sets\n",
    "\n",
    "When a variable is highly cardinal, often some categories land only on the training set, or only on the testing set. If present only in the training set, they may lead to overfitting. If present only on the testing set, the machine learning algorithm will not know how to handle them, as it has not seen it during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's find out labels present only in the training set\n",
    "\n",
    "unique_to_train_set = [\n",
    "    x for x in X_train.Cabin.unique() if x not in X_test.Cabin.unique()\n",
    "]\n",
    "\n",
    "len(unique_to_train_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 100 Cabins only present in the training set, and not in the testing set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's find out labels present only in the test set\n",
    "\n",
    "unique_to_test_set = [\n",
    "    x for x in X_test.Cabin.unique() if x not in X_train.Cabin.unique()\n",
    "]\n",
    "\n",
    "len(unique_to_test_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Variables with high cardinality tend to have values (i.e., categories) present in the training set, that are not present in the test set, and vice versa. This will bring problems at the time of training (due to over-fitting) and scoring of new data (how should the model deal with unseen categories?).\n",
    "\n",
    "In order to evaluate the effect of categorical variables in machine learning models, I will quickly replace the categories by numbers. See below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'E17': 0,\n",
       " 'D33': 1,\n",
       " nan: 2,\n",
       " 'D26': 3,\n",
       " 'B58 B60': 4,\n",
       " 'C128': 5,\n",
       " 'D17': 6,\n",
       " 'A14': 7,\n",
       " 'F33': 8,\n",
       " 'B19': 9,\n",
       " 'D21': 10,\n",
       " 'C148': 11,\n",
       " 'C30': 12,\n",
       " 'D56': 13,\n",
       " 'E24': 14,\n",
       " 'E40': 15,\n",
       " 'E31': 16,\n",
       " 'E44': 17,\n",
       " 'E38': 18,\n",
       " 'D37': 19,\n",
       " 'E8': 20,\n",
       " 'C92': 21,\n",
       " 'E63': 22,\n",
       " 'C125': 23,\n",
       " 'F4': 24,\n",
       " 'E67': 25,\n",
       " 'C126': 26,\n",
       " 'B73': 27,\n",
       " 'E36': 28,\n",
       " 'C78': 29,\n",
       " 'E46': 30,\n",
       " 'C111': 31,\n",
       " 'E101': 32,\n",
       " 'D15': 33,\n",
       " 'E12': 34,\n",
       " 'G6': 35,\n",
       " 'A32': 36,\n",
       " 'B4': 37,\n",
       " 'A10': 38,\n",
       " 'A5': 39,\n",
       " 'C95': 40,\n",
       " 'E25': 41,\n",
       " 'C90': 42,\n",
       " 'D6': 43,\n",
       " 'A36': 44,\n",
       " 'D': 45,\n",
       " 'D50': 46,\n",
       " 'B96 B98': 47,\n",
       " 'C93': 48,\n",
       " 'E77': 49,\n",
       " 'C101': 50,\n",
       " 'D11': 51,\n",
       " 'C123': 52,\n",
       " 'C32': 53,\n",
       " 'B35': 54,\n",
       " 'C91': 55,\n",
       " 'T': 56,\n",
       " 'B101': 57,\n",
       " 'E58': 58,\n",
       " 'A23': 59,\n",
       " 'B77': 60,\n",
       " 'D28': 61,\n",
       " 'B82 B84': 62,\n",
       " 'B79': 63,\n",
       " 'C45': 64,\n",
       " 'C2': 65,\n",
       " 'B5': 66,\n",
       " 'C104': 67,\n",
       " 'B20': 68,\n",
       " 'A19': 69,\n",
       " 'B51 B53 B55': 70,\n",
       " 'B80': 71,\n",
       " 'B38': 72,\n",
       " 'B22': 73,\n",
       " 'B18': 74,\n",
       " 'C22 C26': 75,\n",
       " 'A16': 76,\n",
       " 'F2': 77,\n",
       " 'D47': 78,\n",
       " 'E121': 79,\n",
       " 'C23 C25 C27': 80,\n",
       " 'B28': 81,\n",
       " 'E10': 82,\n",
       " 'D36': 83,\n",
       " 'C46': 84,\n",
       " 'B39': 85,\n",
       " 'D30': 86,\n",
       " 'E33': 87,\n",
       " 'C50': 88,\n",
       " 'D20': 89,\n",
       " 'C124': 90,\n",
       " 'A34': 91,\n",
       " 'C110': 92,\n",
       " 'D19': 93,\n",
       " 'B86': 94,\n",
       " 'D35': 95,\n",
       " 'C99': 96,\n",
       " 'D46': 97,\n",
       " 'F38': 98,\n",
       " 'A24': 99,\n",
       " 'D7': 100,\n",
       " 'C65': 101,\n",
       " 'C103': 102,\n",
       " 'A31': 103,\n",
       " 'C82': 104,\n",
       " 'F G63': 105,\n",
       " 'A6': 106,\n",
       " 'E50': 107,\n",
       " 'C68': 108,\n",
       " 'C52': 109,\n",
       " 'B3': 110,\n",
       " 'B49': 111,\n",
       " 'C118': 112,\n",
       " 'C70': 113,\n",
       " 'F E69': 114,\n",
       " 'C86': 115,\n",
       " 'D48': 116,\n",
       " 'C49': 117,\n",
       " 'A7': 118,\n",
       " 'A20': 119,\n",
       " 'E49': 120}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's re-map Cabin into numbers so we can use it to train ML models\n",
    "\n",
    "# I will replace each cabin by a number\n",
    "# This will allow me to quickly demonstrate\n",
    "# the effect of labels on machine learning algorithms\n",
    "\n",
    "cabin_dict = {k:i for i, k in enumerate(X_train.Cabin.unique(), 0)} \n",
    "cabin_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Cabin_mapped</th>\n",
       "      <th>Cabin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>857</th>\n",
       "      <td>0</td>\n",
       "      <td>E17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>1</td>\n",
       "      <td>D33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>386</th>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124</th>\n",
       "      <td>3</td>\n",
       "      <td>D26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>578</th>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>549</th>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>4</td>\n",
       "      <td>B58 B60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>157</th>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Cabin_mapped    Cabin\n",
       "857             0      E17\n",
       "52              1      D33\n",
       "386             2      NaN\n",
       "124             3      D26\n",
       "578             2      NaN\n",
       "549             2      NaN\n",
       "118             4  B58 B60\n",
       "12              2      NaN\n",
       "157             2      NaN\n",
       "127             2      NaN"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# replace the labels in Cabin, using the dic created above\n",
    "X_train.loc[:, 'Cabin_mapped'] = X_train.loc[:, 'Cabin'].map(cabin_dict)\n",
    "X_test.loc[:, 'Cabin_mapped'] = X_test.loc[:, 'Cabin'].map(cabin_dict)\n",
    "\n",
    "X_train[['Cabin_mapped', 'Cabin']].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see how NaN takes the value 2 in the new variable, E17 takes the value 0, D33 takes the value 1, and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Cabin_reduced</th>\n",
       "      <th>Cabin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>857</th>\n",
       "      <td>0</td>\n",
       "      <td>E17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>1</td>\n",
       "      <td>D33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>386</th>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124</th>\n",
       "      <td>1</td>\n",
       "      <td>D26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>578</th>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>549</th>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>3</td>\n",
       "      <td>B58 B60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>157</th>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Cabin_reduced    Cabin\n",
       "857              0      E17\n",
       "52               1      D33\n",
       "386              2      NaN\n",
       "124              1      D26\n",
       "578              2      NaN\n",
       "549              2      NaN\n",
       "118              3  B58 B60\n",
       "12               2      NaN\n",
       "157              2      NaN\n",
       "127              2      NaN"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now I will replace the letters in the reduced cabin variable\n",
    "\n",
    "# create replace dictionary\n",
    "cabin_dict = {k: i for i, k in enumerate(X_train['Cabin_reduced'].unique(), 0)}\n",
    "\n",
    "# replace labels by numbers with dictionary\n",
    "X_train.loc[:, 'Cabin_reduced'] = X_train.loc[:, 'Cabin_reduced'].map(cabin_dict)\n",
    "X_test.loc[:, 'Cabin_reduced'] = X_test.loc[:, 'Cabin_reduced'].map(cabin_dict)\n",
    "\n",
    "X_train[['Cabin_reduced', 'Cabin']].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see now that D33 and D26 take the same number, 1, because we are capturing only the letter. They both start with D."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "857    0\n",
       "52     1\n",
       "386    0\n",
       "124    0\n",
       "578    1\n",
       "Name: Sex, dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# re-map the categorical variable Sex into numbers\n",
    "\n",
    "X_train.loc[:, 'Sex'] = X_train.loc[:, 'Sex'].map({'male': 0, 'female': 1})\n",
    "X_test.loc[:, 'Sex'] = X_test.loc[:, 'Sex'].map({'male': 0, 'female': 1})\n",
    "\n",
    "X_train.Sex.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Cabin_mapped     0\n",
       "Cabin_reduced    0\n",
       "Sex              0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check if there are missing values in these variables\n",
    "\n",
    "X_train[['Cabin_mapped','Cabin_reduced', 'Sex']].isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Cabin_mapped     30\n",
       "Cabin_reduced     0\n",
       "Sex               0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test[['Cabin_mapped','Cabin_reduced', 'Sex']].isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the test set, there are now 30 missing values for the highly cardinal variable. These were introduced when encoding the categories into numbers. How? Many categories exist only in the test set. Thus, when we creating our encoding dictionary using only the train set, we did not generate a number to replace those labels present only in the test set. As a consequence, they were encoded as NaN. We will see in future lectures how to tackle this problem. For now, I will fill those missing values with 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(121, 9)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's check the number of categories in the encoded variables\n",
    "len(X_train.Cabin_mapped.unique()), len(X_train.Cabin_reduced.unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see how we reduced the number of different categories from 148 to just 9 in our previous step. Let's go ahead and evaluate the effect of labels in machine learning algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Forests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Random Forests roc-auc: 0.8617329342096701\n",
      "Test set\n",
      "Random Forests roc-auc: 0.8078571428571428\n"
     ]
    }
   ],
   "source": [
    "# model built on data with high cardinality for cabin\n",
    "\n",
    "# call the model\n",
    "rf = RandomForestClassifier(n_estimators=200, random_state=39)\n",
    "\n",
    "# train the model\n",
    "rf.fit(X_train[['Cabin_mapped', 'Sex']], y_train)\n",
    "\n",
    "# make predictions on train and test set\n",
    "pred_train = rf.predict_proba(X_train[['Cabin_mapped', 'Sex']])\n",
    "pred_test = rf.predict_proba(X_test[['Cabin_mapped', 'Sex']].fillna(0))\n",
    "\n",
    "print('Train set')\n",
    "print('Random Forests roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))\n",
    "print('Test set')\n",
    "print('Random Forests roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We observe that the performance of the Random Forests on the training set is quite superior to its performance in the test set. This indicates that the model is over-fitting, which means that it does a great job at predicting the outcome on the dataset it was trained on, but it lacks the power to generalise the prediction to unseen data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Random Forests roc-auc: 0.8199550985878832\n",
      "Test set\n",
      "Random Forests roc-auc: 0.8332142857142857\n"
     ]
    }
   ],
   "source": [
    "# model built on data with low cardinality for cabin\n",
    "\n",
    "# call the model\n",
    "rf = RandomForestClassifier(n_estimators=200, random_state=39)\n",
    "\n",
    "# train the model\n",
    "rf.fit(X_train[['Cabin_reduced', 'Sex']], y_train)\n",
    "\n",
    "# make predictions on train and test set\n",
    "pred_train = rf.predict_proba(X_train[['Cabin_reduced', 'Sex']])\n",
    "pred_test = rf.predict_proba(X_test[['Cabin_reduced', 'Sex']])\n",
    "\n",
    "print('Train set')\n",
    "print('Random Forests roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))\n",
    "print('Test set')\n",
    "print('Random Forests roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see now that the Random Forests no longer over-fit to the training set. In addition, the model is much better at generalising the predictions (compare the roc-auc of this model on the test set vs the roc-auc of the model above also in the test set: 0.83 vs 0.80)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### AdaBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Adaboost roc-auc: 0.8399546647578143\n",
      "Test set\n",
      "Adaboost roc-auc: 0.809375\n"
     ]
    }
   ],
   "source": [
    "# model build on data with plenty of categories in Cabin variable\n",
    "\n",
    "# call the model\n",
    "ada = AdaBoostClassifier(n_estimators=200, random_state=44)\n",
    "\n",
    "# train the model\n",
    "ada.fit(X_train[['Cabin_mapped', 'Sex']], y_train)\n",
    "\n",
    "# make predictions on train and test set\n",
    "pred_train = ada.predict_proba(X_train[['Cabin_mapped', 'Sex']])\n",
    "pred_test = ada.predict_proba(X_test[['Cabin_mapped', 'Sex']].fillna(0))\n",
    "\n",
    "print('Train set')\n",
    "print('Adaboost roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))\n",
    "print('Test set')\n",
    "print('Adaboost roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Adaboost roc-auc: 0.8195863430294353\n",
      "Test set\n",
      "Adaboost roc-auc: 0.8332142857142857\n"
     ]
    }
   ],
   "source": [
    "# model build on data with fewer categories in Cabin Variable\n",
    "\n",
    "# call the model\n",
    "ada = AdaBoostClassifier(n_estimators=200, random_state=44)\n",
    "\n",
    "# train the model\n",
    "ada.fit(X_train[['Cabin_reduced', 'Sex']], y_train)\n",
    "\n",
    "# make predictions on train and test set\n",
    "pred_train = ada.predict_proba(X_train[['Cabin_reduced', 'Sex']])\n",
    "pred_test = ada.predict_proba(X_test[['Cabin_reduced', 'Sex']].fillna(0))\n",
    "\n",
    "print('Train set')\n",
    "print('Adaboost roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))\n",
    "print('Test set')\n",
    "print('Adaboost roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Similarly, the Adaboost model trained on the variable with high cardinality is slightly overfit to the train set. Whereas the Adaboost trained on the low cardinal variable is not overfitting and it does a better job in generalising the predictions.\n",
    "\n",
    "In addition, building an AdaBoost on a model with less categories in Cabin, is a) simpler and b) should a different category in the test set appear, by taking just the front letter of cabin, the ML model will know how to handle it because it was seen during training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Logistic regression roc-auc: 0.8094564109238411\n",
      "Test set\n",
      "Logistic regression roc-auc: 0.759107142857143\n"
     ]
    }
   ],
   "source": [
    "# model build on data with plenty of categories in Cabin variable\n",
    "\n",
    "# call the model\n",
    "logit = LogisticRegression(random_state=44)\n",
    "\n",
    "# train the model\n",
    "logit.fit(X_train[['Cabin_mapped', 'Sex']], y_train)\n",
    "\n",
    "# make predictions on train and test set\n",
    "pred_train = logit.predict_proba(X_train[['Cabin_mapped', 'Sex']])\n",
    "pred_test = logit.predict_proba(X_test[['Cabin_mapped', 'Sex']].fillna(0))\n",
    "\n",
    "print('Train set')\n",
    "print('Logistic regression roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))\n",
    "print('Test set')\n",
    "print('Logistic regression roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Logistic regression roc-auc: 0.7672664367367301\n",
      "Test set\n",
      "Logistic regression roc-auc: 0.7957738095238095\n"
     ]
    }
   ],
   "source": [
    "# model build on data with fewer categories in Cabin Variable\n",
    "\n",
    "# call the model\n",
    "logit = LogisticRegression(random_state=44)\n",
    "\n",
    "# train the model\n",
    "logit.fit(X_train[['Cabin_reduced', 'Sex']], y_train)\n",
    "\n",
    "# make predictions on train and test set\n",
    "pred_train = logit.predict_proba(X_train[['Cabin_reduced', 'Sex']])\n",
    "pred_test = logit.predict_proba(X_test[['Cabin_reduced', 'Sex']].fillna(0))\n",
    "\n",
    "print('Train set')\n",
    "print('Logistic regression roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))\n",
    "print('Test set')\n",
    "print('Logistic regression roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can draw the same conclusion for Logistic Regression. Reducing the cardinality improves the performance and generalisation of the algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient Boosted Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Gradient Boosted Trees roc-auc: 0.8731860480249887\n",
      "Test set\n",
      "Gradient Boosted Trees roc-auc: 0.8168452380952381\n"
     ]
    }
   ],
   "source": [
    "# model build on data with plenty of categories in Cabin variable\n",
    "\n",
    "# call the model\n",
    "gbc = GradientBoostingClassifier(n_estimators=300, random_state=44)\n",
    "\n",
    "# train the model\n",
    "gbc.fit(X_train[['Cabin_mapped', 'Sex']], y_train)\n",
    "\n",
    "# make predictions on train and test set\n",
    "pred_train = gbc.predict_proba(X_train[['Cabin_mapped', 'Sex']])\n",
    "pred_test = gbc.predict_proba(X_test[['Cabin_mapped', 'Sex']].fillna(0))\n",
    "\n",
    "print('Train set')\n",
    "print('Gradient Boosted Trees roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))\n",
    "print('Test set')\n",
    "print('Gradient Boosted Trees roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Gradient Boosted Trees roc-auc: 0.8204756946703976\n",
      "Test set\n",
      "Gradient Boosted Trees roc-auc: 0.8332142857142857\n"
     ]
    }
   ],
   "source": [
    "# model build on data with plenty of categories in Cabin variable\n",
    "\n",
    "# call the model\n",
    "gbc = GradientBoostingClassifier(n_estimators=300, random_state=44)\n",
    "\n",
    "# train the model\n",
    "gbc.fit(X_train[['Cabin_reduced', 'Sex']], y_train)\n",
    "\n",
    "# make predictions on train and test set\n",
    "pred_train = gbc.predict_proba(X_train[['Cabin_reduced', 'Sex']])\n",
    "pred_test = gbc.predict_proba(X_test[['Cabin_reduced', 'Sex']].fillna(0))\n",
    "\n",
    "print('Train set')\n",
    "print('Gradient Boosted Trees roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))\n",
    "print('Test set')\n",
    "print('Gradient Boosted Trees roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradient Boosted trees are indeed over-fitting to the training set in those cases where the variable Cabin has a lot of labels. This was expected as tree methods tend to be biased to variables with plenty of categories.\n",
    "\n",
    "**That is all for this demonstration. I hope you enjoyed the notebook, and see you in the next one.**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}