ch_09/preprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing\n",
    "\n",
    "This notebook will give you a taste of what `scikit-learn` provides for preprocessing data.\n",
    "\n",
    "## Data used\n",
    "We will be using the planets data and red wine data:\n",
    "\n",
    "### Data License for Planet Data\n",
    "Copyright (C) 2012 Hanno Rein\n",
    "\n",
    "Permission is hereby granted, free of charge, to any person obtaining a copy of this database and associated scripts (the \"Database\"), to deal in the Database without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Database, and to permit persons to whom the Database is furnished to do so, subject to the following conditions:\n",
    "\n",
    "The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Database. A reference to the Database shall be included in all scientific publications that make use of the Database.\n",
    "\n",
    "THE DATABASE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE DATABASE OR THE USE OR OTHER DEALINGS IN THE DATABASE.\n",
    "\n",
    "### Citations for Red Wine Data\n",
    "P. Cortez, A. Cerdeira, F. Almeida, T. Matos and J. Reis. \n",
    "Modeling wine preferences by data mining from physicochemical properties.\n",
    "In Decision Support Systems, Elsevier, 47(4):547-553. ISSN: 0167-9236.\n",
    "\n",
    "Available at:\n",
    "- [@Elsevier](http://dx.doi.org/10.1016/j.dss.2009.05.016)\n",
    "- [Pre-press (pdf)](http://www3.dsi.uminho.pt/pcortez/winequality09.pdf)\n",
    "- [bib](http://www3.dsi.uminho.pt/pcortez/dss09.bib)\n",
    "\n",
    "Dua, D. and Karra Taniskidou, E. (2017). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml/index.php](http://archive.ics.uci.edu/ml/index.php). Irvine, CA: University of California, School of Information and Computer Science.\n",
    "\n",
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "planets = pd.read_csv('data/planets.csv')\n",
    "red_wine = pd.read_csv('data/winequality-red.csv')\n",
    "wine = pd.concat([\n",
    "    pd.read_csv('data/winequality-white.csv', sep=';').assign(kind='white'), \n",
    "    red_wine.assign(kind='red')\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train Test Split\n",
    "\n",
    "Rather than having to write something like this every time:\n",
    "\n",
    "```python\n",
    "shuffled = planets.reindex(np.random.permutation(planets.index))\n",
    "train_end_index = int(np.ceil(shuffled.shape[0] * .75))\n",
    "training = shuffled.iloc[:train_end_index,]\n",
    "testing = shuffled.iloc[train_end_index:,]\n",
    "```\n",
    "\n",
    "We can use scikit-learn's `train_test_split()` function to get our training and testing sets. (We will discuss the validation set in chapter 10.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X = planets[['eccentricity', 'semimajoraxis', 'mass']]\n",
    "y = planets.period\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.25, random_state=0\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The original data had this shape:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((4094, 3), (4094,))"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape, y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our training data has this shape:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((3070, 3), (3070,))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape, y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our testing data has this shape:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1024, 3), (1024,))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test.shape, y_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the first 5 entries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>eccentricity</th>\n",
       "      <th>semimajoraxis</th>\n",
       "      <th>mass</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1390</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2837</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3619</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0701</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1867</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1869</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      eccentricity  semimajoraxis  mass\n",
       "1390           NaN            NaN   NaN\n",
       "2837           NaN            NaN   NaN\n",
       "3619           NaN         0.0701   NaN\n",
       "1867           NaN            NaN   NaN\n",
       "1869           NaN            NaN   NaN"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our y data will be for the same rows as our X:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1390     1.434742\n",
       "2837    51.079263\n",
       "3619     7.171000\n",
       "1867    51.111024\n",
       "1869    62.869161\n",
       "Name: period, dtype: float64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scaling data\n",
    "### Standardizing with `StandardScaler`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-5.43618156e-02,  1.43278593e+00,  1.95196592e+00,  4.51498477e-03,\n",
       "       -1.96265630e-01,  7.79591646e-02, -4.74717586e-02, -3.12856028e-01,\n",
       "       -4.18101448e-01, -5.47587283e-02, -2.46399501e-01,  1.65946487e+00,\n",
       "       -8.59044215e-01, -5.47511116e-02, -4.04573808e-01,  1.88194856e-01,\n",
       "       -5.41905011e-02, -4.75421907e-01,  1.33077010e-01, -3.01831439e-02,\n",
       "       -1.08822831e-01,  1.62409605e-01,  1.21526007e+01,  1.73149454e+00,\n",
       "       -2.25664815e-02,  9.91013258e-01, -7.48808523e-01, -4.99260165e-02,\n",
       "       -8.59044215e-01, -5.49264158e-02])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "standardized = StandardScaler().fit_transform(X_train)\n",
    "\n",
    "# examine some of the non-NaN values\n",
    "standardized[~np.isnan(standardized)][:30]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalizing with `MinMaxScaler`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2.28055906e-05, 1.24474091e-01, 5.33472803e-01, 1.71374569e-03,\n",
       "       1.83543340e-02, 1.77824268e-01, 2.20687839e-04, 1.07593965e-02,\n",
       "       8.36820084e-02, 1.14062675e-05, 1.50885109e-02, 1.39240422e-01,\n",
       "       0.00000000e+00, 1.16250178e-05, 4.78471239e-03, 1.98744770e-01,\n",
       "       2.77257370e-05, 1.69522066e-04, 1.88284519e-01, 7.17216379e-04,\n",
       "       2.40505371e-02, 6.24847487e-03, 8.22784793e-01, 4.91631799e-01,\n",
       "       9.35966714e-04, 9.56961137e-02, 2.09205021e-02, 1.50201619e-04,\n",
       "       0.00000000e+00, 6.59028789e-06])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "normalized = MinMaxScaler().fit_transform(X_train)\n",
    "\n",
    "# examine some of the non-NaN values\n",
    "normalized[~np.isnan(normalized)][:30]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the Median and IQR with `RobustScaler`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-5.64660112e-02,  3.90058747e+00,  1.87999087e+00,  4.32924097e+00,\n",
       "        2.79222462e-01,  3.28542094e-01,  4.56771306e-01,  2.00431965e-02,\n",
       "       -8.21355236e-02, -8.60318668e-02,  1.67775378e-01,  4.40449244e+00,\n",
       "       -4.47182295e-01, -8.54645050e-02, -1.83844492e-01,  4.19803787e-01,\n",
       "       -4.37048728e-02, -3.41339093e-01,  3.74172941e-01,  1.74459261e+00,\n",
       "        4.73606911e-01,  1.60907419e+01,  2.77306263e+01,  1.69746749e+00,\n",
       "        2.31195445e+00,  2.91853132e+00, -3.55920602e-01,  2.73954715e-01,\n",
       "       -4.47182295e-01, -9.85228329e-02])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import RobustScaler\n",
    "\n",
    "robust_scaled = RobustScaler().fit_transform(X_train)\n",
    "\n",
    "# examine some of the non-NaN values\n",
    "robust_scaled[~np.isnan(robust_scaled)][:30]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoding\n",
    "### Binary encoding with `np.where()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, ..., 1, 1, 1])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.where(wine.kind == 'red', 1, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the `LabelBinarizer` class from scikit-learn. By calling the `inverse_transform()` method, we see the labels assigned to each value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['red', 'white'], dtype='<U5')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelBinarizer\n",
    "\n",
    "binary_labels = LabelBinarizer().fit(wine.kind)\n",
    "binary_labels.inverse_transform(np.array([0, 1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `Binarizer` class for binary encoding of values based on a threshold. Values less than or equal to `threshold` will be 0; values greater than `threshold` will be 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1382\n",
       "1     217\n",
       "dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import Binarizer\n",
    "\n",
    "pd.Series(\n",
    "    Binarizer(threshold=6).fit_transform(red_wine.quality.values.reshape(-1, 1)).flatten()\n",
    ").value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ordinal Encoding with `LabelEncoder`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    1372\n",
       "2     217\n",
       "0      10\n",
       "dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "pd.Series(LabelEncoder().fit_transform(pd.cut(\n",
    "    red_wine.quality,\n",
    "    bins=[-1, 3, 6, 10],\n",
    "    labels=['0-3 (low)', '4-6 (med)', '7-10 (high)']\n",
    "))).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One-hot encoding\n",
    "In some cases, label encoding may yield some associations that aren't something we want the model to be trained on. A safer strategy is to use one-hot encoding.\n",
    "\n",
    "Our planets data has a `list` column that we can one-hot encode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Confirmed planets                    3972\n",
       "Controversial                          97\n",
       "Retracted planet candidate             11\n",
       "Solar System                            9\n",
       "Kepler Objects of Interest              4\n",
       "Planets in binary systems, S-type       1\n",
       "Name: list, dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "planets.list.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `pd.get_dummies()` to one-hot encode this information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Confirmed planets</th>\n",
       "      <th>Controversial</th>\n",
       "      <th>Kepler Objects of Interest</th>\n",
       "      <th>Planets in binary systems, S-type</th>\n",
       "      <th>Retracted planet candidate</th>\n",
       "      <th>Solar System</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Confirmed planets  Controversial  Kepler Objects of Interest  \\\n",
       "0                  1              0                           0   \n",
       "1                  1              0                           0   \n",
       "2                  1              0                           0   \n",
       "3                  1              0                           0   \n",
       "4                  0              1                           0   \n",
       "\n",
       "   Planets in binary systems, S-type  Retracted planet candidate  Solar System  \n",
       "0                                  0                           0             0  \n",
       "1                                  0                           0             0  \n",
       "2                                  0                           0             0  \n",
       "3                                  0                           0             0  \n",
       "4                                  0                           0             0  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.get_dummies(planets.list).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us a redundant column. Note that we only need one less column than the number of planet lists. Pandas makes it easy to remove one of the columns to address multicollinearity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Controversial</th>\n",
       "      <th>Kepler Objects of Interest</th>\n",
       "      <th>Planets in binary systems, S-type</th>\n",
       "      <th>Retracted planet candidate</th>\n",
       "      <th>Solar System</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Controversial  Kepler Objects of Interest  \\\n",
       "0              0                           0   \n",
       "1              0                           0   \n",
       "2              0                           0   \n",
       "3              0                           0   \n",
       "4              1                           0   \n",
       "\n",
       "   Planets in binary systems, S-type  Retracted planet candidate  Solar System  \n",
       "0                                  0                           0             0  \n",
       "1                                  0                           0             0  \n",
       "2                                  0                           0             0  \n",
       "3                                  0                           0             0  \n",
       "4                                  0                           0             0  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.get_dummies(planets.list, drop_first=True).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the `LabelBinarizer` class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 0, 0, 0, 0, 0],\n",
       "       [1, 0, 0, 0, 0, 0],\n",
       "       [1, 0, 0, 0, 0, 0],\n",
       "       ...,\n",
       "       [1, 0, 0, 0, 0, 0],\n",
       "       [1, 0, 0, 0, 0, 0],\n",
       "       [1, 0, 0, 0, 0, 0]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelBinarizer\n",
    "\n",
    "LabelBinarizer().fit_transform(planets.list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imputing\n",
    "The planets data has some missing values. We can use imputing strategies to avoid having to drop them from our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>semimajoraxis</th>\n",
       "      <th>mass</th>\n",
       "      <th>eccentricity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4089</th>\n",
       "      <td>0.08150</td>\n",
       "      <td>1.9000</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4090</th>\n",
       "      <td>0.04421</td>\n",
       "      <td>0.7090</td>\n",
       "      <td>0.038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4091</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0.3334</td>\n",
       "      <td>0.310</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4092</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4093</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0.4200</td>\n",
       "      <td>0.160</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      semimajoraxis    mass  eccentricity\n",
       "4089        0.08150  1.9000         0.000\n",
       "4090        0.04421  0.7090         0.038\n",
       "4091            NaN  0.3334         0.310\n",
       "4092            NaN  0.4000         0.270\n",
       "4093            NaN  0.4200         0.160"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "planets[['semimajoraxis', 'mass', 'eccentricity']].tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `SimpleImputer`\n",
    "We can fill with the `mean`, `median`, `most_frequent` (mode), or a constant value by specifiying the `strategy`. The default is the mean:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.29      , 19.4       ,  0.231     ],\n",
       "       [ 1.54      , 11.2       ,  0.08      ],\n",
       "       [ 0.83      ,  4.8       ,  0.        ],\n",
       "       ...,\n",
       "       [ 5.83796389,  0.3334    ,  0.31      ],\n",
       "       [ 5.83796389,  0.4       ,  0.27      ],\n",
       "       [ 5.83796389,  0.42      ,  0.16      ]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "SimpleImputer().fit_transform(\n",
    "    planets[['semimajoraxis', 'mass', 'eccentricity']]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Changing to the median is just a matter of passing that as the `strategy`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.29  , 19.4   ,  0.231 ],\n",
       "       [ 1.54  , 11.2   ,  0.08  ],\n",
       "       [ 0.83  ,  4.8   ,  0.    ],\n",
       "       ...,\n",
       "       [ 0.1409,  0.3334,  0.31  ],\n",
       "       [ 0.1409,  0.4   ,  0.27  ],\n",
       "       [ 0.1409,  0.42  ,  0.16  ]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "SimpleImputer(strategy='median').fit_transform(\n",
    "    planets[['semimajoraxis', 'mass', 'eccentricity']]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `KNNImputer`\n",
    "Since this data isn't something that is easily measured, assuming that the planets we don't have the data for are similar to the rest is dangerous. It could be that the ones that have missing data have something in common. Replacing missing values for the semi-major axis with the average of the ones we know is hardly a good strategy. Instead, we could try to use the mass and eccentricity columns to find similar planets and use their semi-major axes to impute the missing data. This can be done with the `KNNImputer` class. Notice the first column in the bottom 3 rows (the imputed semi-major axis is drastically different from what we got using the overall mean:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.29    , 19.4     ,  0.231   ],\n",
       "       [ 1.54    , 11.2     ,  0.08    ],\n",
       "       [ 0.83    ,  4.8     ,  0.      ],\n",
       "       ...,\n",
       "       [ 0.404726,  0.3334  ,  0.31    ],\n",
       "       [ 0.85486 ,  0.4     ,  0.27    ],\n",
       "       [ 0.15324 ,  0.42    ,  0.16    ]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.impute import KNNImputer\n",
    "\n",
    "KNNImputer().fit_transform(\n",
    "    planets[['semimajoraxis', 'mass', 'eccentricity']]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `MissingIndicator`\n",
    "In some cases, we don't want to fill in a value, but rather use the fact that the data is missing as a feature in our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[False, False, False],\n",
       "       [False, False, False],\n",
       "       [False, False, False],\n",
       "       ...,\n",
       "       [ True, False, False],\n",
       "       [ True, False, False],\n",
       "       [ True, False, False]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.impute import MissingIndicator\n",
    "\n",
    "MissingIndicator().fit_transform(\n",
    "    planets[['semimajoraxis', 'mass', 'eccentricity']]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Transformers\n",
    "### `FunctionTransformer`\n",
    "With the `FunctionTransformer` class, we can use any function on the data. By passing `validate=True`, we will convert the result to two-dimensional NumPy array and raise an error if there is an issue:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.51   , 4.94   , 1.45   ],\n",
       "       [0.17   , 0.64   , 0.85   ],\n",
       "       [0.08   , 0.03727, 1.192  ],\n",
       "       ...,\n",
       "       [0.295  , 4.46   , 1.8    ],\n",
       "       [0.34   , 0.0652 , 0.0087 ],\n",
       "       [0.3    , 1.26   , 0.5    ]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import FunctionTransformer\n",
    "\n",
    "FunctionTransformer(\n",
    "    np.abs, validate=True\n",
    ").fit_transform(X_train.dropna())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `ColumnTransformer`\n",
    "Sometimes we don't want to perform the same transformation on all of our features, the `ColumnTransformer` class lets us specify which tranformations to use on each column. We pass a list of tuples in the form `(name, transformer object, columns to apply to)`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.17      , -0.04747176,  0.0107594 ],\n",
       "       [ 0.08      , -0.05475873,  0.01508851],\n",
       "       [ 0.15585591,         nan,  0.13924042],\n",
       "       [ 0.15585591,         nan,         nan],\n",
       "       [ 0.        , -0.05475111,  0.00478471]])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.compose import ColumnTransformer \n",
    "from sklearn.impute import KNNImputer\n",
    "from sklearn.preprocessing import MinMaxScaler, StandardScaler\n",
    "\n",
    "ColumnTransformer([\n",
    "    ('impute', KNNImputer(), [0]),\n",
    "    ('standard_scale', StandardScaler(), [1]),\n",
    "    ('min_max', MinMaxScaler(), [2])\n",
    "]).fit_transform(X_train)[10:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the `make_column_transformer()` function, which will name the transformers for us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 3.09267587, -0.2351423 , -0.40487424, ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [ 1.432445  , -0.24215395, -0.28360905, ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [ 0.13665505, -0.24208849, -0.62800218, ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       ...,\n",
       "       [-0.83289954, -0.76197788, -0.84918988, ...,  0.        ,\n",
       "         1.        ,  0.        ],\n",
       "       [ 0.25813535,  0.38683239, -0.92873984, ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [-0.26827931, -0.21657671, -0.70076129, ...,  0.        ,\n",
       "         0.        ,  1.        ]])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.compose import make_column_transformer\n",
    "from sklearn.preprocessing import OneHotEncoder, StandardScaler\n",
    "\n",
    "categorical = [\n",
    "    col for col in planets.columns\n",
    "    if col in [\n",
    "        'list', 'name', 'description', \n",
    "        'discoverymethod', 'lastupdate'\n",
    "    ]\n",
    "]\n",
    "numeric = [col for col in planets.columns if col not in categorical]\n",
    "\n",
    "make_column_transformer(\n",
    "    (StandardScaler(), numeric),\n",
    "    (OneHotEncoder(sparse=False), categorical)\n",
    ").fit_transform(planets.dropna())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `Pipeline`\n",
    "Using pipelines ensures the whole model training and testing process is consistent. To make a pipeline, we pass in a list of steps as tuples of `(name, object)`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('scale', StandardScaler()), ('lr', LinearRegression())])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "Pipeline([('scale', StandardScaler()), ('lr', LinearRegression())])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We aren't limited to using pipelines with models &mdash; they can be used inside other `sklearn` objects. This makes it possible for us to first use k-NN imputing on the semi-major axis data and then standard scale the result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.13531604, -0.04747176,  0.0107594 ],\n",
       "       [-0.7257111 , -0.05475873,  0.01508851],\n",
       "       [ 0.        ,         nan,  0.13924042],\n",
       "       [ 0.        ,         nan,         nan],\n",
       "       [-1.49106856, -0.05475111,  0.00478471]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.compose import ColumnTransformer \n",
    "from sklearn.impute import KNNImputer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import MinMaxScaler, StandardScaler\n",
    "\n",
    "ColumnTransformer([\n",
    "    ('impute', Pipeline([\n",
    "        ('impute', KNNImputer()), ('scale', StandardScaler())\n",
    "    ]), [0]),\n",
    "    ('standard_scale', StandardScaler(), [1]),\n",
    "    ('min_max', MinMaxScaler(), [2])\n",
    "]).fit_transform(X_train)[10:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then include this as part of a pipeline, which gives us tremendous flexibility in how we build our models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('preprocessing',\n",
       "                 ColumnTransformer(transformers=[('impute',\n",
       "                                                  Pipeline(steps=[('impute',\n",
       "                                                                   KNNImputer()),\n",
       "                                                                  ('scale',\n",
       "                                                                   StandardScaler())]),\n",
       "                                                  [0]),\n",
       "                                                 ('standard_scale',\n",
       "                                                  StandardScaler(), [1]),\n",
       "                                                 ('min_max', MinMaxScaler(),\n",
       "                                                  [2])])),\n",
       "                ('model', LinearRegression())])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Pipeline([\n",
    "    (\n",
    "        'preprocessing', \n",
    "        ColumnTransformer([\n",
    "            ('impute', Pipeline([\n",
    "                ('impute', KNNImputer()), ('scale', StandardScaler())\n",
    "            ]), [0]),\n",
    "            ('standard_scale', StandardScaler(), [1]),\n",
    "            ('min_max', MinMaxScaler(), [2])\n",
    "        ])\n",
    "    ),\n",
    "    ('model', LinearRegression())\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the `make_pipeline()` function to make the pipeline without naming the steps ourselves:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('standardscaler', StandardScaler()),\n",
       "                ('linearregression', LinearRegression())])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.pipeline import make_pipeline\n",
    "\n",
    "make_pipeline(StandardScaler(), LinearRegression())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "<div style=\"overflow: hidden; margin-bottom: 10px;\">\n",
    "    <div style=\"float: left;\">\n",
    "        <a href=\"./planets_ml.ipynb\">\n",
    "            <button>Planets</button>\n",
    "        </a>\n",
    "        <a href=\"./red_wine.ipynb\">\n",
    "            <button>Red Wine</button>\n",
    "        </a>\n",
    "        <a href=\"./wine.ipynb\">\n",
    "            <button>Red + White Wine</button>\n",
    "        </a>\n",
    "    </div>\n",
    "    <div style=\"float: right;\">\n",
    "        <a href=\"../ch_09/planets_ml.ipynb\">\n",
    "            <button>Next Notebook &#8594;</button>\n",
    "        </a>\n",
    "    </div>\n",
    "</div>\n",
    "<hr>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}