pa_pynb_sess6_CreditCards.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.3"
    },
    "colab": {
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/danielbauer1979/CAS_PredMod/blob/main/pa_pynb_sess6_CreditCards.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Credit Card Default Case Study\n",
        "\n",
        "Dani Bauer, 2022"
      ],
      "metadata": {
        "id": "bFNuMreLe4xh"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cQkFLmYWqmN3"
      },
      "source": [
        "In this tutorial, we will introduce one of the examples we will use in the coming tutorials: We will attempt to predict defaults on credits cards using a public dataset.\n",
        "\n",
        "We will apply several of our predictive models from the past weeks (glm and gam) but also check the performance of a purley algorithmic approach: the *k-nearest-neighbor classifier*. \n",
        "\n",
        "## Review of Concepts and Maths\n",
        "\n",
        "As we saw in a previously, non-linear regression models are powerful approaches for depicting non-linear relationships -- with the key caveat that our explanatory variable had a single dimension only.  Once we venture into higher dimensions -- that means multiple, potentially interrelated features -- obtaining a similar fit will require a *very* large number of data points becaus of the **curse of dimensionality**.  Hence, statistical learning methods (need to) impose structure, in one way or another, and picking a learner in some way depends on picking an appropiate structure. **Generalized Additive Models** (GAMs) -- leverages the performance of non-linear regression models in lower dimensions but imposes an *additive* structure between the functions of the individual features:\n",
        "$$\n",
        "g\\left(\\mathbb{E}\\left[Y | X_1,...,X_p\\right]\\right) = \\alpha + f_1(X_1) + \\ldots + f_p(X_p).\n",
        "$$\n",
        "As such, GAMs take on an intermediate position between linear regression and a general non-linear model:  They generalize the impact each feature can have on the outcome, but they keep the same structure on their relationship.  In particular, they do not allow for rich interactions between the variables -- which is the key downside of GAMs.\n",
        "\n",
        "Other so-called *algorithmic* learners use different structural assumptions. For instance, we illustrate a **k-nearest neighbor (knn)** approach, where the predicted class at a point $x_0$ is chosen based on the $k$ points that are closest:\n",
        "$$\n",
        "y(x_0) = \\max_j\\left\\{\\frac{1}{K} \\sum_{i \\in N_K(x_0)} 1_{\\{y_i=j\\}}\\right\\},\n",
        "$$\n",
        "where $N_k(x_0)$ denotes the index set of the $K$ points in the training sample that are closest to the point $x_0$ (usually in the sense of Euclidean distance).  This is very differnt than what we have seen before in that we don't have an underlying \"probabilistic\" approach.\n",
        "\n",
        "## Credit Card Default Application\n",
        "\n",
        "We rely on the dataset `pa_data_UCI_Credit_Card.csv` from the UCI Machine Learning Repository (Lichman, M., 2013. [UCI Machine Learning Repository](http://archive.ics.uci.edu/ml). Irvine, CA: University of California, School of Information and Computer Science).  This datasets provides credit card defaults for customers in Taiwan.  We are given some demographic information ($X_1$-$X_5$), the previous history of payments ($X_6$-$X_{11}$), the amount of previous bills ($X_{12}$-$X_{17}$), and amounts of previous payments ($X_{18}$-$X_{23}$).  Finally, variable 24 is our target, whetyher there was a default in the next months.\n",
        "\n",
        "\n",
        "As always, let's start with importing the libraries:"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install pygam"
      ],
      "metadata": {
        "id": "8aOk8EV7dgd9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ANNbSksYqWS3"
      },
      "source": [
        "import numpy as np \n",
        "import matplotlib.pyplot as plt  \n",
        "import pandas as pd \n",
        "from sklearn.model_selection import train_test_split\n",
        "import seaborn as sns\n",
        "\n",
        "from sklearn.linear_model import LogisticRegression \n",
        "from sklearn.metrics import confusion_matrix, classification_report, precision_score, roc_curve, auc\n",
        "from sklearn import preprocessing\n",
        "from sklearn.preprocessing import scale\n",
        "from pygam import LogisticGAM, LinearGAM, GAM, s, f, l\n",
        "from sklearn.neighbors import KNeighborsClassifier"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's load the dataset"
      ],
      "metadata": {
        "id": "okvP8ddqfGwX"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "70QK-AnxqWTB"
      },
      "source": [
        "!git clone https://github.com/danielbauer1979/CAS_PredMod.git"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uzW2COB-qWTC"
      },
      "source": [
        "mydata = pd.read_csv('CAS_PredMod/pa_data_UCI_Credit_Card.csv', index_col=0)"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Data Exploration and Preparation"
      ],
      "metadata": {
        "id": "Y4q8flL3fLw8"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dZ7xrrgfqWTC"
      },
      "source": [
        "mydata.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's look at some aggregate statistics."
      ],
      "metadata": {
        "id": "bdiFv-p4fQ1j"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xWYoH_ngqWTD"
      },
      "source": [
        "mydata.describe()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "First, a number of the variables are included numerically but really they have factor character, particularly Gender (1 = male; 2 = female), Education (1 = graduate school; 2 = university; 3 = high school; 4 = others), Marital status (1 = married; 2 = single; 3 = others), and default payment. Let's store them as factors.  We will do the same for history of past payment ($X_6$-$X_{11}$), although they really have ordinal character."
      ],
      "metadata": {
        "id": "651lfbJMgE04"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "factor = ['SEX', 'EDUCATION', 'MARRIAGE', 'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'default.payment.next.month']"
      ],
      "metadata": {
        "id": "ZAwjAtXxgJ9B"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Also, a number of the levels occur very sparsely: there are 11 levels for all the `PAY` variables, but only the first six seem to be frequent.  So let's collapse levels 7 through 11 to one:"
      ],
      "metadata": {
        "id": "23vcLxDYgRcb"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Jr5L46eWqWTD"
      },
      "source": [
        "mydata['PAY_0'][mydata['PAY_0']>4] = 4\n",
        "mydata['PAY_2'][mydata['PAY_2']>4] = 4\n",
        "mydata['PAY_3'][mydata['PAY_3']>4] = 4\n",
        "mydata['PAY_4'][mydata['PAY_4']>4] = 4\n",
        "mydata['PAY_5'][mydata['PAY_5']>4] = 4\n",
        "mydata['PAY_6'][mydata['PAY_6']>4] = 4"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, we rescale the numerical columns, as we know a number of learners require scaled inputs (and it should not matter for others):"
      ],
      "metadata": {
        "id": "Q4xm1bOegjmw"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-rfdm9IdqWTE"
      },
      "source": [
        "mydata_numcols = mydata.drop(columns = factor)\n",
        "mydata_faccols = mydata[factor].drop(columns = ['default.payment.next.month']).astype('category')\n",
        "dummies = pd.get_dummies(mydata_faccols, drop_first=True)\n",
        "mydata_numcols_sc_0 = scale(mydata_numcols)\n",
        "mydata_numcols_sc = pd.DataFrame(data=mydata_numcols_sc_0, columns = mydata_numcols.columns, index = dummies.index)\n",
        "mydata_sc = pd.concat([mydata_numcols_sc, dummies], axis = 1)\n",
        "mydata_sc = pd.concat([mydata_sc, mydata['default.payment.next.month']], axis = 1)"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "And Let's relabel the long name of the dependent variable:"
      ],
      "metadata": {
        "id": "tSOZn3i-gtKf"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dCaaHETUqWTF"
      },
      "source": [
        "mydata = mydata.rename(columns={\"default.payment.next.month\": \"default\"})"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's take a look:"
      ],
      "metadata": {
        "id": "fwfWHqovgynk"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0UfQEQiEqWTF"
      },
      "source": [
        "mydata.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zibCpbmrqWTG",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 393
        },
        "outputId": "fd2fe5ff-3860-4142-a56e-9dcaab271006"
      },
      "source": [
        "mydata.describe()"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "            LIMIT_BAL           SEX     EDUCATION      MARRIAGE           AGE  \\\n",
              "count    30000.000000  30000.000000  30000.000000  30000.000000  30000.000000   \n",
              "mean    167484.322667      1.603733      1.853133      1.551867     35.485500   \n",
              "std     129747.661567      0.489129      0.790349      0.521970      9.217904   \n",
              "min      10000.000000      1.000000      0.000000      0.000000     21.000000   \n",
              "25%      50000.000000      1.000000      1.000000      1.000000     28.000000   \n",
              "50%     140000.000000      2.000000      2.000000      2.000000     34.000000   \n",
              "75%     240000.000000      2.000000      2.000000      2.000000     41.000000   \n",
              "max    1000000.000000      2.000000      6.000000      3.000000     79.000000   \n",
              "\n",
              "              PAY_0         PAY_2         PAY_3         PAY_4         PAY_5  \\\n",
              "count  30000.000000  30000.000000  30000.000000  30000.000000  30000.000000   \n",
              "mean      -0.021733     -0.137533     -0.171533     -0.228233     -0.272967   \n",
              "std        1.098773      1.180310      1.172414      1.132542      1.098770   \n",
              "min       -2.000000     -2.000000     -2.000000     -2.000000     -2.000000   \n",
              "25%       -1.000000     -1.000000     -1.000000     -1.000000     -1.000000   \n",
              "50%        0.000000      0.000000      0.000000      0.000000      0.000000   \n",
              "75%        0.000000      0.000000      0.000000      0.000000      0.000000   \n",
              "max        4.000000      4.000000      4.000000      4.000000      4.000000   \n",
              "\n",
              "       ...      BILL_AMT4      BILL_AMT5      BILL_AMT6       PAY_AMT1  \\\n",
              "count  ...   30000.000000   30000.000000   30000.000000   30000.000000   \n",
              "mean   ...   43262.948967   40311.400967   38871.760400    5663.580500   \n",
              "std    ...   64332.856134   60797.155770   59554.107537   16563.280354   \n",
              "min    ... -170000.000000  -81334.000000 -339603.000000       0.000000   \n",
              "25%    ...    2326.750000    1763.000000    1256.000000    1000.000000   \n",
              "50%    ...   19052.000000   18104.500000   17071.000000    2100.000000   \n",
              "75%    ...   54506.000000   50190.500000   49198.250000    5006.000000   \n",
              "max    ...  891586.000000  927171.000000  961664.000000  873552.000000   \n",
              "\n",
              "           PAY_AMT2      PAY_AMT3       PAY_AMT4       PAY_AMT5  \\\n",
              "count  3.000000e+04   30000.00000   30000.000000   30000.000000   \n",
              "mean   5.921163e+03    5225.68150    4826.076867    4799.387633   \n",
              "std    2.304087e+04   17606.96147   15666.159744   15278.305679   \n",
              "min    0.000000e+00       0.00000       0.000000       0.000000   \n",
              "25%    8.330000e+02     390.00000     296.000000     252.500000   \n",
              "50%    2.009000e+03    1800.00000    1500.000000    1500.000000   \n",
              "75%    5.000000e+03    4505.00000    4013.250000    4031.500000   \n",
              "max    1.684259e+06  896040.00000  621000.000000  426529.000000   \n",
              "\n",
              "            PAY_AMT6       default  \n",
              "count   30000.000000  30000.000000  \n",
              "mean     5215.502567      0.221200  \n",
              "std     17777.465775      0.415062  \n",
              "min         0.000000      0.000000  \n",
              "25%       117.750000      0.000000  \n",
              "50%      1500.000000      0.000000  \n",
              "75%      4000.000000      0.000000  \n",
              "max    528666.000000      1.000000  \n",
              "\n",
              "[8 rows x 24 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-ca422291-a775-4a9b-8a81-43de9599e9fb\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>LIMIT_BAL</th>\n",
              "      <th>SEX</th>\n",
              "      <th>EDUCATION</th>\n",
              "      <th>MARRIAGE</th>\n",
              "      <th>AGE</th>\n",
              "      <th>PAY_0</th>\n",
              "      <th>PAY_2</th>\n",
              "      <th>PAY_3</th>\n",
              "      <th>PAY_4</th>\n",
              "      <th>PAY_5</th>\n",
              "      <th>...</th>\n",
              "      <th>BILL_AMT4</th>\n",
              "      <th>BILL_AMT5</th>\n",
              "      <th>BILL_AMT6</th>\n",
              "      <th>PAY_AMT1</th>\n",
              "      <th>PAY_AMT2</th>\n",
              "      <th>PAY_AMT3</th>\n",
              "      <th>PAY_AMT4</th>\n",
              "      <th>PAY_AMT5</th>\n",
              "      <th>PAY_AMT6</th>\n",
              "      <th>default</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>30000.000000</td>\n",
              "      <td>30000.000000</td>\n",
              "      <td>30000.000000</td>\n",
              "      <td>30000.000000</td>\n",
              "      <td>30000.000000</td>\n",
              "      <td>30000.000000</td>\n",
              "      <td>30000.000000</td>\n",
              "      <td>30000.000000</td>\n",
              "      <td>30000.000000</td>\n",
              "      <td>30000.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>30000.000000</td>\n",
              "      <td>30000.000000</td>\n",
              "      <td>30000.000000</td>\n",
              "      <td>30000.000000</td>\n",
              "      <td>3.000000e+04</td>\n",
              "      <td>30000.00000</td>\n",
              "      <td>30000.000000</td>\n",
              "      <td>30000.000000</td>\n",
              "      <td>30000.000000</td>\n",
              "      <td>30000.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>167484.322667</td>\n",
              "      <td>1.603733</td>\n",
              "      <td>1.853133</td>\n",
              "      <td>1.551867</td>\n",
              "      <td>35.485500</td>\n",
              "      <td>-0.021733</td>\n",
              "      <td>-0.137533</td>\n",
              "      <td>-0.171533</td>\n",
              "      <td>-0.228233</td>\n",
              "      <td>-0.272967</td>\n",
              "      <td>...</td>\n",
              "      <td>43262.948967</td>\n",
              "      <td>40311.400967</td>\n",
              "      <td>38871.760400</td>\n",
              "      <td>5663.580500</td>\n",
              "      <td>5.921163e+03</td>\n",
              "      <td>5225.68150</td>\n",
              "      <td>4826.076867</td>\n",
              "      <td>4799.387633</td>\n",
              "      <td>5215.502567</td>\n",
              "      <td>0.221200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>129747.661567</td>\n",
              "      <td>0.489129</td>\n",
              "      <td>0.790349</td>\n",
              "      <td>0.521970</td>\n",
              "      <td>9.217904</td>\n",
              "      <td>1.098773</td>\n",
              "      <td>1.180310</td>\n",
              "      <td>1.172414</td>\n",
              "      <td>1.132542</td>\n",
              "      <td>1.098770</td>\n",
              "      <td>...</td>\n",
              "      <td>64332.856134</td>\n",
              "      <td>60797.155770</td>\n",
              "      <td>59554.107537</td>\n",
              "      <td>16563.280354</td>\n",
              "      <td>2.304087e+04</td>\n",
              "      <td>17606.96147</td>\n",
              "      <td>15666.159744</td>\n",
              "      <td>15278.305679</td>\n",
              "      <td>17777.465775</td>\n",
              "      <td>0.415062</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>10000.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>21.000000</td>\n",
              "      <td>-2.000000</td>\n",
              "      <td>-2.000000</td>\n",
              "      <td>-2.000000</td>\n",
              "      <td>-2.000000</td>\n",
              "      <td>-2.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>-170000.000000</td>\n",
              "      <td>-81334.000000</td>\n",
              "      <td>-339603.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000e+00</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>50000.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>28.000000</td>\n",
              "      <td>-1.000000</td>\n",
              "      <td>-1.000000</td>\n",
              "      <td>-1.000000</td>\n",
              "      <td>-1.000000</td>\n",
              "      <td>-1.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>2326.750000</td>\n",
              "      <td>1763.000000</td>\n",
              "      <td>1256.000000</td>\n",
              "      <td>1000.000000</td>\n",
              "      <td>8.330000e+02</td>\n",
              "      <td>390.00000</td>\n",
              "      <td>296.000000</td>\n",
              "      <td>252.500000</td>\n",
              "      <td>117.750000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>140000.000000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>34.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>19052.000000</td>\n",
              "      <td>18104.500000</td>\n",
              "      <td>17071.000000</td>\n",
              "      <td>2100.000000</td>\n",
              "      <td>2.009000e+03</td>\n",
              "      <td>1800.00000</td>\n",
              "      <td>1500.000000</td>\n",
              "      <td>1500.000000</td>\n",
              "      <td>1500.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>240000.000000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>41.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>54506.000000</td>\n",
              "      <td>50190.500000</td>\n",
              "      <td>49198.250000</td>\n",
              "      <td>5006.000000</td>\n",
              "      <td>5.000000e+03</td>\n",
              "      <td>4505.00000</td>\n",
              "      <td>4013.250000</td>\n",
              "      <td>4031.500000</td>\n",
              "      <td>4000.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>1000000.000000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>6.000000</td>\n",
              "      <td>3.000000</td>\n",
              "      <td>79.000000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>891586.000000</td>\n",
              "      <td>927171.000000</td>\n",
              "      <td>961664.000000</td>\n",
              "      <td>873552.000000</td>\n",
              "      <td>1.684259e+06</td>\n",
              "      <td>896040.00000</td>\n",
              "      <td>621000.000000</td>\n",
              "      <td>426529.000000</td>\n",
              "      <td>528666.000000</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>8 rows × 24 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ca422291-a775-4a9b-8a81-43de9599e9fb')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-ca422291-a775-4a9b-8a81-43de9599e9fb button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-ca422291-a775-4a9b-8a81-43de9599e9fb');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's check a correlation plot to make sure none of the variables is extremely correlated:"
      ],
      "metadata": {
        "id": "HdHdC-Kyg5A2"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xpsVMCpCqWTG"
      },
      "source": [
        "mask = np.triu(np.ones_like(mydata.corr(), dtype=bool))\n",
        "cmap = sns.diverging_palette(230, 20, as_cmap=True)\n",
        "sns.heatmap(mydata.corr(), mask=mask, cmap=cmap, vmax=.3, center=0,\n",
        "            square=True, linewidths=.5, cbar_kws={\"shrink\": .5})"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "So it looks like the bill amounts are highly correlated.  Let's just keep the most recent one and then the average of all of them:\n"
      ],
      "metadata": {
        "id": "ihKWiDkphM_3"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bLIXJYlyqWTH"
      },
      "source": [
        "mydata.insert(17, \"BILL_AVG\", (mydata['BILL_AMT1']+mydata['BILL_AMT2']+mydata['BILL_AMT3']+mydata['BILL_AMT4']+mydata['BILL_AMT5']+mydata['BILL_AMT6'])/6, True) \n",
        "mydata = mydata.rename(columns={\"BILL_AMT1\": \"BILL_REC\"})\n",
        "del mydata['BILL_AMT2']\n",
        "del mydata['BILL_AMT3']\n",
        "del mydata['BILL_AMT4']\n",
        "del mydata['BILL_AMT5']\n",
        "del mydata['BILL_AMT6']\n",
        "mydata.describe()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's save the dataset so that we can use it in coming tutorials without having to go through this procedure again:"
      ],
      "metadata": {
        "id": "IdJvKO_FhSkU"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tRIpqkhsqWTH"
      },
      "source": [
        "mydata.to_csv('pa_data_UCI_Credit_Card_prepped.csv') "
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Predictive Modeling"
      ],
      "metadata": {
        "id": "Xwm41PjahXf0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "As usually, let's split our dataset:"
      ],
      "metadata": {
        "id": "e80dFeKijLfn"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lK8HBxrBqWTI"
      },
      "source": [
        "Train, Test = train_test_split(mydata, test_size=0.25)\n",
        "Train_y = Train['default']\n",
        "Train = Train.drop(columns = ['default'])\n",
        "Test_y = Test['default']\n",
        "Test = Test.drop(columns = ['default'])"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's run a logistic regression model:"
      ],
      "metadata": {
        "id": "l1BP7jjcjPzy"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sZ5lP5pnBP7F"
      },
      "source": [
        "logistic_model1 = LogisticRegression(fit_intercept=True, max_iter=500).fit(Train,Train_y)\n",
        "print(logistic_model1.intercept_)\n",
        "print(logistic_model1.coef_)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's check predictions:"
      ],
      "metadata": {
        "id": "PxNquZnljYn2"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "egLqSR1RD1bb"
      },
      "source": [
        "logistic_pred_1 = logistic_model1.predict_proba(Test)\n",
        "np.sum(logistic_pred_1[:,1] > 0.5)\n",
        "np.sum(logistic_pred_1[:,1] > 0.38)\n",
        "logistic_pred_1_lab = logistic_pred_1[:,1] > 0.36\n",
        "confusion_matrix(Test_y, logistic_pred_1_lab)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "So we are missing quite a few.  Let's condider the AUC:"
      ],
      "metadata": {
        "id": "4pC5K5zNjbXe"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OIY4iAiwqWTL"
      },
      "source": [
        "fpr, tpr, threshold = roc_curve(Test_y, logistic_pred_1[:,1])\n",
        "roc_auc = auc(fpr, tpr)\n",
        "plt.title('Receiver Operating Characteristic')\n",
        "plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)\n",
        "plt.legend(loc = 'lower right')\n",
        "plt.plot([0, 1], [0, 1],'r--')\n",
        "plt.xlim([0, 1])\n",
        "plt.ylim([0, 1])\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.xlabel('False Positive Rate')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's see if we can build a GAM that beats the linear learners.  We use some of the information from the GLM, and ignore some of the insignificant variables.  We include non-linear effects in the significant continuous features, and also interact `PAY_0` with `LIMIT_BAL` -- simply because that seems to be a good idea.  Remember, incorporating intuition is legitimate and even important given the *No Free Lunch Theorem* (of course, we could improve on the GLMs as well by using interaction terms)."
      ],
      "metadata": {
        "id": "UX4ppWSQjieW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "gam = LogisticGAM(s(0,n_splines=5) + l(1) + f(2) + f(3) + l(4) + f(5) + f(6) + f(7) + f(8) + f(9) + f(10) + s(11, n_splines=5) + s(12, n_splines=5) + s(13, n_splines=5) + l(14) + l(15) + l(16)).fit(Train, Train_y)\n",
        "XX = gam.generate_X_grid\n",
        "plt.rcParams['figure.figsize'] = (28, 8)\n",
        "fig, axs = plt.subplots(1, 17)\n",
        "titles = list(Train.columns)\n",
        "for i, ax in enumerate(axs):\n",
        "    XX = gam.generate_X_grid(term=i)\n",
        "    ax.plot(XX[:, i], gam.partial_dependence(term=i, X=XX))\n",
        "    ax.plot(XX[:, i], gam.partial_dependence(term=i, X=XX, width=.95)[1], c='r', ls='--')\n",
        "    ax.set_title(titles[i]);\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "BuN17cp8SInB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's check if it does better:"
      ],
      "metadata": {
        "id": "R7f3lNM6joLa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "gam_preds = gam.predict_proba(Test)\n",
        "gam_pred_labels = np.zeros(len(Test_y))\n",
        "gam_pred_labels[gam_preds >0.5] = 1\n",
        "confusion_matrix(Test_y, gam_pred_labels)"
      ],
      "metadata": {
        "id": "OdB8F3GiSMe7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Looks like it!\n",
        "\n",
        "Let's check the AUC:"
      ],
      "metadata": {
        "id": "mxQQDqXvjrX3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fpr, tpr, threshold = roc_curve(Test_y, gam_preds)\n",
        "roc_auc = auc(fpr, tpr)\n",
        "plt.title('Receiver Operating Characteristic')\n",
        "plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)\n",
        "plt.legend(loc = 'lower right')\n",
        "plt.plot([0, 1], [0, 1],'r--')\n",
        "plt.xlim([0, 1])\n",
        "plt.ylim([0, 1])\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.xlabel('False Positive Rate')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "c7i7SdD3SUMC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's also check the knn classifier:"
      ],
      "metadata": {
        "id": "yrAs83vDj0Hm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "knn_model = KNeighborsClassifier(n_neighbors=5)\n",
        "knn_model.fit(Train, Train_y)\n",
        "Test_y_knn = knn_model.predict(Test)\n",
        "confusion_matrix(Test_y, Test_y_knn)"
      ],
      "metadata": {
        "id": "RYAqm5Ikj8vn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "So it doesn't look it works too well here."
      ],
      "metadata": {
        "id": "soOptsWJkcg2"
      }
    }
  ]
}