predict_readmission.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "dutch-handle",
   "metadata": {},
   "source": [
    "# Predicting Hospital Readmission for Patients with Diabetes\n",
    "### by Nishant Nayak\n",
    "### May 9th, 2021"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "peaceful-blank",
   "metadata": {},
   "source": [
    "## Project summary\n",
    "The objective of this project is to develop a model that will predict whether or not a patient will be readmitted within 30 days. The report is divided into six sections:\n",
    "- Data exploration\n",
    "- Feature extraction\n",
    "- Creating Training / Validation / Test Data\n",
    "- Model Selection\n",
    "- Test Evaluation\n",
    "- Conclusion\n",
    "\n",
    "The best model, the SGD Logistic Regression classifer, has a 67% test accuracy on held-out data and an AUC score of 0.66."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "advised-specialist",
   "metadata": {},
   "source": [
    "## Data exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "laden-beast",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "attended-savings",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number samples:  101766\n"
     ]
    }
   ],
   "source": [
    "hospital_df = pd.read_csv('diabetic_data.csv')\n",
    "print('Total number samples: ', len(hospital_df))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "alpine-minute",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>encounter_id</th>\n",
       "      <th>patient_nbr</th>\n",
       "      <th>race</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>weight</th>\n",
       "      <th>admission_type_id</th>\n",
       "      <th>discharge_disposition_id</th>\n",
       "      <th>admission_source_id</th>\n",
       "      <th>time_in_hospital</th>\n",
       "      <th>...</th>\n",
       "      <th>citoglipton</th>\n",
       "      <th>insulin</th>\n",
       "      <th>glyburide-metformin</th>\n",
       "      <th>glipizide-metformin</th>\n",
       "      <th>glimepiride-pioglitazone</th>\n",
       "      <th>metformin-rosiglitazone</th>\n",
       "      <th>metformin-pioglitazone</th>\n",
       "      <th>change</th>\n",
       "      <th>diabetesMed</th>\n",
       "      <th>readmitted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2278392</td>\n",
       "      <td>8222157</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>Female</td>\n",
       "      <td>[0-10)</td>\n",
       "      <td>?</td>\n",
       "      <td>6</td>\n",
       "      <td>25</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>149190</td>\n",
       "      <td>55629189</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>Female</td>\n",
       "      <td>[10-20)</td>\n",
       "      <td>?</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>Up</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Ch</td>\n",
       "      <td>Yes</td>\n",
       "      <td>&gt;30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>64410</td>\n",
       "      <td>86047875</td>\n",
       "      <td>AfricanAmerican</td>\n",
       "      <td>Female</td>\n",
       "      <td>[20-30)</td>\n",
       "      <td>?</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>500364</td>\n",
       "      <td>82442376</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>Male</td>\n",
       "      <td>[30-40)</td>\n",
       "      <td>?</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>Up</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Ch</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>16680</td>\n",
       "      <td>42519267</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>Male</td>\n",
       "      <td>[40-50)</td>\n",
       "      <td>?</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>Steady</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Ch</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 50 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   encounter_id  patient_nbr             race  gender      age weight  \\\n",
       "0       2278392      8222157        Caucasian  Female   [0-10)      ?   \n",
       "1        149190     55629189        Caucasian  Female  [10-20)      ?   \n",
       "2         64410     86047875  AfricanAmerican  Female  [20-30)      ?   \n",
       "3        500364     82442376        Caucasian    Male  [30-40)      ?   \n",
       "4         16680     42519267        Caucasian    Male  [40-50)      ?   \n",
       "\n",
       "   admission_type_id  discharge_disposition_id  admission_source_id  \\\n",
       "0                  6                        25                    1   \n",
       "1                  1                         1                    7   \n",
       "2                  1                         1                    7   \n",
       "3                  1                         1                    7   \n",
       "4                  1                         1                    7   \n",
       "\n",
       "   time_in_hospital  ... citoglipton insulin  glyburide-metformin  \\\n",
       "0                 1  ...          No      No                   No   \n",
       "1                 3  ...          No      Up                   No   \n",
       "2                 2  ...          No      No                   No   \n",
       "3                 2  ...          No      Up                   No   \n",
       "4                 1  ...          No  Steady                   No   \n",
       "\n",
       "   glipizide-metformin  glimepiride-pioglitazone  metformin-rosiglitazone  \\\n",
       "0                   No                        No                       No   \n",
       "1                   No                        No                       No   \n",
       "2                   No                        No                       No   \n",
       "3                   No                        No                       No   \n",
       "4                   No                        No                       No   \n",
       "\n",
       "   metformin-pioglitazone  change diabetesMed readmitted  \n",
       "0                      No      No          No         NO  \n",
       "1                      No      Ch         Yes        >30  \n",
       "2                      No      No         Yes         NO  \n",
       "3                      No      Ch         Yes         NO  \n",
       "4                      No      Ch         Yes         NO  \n",
       "\n",
       "[5 rows x 50 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hospital_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "opened-handling",
   "metadata": {},
   "source": [
    "Each sample in our dataset has 49 features. The two main types of features are categorical (`race`, `gender`), and numerical (`time_in_hospital`). To understand the features better, I'll look at the unique values for each feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "published-canada",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "encounter_id:\n",
      "101766 unique values\n",
      "patient_nbr:\n",
      "71518 unique values\n",
      "race:\n",
      "['Caucasian' 'AfricanAmerican' '?' 'Other' 'Asian' 'Hispanic']\n",
      "gender:\n",
      "['Female' 'Male' 'Unknown/Invalid']\n",
      "age:\n",
      "['[0-10)' '[10-20)' '[20-30)' '[30-40)' '[40-50)' '[50-60)' '[60-70)'\n",
      " '[70-80)' '[80-90)' '[90-100)']\n",
      "weight:\n",
      "['?' '[75-100)' '[50-75)' '[0-25)' '[100-125)' '[25-50)' '[125-150)'\n",
      " '[175-200)' '[150-175)' '>200']\n",
      "admission_type_id:\n",
      "[6 1 2 3 4 5 8 7]\n",
      "discharge_disposition_id:\n",
      "26 unique values\n",
      "admission_source_id:\n",
      "[ 1  7  2  4  5  6 20  3 17  8  9 14 10 22 11 25 13]\n",
      "time_in_hospital:\n",
      "[ 1  3  2  4  5 13 12  9  7 10  6 11  8 14]\n",
      "payer_code:\n",
      "['?' 'MC' 'MD' 'HM' 'UN' 'BC' 'SP' 'CP' 'SI' 'DM' 'CM' 'CH' 'PO' 'WC' 'OT'\n",
      " 'OG' 'MP' 'FR']\n",
      "medical_specialty:\n",
      "73 unique values\n",
      "num_lab_procedures:\n",
      "118 unique values\n",
      "num_procedures:\n",
      "[0 5 1 6 2 3 4]\n",
      "num_medications:\n",
      "75 unique values\n",
      "number_outpatient:\n",
      "39 unique values\n",
      "number_emergency:\n",
      "33 unique values\n",
      "number_inpatient:\n",
      "21 unique values\n",
      "diag_1:\n",
      "717 unique values\n",
      "diag_2:\n",
      "749 unique values\n",
      "diag_3:\n",
      "790 unique values\n",
      "number_diagnoses:\n",
      "[ 1  9  6  7  5  8  3  4  2 16 12 13 15 10 11 14]\n",
      "max_glu_serum:\n",
      "['None' '>300' 'Norm' '>200']\n",
      "A1Cresult:\n",
      "['None' '>7' '>8' 'Norm']\n",
      "metformin:\n",
      "['No' 'Steady' 'Up' 'Down']\n",
      "repaglinide:\n",
      "['No' 'Up' 'Steady' 'Down']\n",
      "nateglinide:\n",
      "['No' 'Steady' 'Down' 'Up']\n",
      "chlorpropamide:\n",
      "['No' 'Steady' 'Down' 'Up']\n",
      "glimepiride:\n",
      "['No' 'Steady' 'Down' 'Up']\n",
      "acetohexamide:\n",
      "['No' 'Steady']\n",
      "glipizide:\n",
      "['No' 'Steady' 'Up' 'Down']\n",
      "glyburide:\n",
      "['No' 'Steady' 'Up' 'Down']\n",
      "tolbutamide:\n",
      "['No' 'Steady']\n",
      "pioglitazone:\n",
      "['No' 'Steady' 'Up' 'Down']\n",
      "rosiglitazone:\n",
      "['No' 'Steady' 'Up' 'Down']\n",
      "acarbose:\n",
      "['No' 'Steady' 'Up' 'Down']\n",
      "miglitol:\n",
      "['No' 'Steady' 'Down' 'Up']\n",
      "troglitazone:\n",
      "['No' 'Steady']\n",
      "tolazamide:\n",
      "['No' 'Steady' 'Up']\n",
      "examide:\n",
      "['No']\n",
      "citoglipton:\n",
      "['No']\n",
      "insulin:\n",
      "['No' 'Up' 'Steady' 'Down']\n",
      "glyburide-metformin:\n",
      "['No' 'Steady' 'Down' 'Up']\n",
      "glipizide-metformin:\n",
      "['No' 'Steady']\n",
      "glimepiride-pioglitazone:\n",
      "['No' 'Steady']\n",
      "metformin-rosiglitazone:\n",
      "['No' 'Steady']\n",
      "metformin-pioglitazone:\n",
      "['No' 'Steady']\n",
      "change:\n",
      "['No' 'Ch']\n",
      "diabetesMed:\n",
      "['No' 'Yes']\n",
      "readmitted:\n",
      "['NO' '>30' '<30']\n"
     ]
    }
   ],
   "source": [
    "for column in list(hospital_df):\n",
    "    unique_vals = hospital_df[column].unique()\n",
    "    \n",
    "    if len(unique_vals) < 20:\n",
    "        print(column + ':')\n",
    "        print(unique_vals)\n",
    "    else:\n",
    "        print(column + ':')\n",
    "        print(str(len(unique_vals)) + ' unique values')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rapid-payday",
   "metadata": {},
   "source": [
    "Using the information from IDs_mapping.csv, we notice that IDs 11, 13, 14, 19, 20, and 21 in the `discharge_disposition_id` variable relate to death or hospice. The samples with these discharge_disposition_ids are not related to our predictive task, so we will remove these samples from our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "trained-thousand",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number samples:  99343\n"
     ]
    }
   ],
   "source": [
    "hospital_df = hospital_df.loc[~hospital_df.discharge_disposition_id.isin([11, 13, 14, 19, 20, 21])]\n",
    "print('Total number samples: ', len(hospital_df))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "exotic-merit",
   "metadata": {},
   "source": [
    "We define a output label for our classification task. If the patient is readmitted within 30 days, the output  label is 1, otherwise the output label is 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "posted-smoke",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>readmitted</th>\n",
       "      <th>output_label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NO</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&gt;30</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NO</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NO</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NO</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  readmitted  output_label\n",
       "0         NO             0\n",
       "1        >30             0\n",
       "2         NO             0\n",
       "3         NO             0\n",
       "4         NO             0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hospital_df['output_label'] = (hospital_df['readmitted'] == '<30').astype('int')\n",
    "hospital_df[['readmitted', 'output_label']].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "direct-mechanics",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prevalance of readmitted patients: 0.11388824577474005\n"
     ]
    }
   ],
   "source": [
    "print('Prevalance of readmitted patients:', \n",
    "      sum(hospital_df['output_label'].values) / len(hospital_df['output_label'].values))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "nonprofit-letters",
   "metadata": {},
   "source": [
    "### Takeaways\n",
    "- In this dataset age and weight are categorical variables (ranges) and not numerical variables (specific values)\n",
    "- We will not use the `examide` or `citoglipton` feature because it only has a single value, 'No'\n",
    "- Most of the features have between 2-20 unique values. A few of the features such as `medical_speciality` and `num_lab_procedures` have more than 20 unique values\n",
    "- 11.38% of the sample population is readmitted. This represents an imbalanced classification problem."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "republican-teach",
   "metadata": {},
   "source": [
    "## Feature Extraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "forty-livestock",
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace ? with nan\n",
    "hospital_df = hospital_df.replace('?', np.nan)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "waiting-nightlife",
   "metadata": {},
   "source": [
    "### Numerical features\n",
    "These features have specific integer values and do not need to be modified. The columns that have numerical features are shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "remarkable-major",
   "metadata": {},
   "outputs": [],
   "source": [
    "numerical_cols = ['time_in_hospital', 'num_lab_procedures', 'num_procedures', 'num_medications',\n",
    "                  'number_outpatient', 'number_emergency', 'number_inpatient', 'number_diagnoses']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bizarre-ground",
   "metadata": {},
   "source": [
    "We want to ensure there are no missing values in the numerical features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "dressed-alcohol",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "time_in_hospital      0\n",
       "num_lab_procedures    0\n",
       "num_procedures        0\n",
       "num_medications       0\n",
       "number_outpatient     0\n",
       "number_emergency      0\n",
       "number_inpatient      0\n",
       "number_diagnoses      0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hospital_df[numerical_cols].isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "level-startup",
   "metadata": {},
   "source": [
    "The `age` and `weight` columns are categorical ranges, but we want to convert them to numerical data. We use a dictionary to map the age ranges to specific integer values and since most samples do not have a weight measurement, we create a `has_weight` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "enhanced-racing",
   "metadata": {},
   "outputs": [],
   "source": [
    "age_id = {'[0-10)': 0, \n",
    "          '[10-20)': 10, \n",
    "          '[20-30)': 20, \n",
    "          '[30-40)': 30, \n",
    "          '[40-50)': 40, \n",
    "          '[50-60)': 50,\n",
    "          '[60-70)': 60, \n",
    "          '[70-80)': 70, \n",
    "          '[80-90)': 80, \n",
    "          '[90-100)': 90}\n",
    "hospital_df['age_group'] = hospital_df.age.replace(age_id)\n",
    "\n",
    "hospital_df['has_weight'] = hospital_df.weight.notnull().astype('int')\n",
    "numerical_cols.append('age_group')\n",
    "numerical_cols.append('has_weight')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "regular-wagon",
   "metadata": {},
   "source": [
    "### Categorical features\n",
    "Categorical features are qualitative, non-numeric data such as gender or race. We use one-hot encoding to turn the non-numeric data into numeric variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "billion-donna",
   "metadata": {},
   "outputs": [],
   "source": [
    "medicines = list(hospital_df.columns)[21:46]\n",
    "medicines.remove('examide')\n",
    "medicines.remove('citoglipton')\n",
    "categorical_cols = ['race', 'gender', 'payer_code', 'medical_specialty'] + medicines + ['change', 'diabetesMed'] "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ignored-slave",
   "metadata": {},
   "source": [
    "We want to ensure there are no missing values in the numerical features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "brazilian-sandwich",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "race                         2234\n",
       "gender                          0\n",
       "payer_code                  39398\n",
       "medical_specialty           48616\n",
       "number_diagnoses                0\n",
       "max_glu_serum                   0\n",
       "A1Cresult                       0\n",
       "metformin                       0\n",
       "repaglinide                     0\n",
       "nateglinide                     0\n",
       "chlorpropamide                  0\n",
       "glimepiride                     0\n",
       "acetohexamide                   0\n",
       "glipizide                       0\n",
       "glyburide                       0\n",
       "tolbutamide                     0\n",
       "pioglitazone                    0\n",
       "rosiglitazone                   0\n",
       "acarbose                        0\n",
       "miglitol                        0\n",
       "troglitazone                    0\n",
       "tolazamide                      0\n",
       "insulin                         0\n",
       "glyburide-metformin             0\n",
       "glipizide-metformin             0\n",
       "glimepiride-pioglitazone        0\n",
       "metformin-rosiglitazone         0\n",
       "change                          0\n",
       "diabetesMed                     0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hospital_df[categorical_cols].isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "respiratory-alaska",
   "metadata": {},
   "source": [
    "`race`, `payer_code`, and `medical_specialty` columns all have missing data. We replace these unknown with an UNK token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "labeled-singles",
   "metadata": {},
   "outputs": [],
   "source": [
    "hospital_df['race'] = hospital_df['race'].fillna('UNK')\n",
    "hospital_df['payer_code'] = hospital_df['payer_code'].fillna('UNK')\n",
    "hospital_df['medical_specialty'] = hospital_df['medical_specialty'].fillna('UNK')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "conceptual-democrat",
   "metadata": {},
   "source": [
    "Since we want to change the categorical features to one-hot encoding, we want to limit sparsity. The `race` and `payer_code` columns have only a few unique values, but the `medical_speciality` column has 73 unique values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "suspected-electric",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "medical_specialty\n",
       "UNK                           48616\n",
       "InternalMedicine              14237\n",
       "Emergency/Trauma               7419\n",
       "Family/GeneralPractice         7252\n",
       "Cardiology                     5279\n",
       "Surgery-General                3059\n",
       "Nephrology                     1539\n",
       "Orthopedics                    1392\n",
       "Orthopedics-Reconstructive     1230\n",
       "Radiologist                    1121\n",
       "dtype: int64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hospital_df.groupby('medical_specialty').size().sort_values(ascending = False)[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lovely-professional",
   "metadata": {},
   "source": [
    "Most of the samples are categorized as the UNK token and the counts drop off quickly. We want to reduce the number of unique values in the `medical_speciality` column so we only keep the top 10 most common categories, and add a 'Other' value for the remaining categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "funded-community",
   "metadata": {},
   "outputs": [],
   "source": [
    "top_10 = list(hospital_df.groupby('medical_specialty').size().sort_values(ascending = False)[:10].index)\n",
    "\n",
    "hospital_df['med_spec'] = hospital_df['medical_specialty']\n",
    "\n",
    "# replace all specialties not in the top 10 with the 'Other' category\n",
    "hospital_df.loc[~hospital_df.med_spec.isin(top_10), 'med_spec'] = 'Other'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "aggressive-dialogue",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "med_spec\n",
       "UNK                           48616\n",
       "InternalMedicine              14237\n",
       "Other                          8199\n",
       "Emergency/Trauma               7419\n",
       "Family/GeneralPractice         7252\n",
       "Cardiology                     5279\n",
       "Surgery-General                3059\n",
       "Nephrology                     1539\n",
       "Orthopedics                    1392\n",
       "Orthopedics-Reconstructive     1230\n",
       "Radiologist                    1121\n",
       "dtype: int64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hospital_df.groupby('med_spec').size().sort_values(ascending = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "demographic-blond",
   "metadata": {},
   "source": [
    "Use the get_dummies function to create the one-hot encoding representation of our categorical features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "identified-blend",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert categorical numbers to strings for get_dummies function\n",
    "categorical_num_cols = ['admission_type_id', 'discharge_disposition_id', 'admission_source_id']\n",
    "hospital_df[categorical_num_cols] = hospital_df[categorical_num_cols].astype('str')\n",
    "\n",
    "categorical_cols.remove('medical_specialty')\n",
    "df_encoded = pd.get_dummies(hospital_df[categorical_cols + categorical_num_cols + ['med_spec']], drop_first = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "comic-import",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>number_diagnoses</th>\n",
       "      <th>race_Asian</th>\n",
       "      <th>race_Caucasian</th>\n",
       "      <th>race_Hispanic</th>\n",
       "      <th>race_Other</th>\n",
       "      <th>race_UNK</th>\n",
       "      <th>gender_Male</th>\n",
       "      <th>gender_Unknown/Invalid</th>\n",
       "      <th>payer_code_CH</th>\n",
       "      <th>payer_code_CM</th>\n",
       "      <th>...</th>\n",
       "      <th>med_spec_Emergency/Trauma</th>\n",
       "      <th>med_spec_Family/GeneralPractice</th>\n",
       "      <th>med_spec_InternalMedicine</th>\n",
       "      <th>med_spec_Nephrology</th>\n",
       "      <th>med_spec_Orthopedics</th>\n",
       "      <th>med_spec_Orthopedics-Reconstructive</th>\n",
       "      <th>med_spec_Other</th>\n",
       "      <th>med_spec_Radiologist</th>\n",
       "      <th>med_spec_Surgery-General</th>\n",
       "      <th>med_spec_UNK</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 133 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   number_diagnoses  race_Asian  race_Caucasian  race_Hispanic  race_Other  \\\n",
       "0                 1           0               1              0           0   \n",
       "1                 9           0               1              0           0   \n",
       "2                 6           0               0              0           0   \n",
       "3                 7           0               1              0           0   \n",
       "4                 5           0               1              0           0   \n",
       "\n",
       "   race_UNK  gender_Male  gender_Unknown/Invalid  payer_code_CH  \\\n",
       "0         0            0                       0              0   \n",
       "1         0            0                       0              0   \n",
       "2         0            0                       0              0   \n",
       "3         0            1                       0              0   \n",
       "4         0            1                       0              0   \n",
       "\n",
       "   payer_code_CM  ...  med_spec_Emergency/Trauma  \\\n",
       "0              0  ...                          0   \n",
       "1              0  ...                          0   \n",
       "2              0  ...                          0   \n",
       "3              0  ...                          0   \n",
       "4              0  ...                          0   \n",
       "\n",
       "   med_spec_Family/GeneralPractice  med_spec_InternalMedicine  \\\n",
       "0                                0                          0   \n",
       "1                                0                          0   \n",
       "2                                0                          0   \n",
       "3                                0                          0   \n",
       "4                                0                          0   \n",
       "\n",
       "   med_spec_Nephrology  med_spec_Orthopedics  \\\n",
       "0                    0                     0   \n",
       "1                    0                     0   \n",
       "2                    0                     0   \n",
       "3                    0                     0   \n",
       "4                    0                     0   \n",
       "\n",
       "   med_spec_Orthopedics-Reconstructive  med_spec_Other  med_spec_Radiologist  \\\n",
       "0                                    0               1                     0   \n",
       "1                                    0               0                     0   \n",
       "2                                    0               0                     0   \n",
       "3                                    0               0                     0   \n",
       "4                                    0               0                     0   \n",
       "\n",
       "   med_spec_Surgery-General  med_spec_UNK  \n",
       "0                         0             0  \n",
       "1                         0             1  \n",
       "2                         0             1  \n",
       "3                         0             1  \n",
       "4                         0             1  \n",
       "\n",
       "[5 rows x 133 columns]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_encoded.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "working-population",
   "metadata": {},
   "outputs": [],
   "source": [
    "# add one-hot encoding data to original data\n",
    "hospital_df = pd.concat([hospital_df, df_encoded], axis = 1)\n",
    "all_categorical_cols = list(df_encoded.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "textile-memphis",
   "metadata": {},
   "source": [
    "We create a new dataframe that only has our desired features and the output label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "executed-cemetery",
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_cols = numerical_cols + all_categorical_cols\n",
    "hospital_data = hospital_df[feature_cols + ['output_label']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "sustainable-princeton",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time_in_hospital</th>\n",
       "      <th>num_lab_procedures</th>\n",
       "      <th>num_procedures</th>\n",
       "      <th>num_medications</th>\n",
       "      <th>number_outpatient</th>\n",
       "      <th>number_emergency</th>\n",
       "      <th>number_inpatient</th>\n",
       "      <th>number_diagnoses</th>\n",
       "      <th>number_diagnoses</th>\n",
       "      <th>age_group</th>\n",
       "      <th>...</th>\n",
       "      <th>med_spec_Family/GeneralPractice</th>\n",
       "      <th>med_spec_InternalMedicine</th>\n",
       "      <th>med_spec_Nephrology</th>\n",
       "      <th>med_spec_Orthopedics</th>\n",
       "      <th>med_spec_Orthopedics-Reconstructive</th>\n",
       "      <th>med_spec_Other</th>\n",
       "      <th>med_spec_Radiologist</th>\n",
       "      <th>med_spec_Surgery-General</th>\n",
       "      <th>med_spec_UNK</th>\n",
       "      <th>output_label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>59</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "      <td>13</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>20</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>44</td>\n",
       "      <td>1</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>30</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>51</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>40</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 146 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   time_in_hospital  num_lab_procedures  num_procedures  num_medications  \\\n",
       "0                 1                  41               0                1   \n",
       "1                 3                  59               0               18   \n",
       "2                 2                  11               5               13   \n",
       "3                 2                  44               1               16   \n",
       "4                 1                  51               0                8   \n",
       "\n",
       "   number_outpatient  number_emergency  number_inpatient  number_diagnoses  \\\n",
       "0                  0                 0                 0                 1   \n",
       "1                  0                 0                 0                 9   \n",
       "2                  2                 0                 1                 6   \n",
       "3                  0                 0                 0                 7   \n",
       "4                  0                 0                 0                 5   \n",
       "\n",
       "   number_diagnoses  age_group  ...  med_spec_Family/GeneralPractice  \\\n",
       "0                 1          0  ...                                0   \n",
       "1                 9         10  ...                                0   \n",
       "2                 6         20  ...                                0   \n",
       "3                 7         30  ...                                0   \n",
       "4                 5         40  ...                                0   \n",
       "\n",
       "   med_spec_InternalMedicine  med_spec_Nephrology  med_spec_Orthopedics  \\\n",
       "0                          0                    0                     0   \n",
       "1                          0                    0                     0   \n",
       "2                          0                    0                     0   \n",
       "3                          0                    0                     0   \n",
       "4                          0                    0                     0   \n",
       "\n",
       "   med_spec_Orthopedics-Reconstructive  med_spec_Other  med_spec_Radiologist  \\\n",
       "0                                    0               1                     0   \n",
       "1                                    0               0                     0   \n",
       "2                                    0               0                     0   \n",
       "3                                    0               0                     0   \n",
       "4                                    0               0                     0   \n",
       "\n",
       "   med_spec_Surgery-General  med_spec_UNK  output_label  \n",
       "0                         0             0             0  \n",
       "1                         0             1             0  \n",
       "2                         0             1             0  \n",
       "3                         0             1             0  \n",
       "4                         0             1             0  \n",
       "\n",
       "[5 rows x 146 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hospital_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "negative-score",
   "metadata": {},
   "source": [
    "## Creating Training / Validation / Test Sets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "anticipated-antigua",
   "metadata": {},
   "source": [
    "The data will be split into 70% training, 15% validation, and 15% test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "green-phoenix",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data prevalence (n = 69540): 0.11392004601668104\n",
      "Validation data prevalence (n = 14901): 0.11408630293268908\n",
      "Test data prevalence (n = 14902): 0.11354180646893035\n"
     ]
    }
   ],
   "source": [
    "# Shuffle the samples\n",
    "hospital_data = hospital_data.sample(n = len(hospital_data), random_state = 5)\n",
    "hospital_data = hospital_data.reset_index(drop = True)\n",
    "\n",
    "# Save 30% for val / test\n",
    "hospital_val_test = hospital_data.sample(frac = 0.3, random_state = 5)\n",
    "test_data = hospital_val_test.sample(frac = 0.5, random_state = 5)\n",
    "val_data = hospital_val_test.drop(test_data.index)\n",
    "\n",
    "# Use remaining data as training data\n",
    "training_data = hospital_data.drop(hospital_val_test.index)\n",
    "\n",
    "# Sanity check\n",
    "print('Training data prevalence (n = ' + str(len(training_data)) + '): ' + \n",
    "      str(sum(training_data['output_label'].values) / len(training_data['output_label'].values)))\n",
    "\n",
    "print('Validation data prevalence (n = ' + str(len(val_data)) + '): ' + \n",
    "      str(sum(val_data['output_label'].values) / len(val_data['output_label'].values)))\n",
    "\n",
    "print('Test data prevalence (n = ' + str(len(test_data)) + '): ' + \n",
    "      str(sum(test_data['output_label'].values) / len(test_data['output_label'].values)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "flying-paper",
   "metadata": {},
   "source": [
    "Since the data is unbalanced, we will use a sub-sampling approach to create a balanced training data set with 50% readmissions (positive) and 50% non-readmissions (negative)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "reverse-banking",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data prevalence (n = 15844): 0.5\n"
     ]
    }
   ],
   "source": [
    "# Split training data into positive and negative\n",
    "pos_rows = training_data.output_label == 1\n",
    "train_pos = training_data.loc[pos_rows]\n",
    "train_neg = training_data.loc[~pos_rows]\n",
    "\n",
    "# Merging data in balanced way and shuffle\n",
    "training_data = pd.concat([train_pos, train_neg.sample(len(train_pos), random_state = 5)], axis = 0)\n",
    "training_data = training_data.sample(n = len(training_data), random_state = 5).reset_index(drop = True)\n",
    "\n",
    "print('Training data prevalence (n = ' + str(len(training_data)) + '): ' + \n",
    "      str(sum(training_data['output_label'].values) / len(training_data['output_label'].values)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "electrical-kernel",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time_in_hospital</th>\n",
       "      <th>num_lab_procedures</th>\n",
       "      <th>num_procedures</th>\n",
       "      <th>num_medications</th>\n",
       "      <th>number_outpatient</th>\n",
       "      <th>number_emergency</th>\n",
       "      <th>number_inpatient</th>\n",
       "      <th>number_diagnoses</th>\n",
       "      <th>number_diagnoses</th>\n",
       "      <th>age_group</th>\n",
       "      <th>...</th>\n",
       "      <th>med_spec_Family/GeneralPractice</th>\n",
       "      <th>med_spec_InternalMedicine</th>\n",
       "      <th>med_spec_Nephrology</th>\n",
       "      <th>med_spec_Orthopedics</th>\n",
       "      <th>med_spec_Orthopedics-Reconstructive</th>\n",
       "      <th>med_spec_Other</th>\n",
       "      <th>med_spec_Radiologist</th>\n",
       "      <th>med_spec_Surgery-General</th>\n",
       "      <th>med_spec_UNK</th>\n",
       "      <th>output_label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>37</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>80</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9</td>\n",
       "      <td>80</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>80</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>55</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>70</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>43</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>60</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>25</td>\n",
       "      <td>1</td>\n",
       "      <td>14</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>40</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 146 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   time_in_hospital  num_lab_procedures  num_procedures  num_medications  \\\n",
       "0                 3                  37               0               12   \n",
       "1                 9                  80               0               16   \n",
       "2                 5                  55               2               12   \n",
       "3                 3                  43               0               15   \n",
       "4                 1                  25               1               14   \n",
       "\n",
       "   number_outpatient  number_emergency  number_inpatient  number_diagnoses  \\\n",
       "0                  0                 0                 2                 6   \n",
       "1                  0                 0                 0                 6   \n",
       "2                  0                 0                 1                 6   \n",
       "3                  2                 0                 1                 8   \n",
       "4                  0                 0                 0                 5   \n",
       "\n",
       "   number_diagnoses  age_group  ...  med_spec_Family/GeneralPractice  \\\n",
       "0                 6         80  ...                                1   \n",
       "1                 6         80  ...                                0   \n",
       "2                 6         70  ...                                1   \n",
       "3                 8         60  ...                                0   \n",
       "4                 5         40  ...                                0   \n",
       "\n",
       "   med_spec_InternalMedicine  med_spec_Nephrology  med_spec_Orthopedics  \\\n",
       "0                          0                    0                     0   \n",
       "1                          1                    0                     0   \n",
       "2                          0                    0                     0   \n",
       "3                          0                    1                     0   \n",
       "4                          0                    0                     0   \n",
       "\n",
       "   med_spec_Orthopedics-Reconstructive  med_spec_Other  med_spec_Radiologist  \\\n",
       "0                                    0               0                     0   \n",
       "1                                    0               0                     0   \n",
       "2                                    0               0                     0   \n",
       "3                                    0               0                     0   \n",
       "4                                    0               1                     0   \n",
       "\n",
       "   med_spec_Surgery-General  med_spec_UNK  output_label  \n",
       "0                         0             0             0  \n",
       "1                         0             0             1  \n",
       "2                         0             0             1  \n",
       "3                         0             0             1  \n",
       "4                         0             0             0  \n",
       "\n",
       "[5 rows x 146 columns]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "numeric-success",
   "metadata": {},
   "source": [
    "Machine learning packages use input matrix X and output vector y, so we manipulate our data to fit this format. We use sci-kit learn's Standard Scaler function to scale our data to unit variance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "familiar-jersey",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data shapes: (15844, 149) (15844,)\n",
      "Validation data shapes: (14901, 149) (14901,)\n"
     ]
    }
   ],
   "source": [
    "X_all = hospital_data[feature_cols].values\n",
    "X_train = training_data[feature_cols].values\n",
    "X_val = val_data[feature_cols].values\n",
    "\n",
    "y_train = training_data['output_label'].values\n",
    "y_val = val_data['output_label'].values\n",
    "\n",
    "print('Training data shapes:', X_train.shape, y_train.shape)\n",
    "print('Validation data shapes:', X_val.shape, y_val.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "varying-timer",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "scaler = StandardScaler()\n",
    "scaler.fit(X_all)\n",
    "\n",
    "X_train_tf = scaler.transform(X_train)\n",
    "X_val_tf = scaler.transform(X_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "compact-orchestra",
   "metadata": {},
   "source": [
    "With out transformed, normalized training features and labels, we can proceed to selecting a model for classification."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "intensive-matter",
   "metadata": {},
   "source": [
    "## Model Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "complicated-garlic",
   "metadata": {},
   "source": [
    "In this section, I will compare the K-nearest neighbors model, logistic regression model, and the decision tree classifier. Our metrics for comparing the models will be the AUC score, accuracy score, precision score, and recall score. It's important to note that guessing 'not readmitted' for every sample would result in a 89% accuracy rate, yet the recall would be 0%, since we would never predict a readmission. Guessing randomly would result in a 50% accuracy rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "graduate-penny",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score, accuracy_score, precision_score, recall_score\n",
    "\n",
    "def print_report(y_actual, y_pred, threshold = 0.5):\n",
    "    auc = roc_auc_score(y_actual, y_pred)\n",
    "    accuracy = accuracy_score(y_actual, (y_pred > threshold))\n",
    "    recall = recall_score(y_actual, (y_pred > threshold))\n",
    "    precision = precision_score(y_actual, (y_pred > threshold))\n",
    "    print('AUC: ', auc)\n",
    "    print('accuracy: ', accuracy)\n",
    "    print('recall: ', recall)\n",
    "    print('precision: ', precision)\n",
    "    return auc, accuracy, recall, precision"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "established-floating",
   "metadata": {},
   "source": [
    "### K-nearest neighbors (KNN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "wooden-tracker",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier(n_neighbors=100)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "knn = KNeighborsClassifier(n_neighbors = 100)\n",
    "knn.fit(X_train_tf, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "educational-listing",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KNN\n",
      "Training:\n",
      "AUC:  0.6471683074327126\n",
      "accuracy:  0.6006690229739965\n",
      "recall:  0.5059328452411007\n",
      "precision:  0.6242018377199813\n",
      "\n",
      "Validation:\n",
      "AUC:  0.633711283013319\n",
      "accuracy:  0.6600899268505469\n",
      "recall:  0.5141176470588236\n",
      "precision:  0.17093682769411306\n"
     ]
    }
   ],
   "source": [
    "y_train_pred = knn.predict_proba(X_train_tf)[:, 1]\n",
    "y_val_pred = knn.predict_proba(X_val_tf)[:, 1]\n",
    "\n",
    "print('KNN')\n",
    "print('Training:')\n",
    "knn_train_auc, knn_train_acc, knn_train_recall, knn_train_prec = print_report(y_train, y_train_pred)\n",
    "print()\n",
    "print('Validation:')\n",
    "knn_val_auc, knn_val_acc, knn_val_recall, knn_val_prec = print_report(y_val, y_val_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "first-consumer",
   "metadata": {},
   "source": [
    "### SGD (stochastic gradient descent) Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "voluntary-drilling",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGDClassifier(alpha=0.1, loss='log', random_state=5)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import SGDClassifier\n",
    "sgdc = SGDClassifier(loss = 'log', alpha = 0.1, random_state = 5)\n",
    "\n",
    "sgdc.fit(X_train_tf, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "associate-catering",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SGD Logistic Regression\n",
      "Training:\n",
      "AUC:  0.6663677304106352\n",
      "accuracy:  0.6197298661954052\n",
      "recall:  0.5568038374147942\n",
      "precision:  0.6369675090252708\n",
      "\n",
      "Validation:\n",
      "AUC:  0.6792550475231378\n",
      "accuracy:  0.6630427488088048\n",
      "recall:  0.5741176470588235\n",
      "precision:  0.18509387445476957\n"
     ]
    }
   ],
   "source": [
    "y_train_pred = sgdc.predict_proba(X_train_tf)[:, 1]\n",
    "y_val_pred = sgdc.predict_proba(X_val_tf)[:, 1]\n",
    "\n",
    "print('SGD Logistic Regression')\n",
    "print('Training:')\n",
    "sgd_train_auc, sgd_train_acc, sgd_train_recall, sgd_train_prec = print_report(y_train, y_train_pred)\n",
    "print()\n",
    "print('Validation:')\n",
    "sgd_val_auc, sgd_val_acc, sgd_val_recall, sgd_val_prec = print_report(y_val, y_val_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "employed-consent",
   "metadata": {},
   "source": [
    "### Decision Tree Classifier\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "scenic-receipt",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(max_depth=10, random_state=5)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "tree = DecisionTreeClassifier(max_depth = 10, random_state = 5)\n",
    "tree.fit(X_train_tf, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "protecting-station",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision Tree Classifier\n",
      "Training:\n",
      "AUC:  0.7377043728740985\n",
      "accuracy:  0.6665614743751578\n",
      "recall:  0.6055289068417067\n",
      "precision:  0.6897196261682244\n",
      "\n",
      "Validation:\n",
      "AUC:  0.639067027007758\n",
      "accuracy:  0.6488826253271592\n",
      "recall:  0.5547058823529412\n",
      "precision:  0.17404946474713917\n"
     ]
    }
   ],
   "source": [
    "y_train_pred = tree.predict_proba(X_train_tf)[:, 1]\n",
    "y_val_pred = tree.predict_proba(X_val_tf)[:, 1]\n",
    "\n",
    "print('Decision Tree Classifier')\n",
    "print('Training:')\n",
    "tree_train_auc, tree_train_acc, tree_train_recall, tree_train_prec = print_report(y_train, y_train_pred)\n",
    "print()\n",
    "print('Validation:')\n",
    "tree_val_auc, tree_val_acc, tree_val_recall, tree_val_prec = print_report(y_val, y_val_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "animated-thanks",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfMAAAERCAYAAABvg07CAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAl7ElEQVR4nO3deVxWZf7/8Tc7KoSoQKLCzC8NExfQxEysXDLJ3NKa1Fwrt5JRzG00Ss1SUtRwIbeMwd1B07RS+5pmM+mQlo7r6KgJiYK7iNzCze8Pp3u6BQQEbjj6ej4ePB7c17nOOZ/7Psa7c51zn8suJycnRwAAwLDsy7oAAABQPIQ5AAAGR5gDAGBwhDkAAAZHmAMAYHCEOQAABudY1gUAAO5PV65c0blz53Xr1q2yLsXQnJyc5OPjLQ8Pj3z7PBBhfulSusxmvk4PAAWxt7eTp2elYm/nypUrOns2RZUrV5Ozs4vs7OxKoLoHT05OjkymTJ09myJJ+Qb6AxHmZnMOYQ4ANnTu3HlVrlxNLi6uZV2KodnZ2cnFxVWVK1fTuXPn8w1zrpkDAErcrVu35OzsUtZl3DecnV3uermCMAcAlAqG1ktOQZ8lYQ4AgME9ENfMAQDlQ8VKLnJxtn30ZJqydCM9s0jrnDz5H/36a7JatGh5T/ucNOldnT9/TnPmxN7T+kVBmAMAbMbF2VE9Ry+z+X6XR/UqcpiPGjVCzz0Xds9hHhHxts1uvibMAQDIU/GC2M3NvYTqKBhhDgDAHYYMeUNJSUlavHiBNm3aKElq3bqtdu3aqatXr2jWrLl66CEPzZkzSz/+mKjr16/Ly8tL3bu/rFdf7SvJepj9xx8TNWLEW5o8+UPNmxejX39N1h//+P8UETFaQUHBxa6XMAdQ5twfcpWri1NZl1EkNzNv6drVm2VdBkrJ1KnT1a9fL7Vq1Ua9e/dT//6vKiFhjaKjY+Ts7KxHHw1Q796v6OGHq2vu3E/k4uKiL7/cpDlzZisk5Ak9+mhArm3eunVLixcv0LhxE+ThUVlRUR/q/fff05o164t95z9hDqDMubo4lcl11OJYHtVL10SY3688PDzk4GCvChUqyNPTU5LUsuXTaty4iSTp5s2bev75jnr22efk7e0tSRow4A0tXbpYJ078O88wz8nJ0eDBbykoqLEkqUePXho9OkKXL1+27ONeEeYAABSCr28Ny++urq566aU/adu2LTp06F86c+YXHTt2TGazWdnZ5ny34efnZ/m9UiU3SSqRZ9cT5gAAFMLvH01748YNDR78mrKzs9WqVRs1btxU9evXV5cuHe66DWdn5zxai3/HO2EOAECe8r+O/cMP/9CxY0f19dfbLc9LP336lMxms0oinIuKJ8ABAJCHihUr6cyZX5SampprmY+PjyTp66+/1Nmzv+qf/9yt8ePHSJJMJttP+cqZOQAAeejRo5eio6O0e/cPcnW1nv0tMLC+hg0brr/+danmzv1YDz/8sDp27Kx//ON7HT58UFJ3m9Zql5OTc9/PDXrhwnWmQAXKMS8vd0PezZ6aeq2syyhx9vZ2qlrVrdjbOXjwkHx9/XO1G+lxruXNr7+eVmBgvTyXcWYOALCZG+mZhg/V8ohr5gAAGBxhDgCAwRHmAAAYHGEOAIDBEeYAABgcYQ4AgMER5gAAGBxhDgCAwfHQGKCMeHo4y9HZpazLKJIsU6YuXTGVdRkA7kCYA2XE0dlFP0a9XtZlFEmT0YskEea4d+6VnOSY5zSgpSvLZNK1dNtMgPLjj4l6882B2rDhS3l7+6hLlw7q1KmrBgzI+7/3KVMmKSnpjObPX3jP+yTMAQA24+jsXCb/E9tk9CLJRmF+p08/jc81UUtJI8xxX3B/yFWuLk5lXQYA5OLp6Vnq+yDMcV9wdXEy5KxbAMqnSZPe1dmzv1oNfR86dFADBvTWqlUJ+uabbdq8eaNSUs7K1dVVjz8eojFjxucZ3HcOs69du0rLlv1VFy9e1DPPtFJJTF7K3ewAANyhQ4cX9PPP+3T+/HlL29dfb1aDBg21a9dOrV69XCNHjtaaNes1adKH+vnnn7R06aICt/vll19o9uxo9e07QHFxy+Xt7aMtW74qdr2EOQAAd2jc+HF5e/to27YtkqTs7Gxt3bpFzz//gvz8/BUZOUnNm7dQ9eq+evLJFmre/EmdOHG8wO2uWbNa7ds/ry5dXpS//x/05pvhqlcvsNj1EuYAANzBzs5OYWEdtHXr7bPmxMQ9un79mtq2fU4tWz4td3d3zZ8/R+PGjVLPni/pq682KzvbXOB2//Of4woIeMyqLTCwQbHrJcwBAMhDWNgLOnz4kH755Rd9/fVXlhBfsmSRwsOHKj09Xc2bt1Bk5CS1b/98obZpZ2cnyfoauZNT8W/e5Qa43zHaHdE3M2/p2tWbZV0GANyX/Pz81KBBI23b9rV27tyu9957X5IUH79Ub7wxRD17vmrpe+bML3J0LDhS69QJ0P79+9W9+58sbYcPHyp2rYT57xjtjujlUb10TYQ5AJSWDh1e0Mcfz5KLi4ueeOJJSZK3t4927/67nnyyhcxmsxIS1ujAgf0KDKxf4PZ69eqtv/xltAIDA9W8eQtt27ZV+/f/pIYNg4pVJ2EOAPfAnHVLXl7uZV1GoZWXR/FmmUz/fZKg7fd7L9q0aaeZM6fruec6W8683313sj76aKr69u0pd3d3BQc30dCh4Vq6dLFu3sy46/aefrqVJkx4T0uWLNTcuR+radMQderUVadOnbyn+n5DmBuY0f6YSOXnDwpQXPaOToZ6HG95eRTvtfRbZfYktnvh7u6uHTv+YdX22GP1tGRJXK6+ffr0kyQ1afK4fvhhr6V9/fpNVv3CwjooLKxDidZJmBuY0f6YSOXnDwoA3E+4mx0AAIMjzAEAMDjCHAAAg7NpmGdnZ2vGjBkKDQ1VcHCwwsPDlZaWlm//lJQUhYeHKzg4WM2bN9d7772njIy73ykIACgfSmICEdxW0Gdp0zCPiYnRunXrNG3aNMXHxyslJUXDhg3Ls6/JZFL//v11+fJlrVixQjNnztS3336rjz76yJYlAwDugZOTk0ymzLIu475hMmXe9UlxNrub3WQyKS4uThMmTFCLFi0kSdHR0WrTpo327t2rxo0bW/XfuHGjUlNTtXLlSnl4eEiShg0bphUrVtiqZADAPfLx8dbZsymqXLmanJ1d/vsYUxRVTk6OTKZMXb6cpurVH863n83C/MiRI0pPT1dISIilrWbNmqpRo4YSExNzhfmuXbv05JNPWoJckrp166Zu3brZqmQAwD367W/3uXPndeuWcb5XXh45OTmpevWHrfLwTjYL85SUFEmSj4+PVbu3t7dl2e+dOnVKTzzxhGbNmqUNGzbIzs5O7dq10/Dhw+Xi4mKTmgEA987Dw+OuAYSSY7Mwz8jIkL29fa4xf2dnZ2Vm5r6ucv36da1du1ZPPfWUZs+erXPnzmny5Mm6cOGCoqKiirTvqlXdilU7SpbRnloHaxw/4+LY3b9sFuaurq4ym83KysqymlnGZDKpQoUKuQtzdJSHh4eioqLk4OCgBg0aKCsrS3/+8581btw4eXp6FnrfFy5cl9lc8F2V/EO3jdTUayW+TY6d7XD8jKswx87e3o4TIAOy2d3s1atXlySlpqZatZ8/fz7X0Lt0ezj+kUcekYODg6Wtdu3akqTk5ORSrBQAAGOxWZjXrVtXlSpV0p49eyxtSUlJSk5OVtOmTXP1f/zxx3X48GGrGyeOHTsmBwcH1ahRwyY1AwBgBDYLc2dnZ/Xs2VNRUVHauXOnDh48qIiICIWEhCgoKEgmk0mpqaky/XeauldeeUWZmZkaM2aMTpw4ob///e/66KOP1Llz5yINsQMAcL+z6UNjhg8fro4dO2rUqFHq06ePfH19NXv2bEnSvn37FBoaqn379kmSqlWrpmXLlunKlSt68cUXNXLkSLVr104TJ060ZckAAJR7Np0C1dHRUWPHjtXYsWNzLWvWrJmOHj1q1Va7dm0tXrzYVuUBAGBITLQCAIDBEeYAABgcYQ4AgMER5gAAGBxhDgCAwRHmAAAYHGEOAIDBEeYAABgcYQ4AgMER5gAAGBxhDgCAwRHmAAAYHGEOAIDBEeYAABgcYQ4AgMER5gAAGBxhDgCAwRHmAAAYHGEOAIDBEeYAABgcYQ4AgMER5gAAGBxhDgCAwRHmAAAYHGEOAIDBEeYAABgcYQ4AgMER5gAAGBxhDgCAwRHmAAAYHGEOAIDBEeYAABgcYQ4AgMER5gAAGBxhDgCAwRHmAAAYHGEOAIDBEeYAABgcYQ4AgMHdNczT0tI0ceJEnTt3zqr93XffVWRkpC5evFiqxQEAgILlG+bnz5/XK6+8oq1bt+rChQtWy/z9/bV9+3b16NGDQAcAoIzlG+bz5s1TtWrV9NVXX6levXpWywYMGKANGzbI1dVV8+fPL/UiAQBA/vIN8507d2rEiBFyc3PLc7mnp6dGjBihb7/9trRqAwAAhZBvmF+4cEG1atW668q1a9fW+fPnS7woAABQePmGube3t06fPn3Xlc+cOaOqVasWemfZ2dmaMWOGQkNDFRwcrPDwcKWlpRVq3UGDBql3796F3hcAAA+KfMP8mWeeUWxsrLKzs/Ncnp2drU8++UTNmzcv9M5iYmK0bt06TZs2TfHx8UpJSdGwYcMKXG/lypUM5wMAkI98w3zgwIE6ceKE+vbtqx07dujy5csym826ePGitm/frldffVVHjhzR4MGDC7Ujk8mkuLg4RUREqEWLFgoMDFR0dLT27t2rvXv35rve6dOnNXPmTAUHBxf93QEA8ABwzG+Bl5eXli5dqlGjRmnQoEGys7OzLMvJyVHDhg312WefFXhd/TdHjhxRenq6QkJCLG01a9ZUjRo1lJiYqMaNG+daJzs7W2PGjNHrr7+uU6dO6ZdffinKewMA4IGQb5hLt29wW7dunfbv369Dhw7p6tWr8vT0VFBQkOrUqVOkHaWkpEiSfHx8rNq9vb0ty+70ySefSJJee+01vfPOO0XaHwAAD4q7hvlvGjZsqIYNGxZrRxkZGbK3t5eTk5NVu7OzszIzM3P1/9e//qVPP/1Ua9eulb198Z46W7Vq3l+vQ9nw8nIv6xJQDBw/4+LY3b/yDfPY2Ni8V3B0lIeHhxo0aKC6desWekeurq4ym83KysqSo+P/dmsymVShQgWrvpmZmRo9erSGDx8uf3//Qu8jPxcuXJfZnFNgP/6h20Zq6rUS3ybHznY4fsZVmGNnb2/HCZAB5Rvmq1evzrM9JydHV65cUUZGhlq1aqXZs2fnOtvOS/Xq1SVJqamplt+l24+NvXPo/eeff9aJEyc0ffp0TZ8+XdLt0DebzQoODtamTZvk6+tb8LsDAOABkG+Y/9///d9dVzxy5IgiIiI0f/58hYeHF7ijunXrqlKlStqzZ486d+4sSUpKSlJycrKaNm1q1bdhw4basmWLVVt0dLR+/fVXTZ8+Xd7e3gXuDwCAB8U9X4yuW7euIiIitGnTpkL1d3Z2Vs+ePRUVFaWdO3fq4MGDioiIUEhIiIKCgmQymZSamiqTySRXV1f5+/tb/bi5uVnafz9MDwDAg65Yd5YFBATkeyd6XoYPH66OHTtq1KhR6tOnj3x9fTV79mxJ0r59+xQaGqp9+/YVpyQAAB44xTrFTU9PV8WKFQu/M0dHjR07VmPHjs21rFmzZjp69Gi+606ZMuWeagQA4H5XrDPzFStWqFGjRiVVCwAAuAdF/mqa2WzWtWvXtG/fPh0+fFjLli0rteIAAEDBivzVNCcnJz300EMKDAzUlClT9Mgjj5RacQAAoGD3/NW0a9eu6fPPP9fw4cO1cePGEi8MAAAUTpFvgNu7d69Wr16tr7/+WhkZGUV6ChwAACh5hQrza9euaf369Vq9erWOHz8uSWrRooVef/11PfHEE6VaIAAAuLu7hvmPP/5oOQu/efOm6tWrp4iICM2aNUtjx45V7dq1bVUnAADIR75h/sILL+jEiRN67LHHNHjwYIWFhVkmPZk1a5at6gMAAAXI93vmJ0+elJ+fn1q1aqXHH3+8RGYvAwAAJS/fM/MdO3Zow4YNWrdunebNm6eqVauqffv2eu6552RnZ2fLGgEAwF3ke2ZerVo1DRgwQBs3btTq1avVrl07bdy4UX369FF2drZWrlyps2fP2rJWAACQh0I9zrVBgwaKjIzUrl27NHPmTD399NNauXKl2rZtq7feequ0awQAAHdRpO+ZOzk5qX379mrfvr3S0tL0+eefa/369aVUGgAAKIx7nmilWrVqeu2113j6GwAAZaxYs6YBAICyR5gDAGBwhDkAAAZHmAMAYHCEOQAABkeYAwBgcIQ5AAAGR5gDAGBwhDkAAAZHmAMAYHCEOQAABkeYAwBgcIQ5AAAGR5gDAGBwhDkAAAZHmAMAYHCEOQAABkeYAwBgcIQ5AAAGR5gDAGBwhDkAAAZHmAMAYHCEOQAABkeYAwBgcIQ5AAAGR5gDAGBwhDkAAAZHmAMAYHCEOQAABkeYAwBgcDYN8+zsbM2YMUOhoaEKDg5WeHi40tLS8u2/efNmde7cWUFBQXr22We1YMECZWdn27BiAADKP5uGeUxMjNatW6dp06YpPj5eKSkpGjZsWJ59d+zYobffflsvvfSSNmzYoJEjR2rhwoWKjY21ZckAAJR7Ngtzk8mkuLg4RUREqEWLFgoMDFR0dLT27t2rvXv35uq/cuVKtWvXTq+++qr8/PzUvn179evXTwkJCbYqGQAAQ3C01Y6OHDmi9PR0hYSEWNpq1qypGjVqKDExUY0bN7bqP2TIEFWsWNGqzd7eXlevXrVJvQAAGIXNwjwlJUWS5OPjY9Xu7e1tWfZ7DRs2tHp9/fp1rVixQi1btiy9IgEAMCCbhXlGRobs7e3l5ORk1e7s7KzMzMwC1x06dKgyMzM1cuTIIu+7alW3Iq+D0uPl5V7WJaAYOH7GxbG7f9kszF1dXWU2m5WVlSVHx//t1mQyqUKFCvmud/HiRQ0dOlTHjx/XkiVLVKNGjSLv+8KF6zKbcwrsxz9020hNvVbi2+TY2Q7Hz7gKc+zs7e04ATIgm90AV716dUlSamqqVfv58+dzDb3/JikpST169FBSUpLi4+NzDb0DAAAbhnndunVVqVIl7dmzx9KWlJSk5ORkNW3aNFf/CxcuqE+fPjKbzVqxYoXq1q1rq1IBADAUmw2zOzs7q2fPnoqKipKnp6eqVq2qiRMnKiQkREFBQTKZTLpy5Yo8PDzk7OysiRMn6tKlS/rss8/k6upqOaO3s7NTtWrVbFU2AADlns3CXJKGDx+urKwsjRo1SllZWWrZsqUiIyMlSfv27VOfPn0UFxenRo0aaevWrTKbzXrppZestuHg4KBDhw7ZsmwAAMo1m4a5o6Ojxo4dq7Fjx+Za1qxZMx09etTy+vDhw7YsDQAAw2KiFQAADI4wBwDA4AhzAAAMjjAHAMDgCHMAAAyOMAcAwOAIcwAADI4wBwDA4AhzAAAMjjAHAMDgCHMAAAyOMAcAwOAIcwAADI4wBwDA4AhzAAAMjjAHAMDgCHMAAAyOMAcAwOAIcwAADI4wBwDA4AhzAAAMjjAHAMDgCHMAAAyOMAcAwOAIcwAADI4wBwDA4AhzAAAMjjAHAMDgCHMAAAyOMAcAwOAIcwAADI4wBwDA4AhzAAAMjjAHAMDgCHMAAAyOMAcAwOAIcwAADI4wBwDA4AhzAAAMjjAHAMDgCHMAAAyOMAcAwOAIcwAADI4wBwDA4Gwa5tnZ2ZoxY4ZCQ0MVHBys8PBwpaWl5dv/wIEDeuWVV9SoUSO1a9dO69evt12xAAAYhE3DPCYmRuvWrdO0adMUHx+vlJQUDRs2LM++Fy9e1Ouvv67AwEAlJCSod+/eGj9+vHbt2mXLkgEAKPccbbUjk8mkuLg4TZgwQS1atJAkRUdHq02bNtq7d68aN25s1X/NmjVyc3PT+PHjZW9vr0ceeUSHDh3SkiVLFBoaaquyAQAo92x2Zn7kyBGlp6crJCTE0lazZk3VqFFDiYmJufonJiaqadOmsrf/X4khISHau3evcnJybFIzAABGYLMz85SUFEmSj4+PVbu3t7dl2Z3969Wrl6tvRkaGLl26pCpVqhR63/b2doXuW82zUqH7lgfOD1Ut6xKKrCjHoyiMduwkjt/vcfxKX2GOXWkdX5Qum4V5RkaG7O3t5eTkZNXu7OyszMzMXP1v3rwpZ2fnXH2l20P2ReFZhD8SH4/rUqRtl7UGg6eVdQlFVrWqW6ls12jHTuL4/R7Hr/SV1rFD2bPZMLurq6vMZrOysrKs2k0mkypUqJBn/ztD+7fXefUHAOBBZbMwr169uiQpNTXVqv38+fO5ht4l6eGHH86zb8WKFeXu7l56hQIAYDA2C/O6deuqUqVK2rNnj6UtKSlJycnJatq0aa7+TZo0UWJiotXNbrt371bjxo2tbooDAOBBZ7NUdHZ2Vs+ePRUVFaWdO3fq4MGDioiIUEhIiIKCgmQymZSammoZSu/evbsuXryod999VydOnNBf//pXffHFF3r99ddtVTIAAIZgl2PD73llZWVp+vTpWrdunbKystSyZUtFRkaqSpUq2r17t/r06aO4uDg1a9ZMkvTTTz/p/fff19GjR+Xr66vw8HB16NDBVuUCAGAINg1zAABQ8rj4DACAwRHmAAAYHGEOAIDB2ewJcMhf69at1b17dw0dOtTSlp2drZEjR2r79u2aP3++JkyYIHt7e23cuDHXQ3N69+4tPz8/TZkyRZIUEBCg4OBgLV++PNfX+PLaF4pv/fr1io+P1/Hjx2VnZ6eAgAD16dNHzz//vKWP2WzWqlWrtH79ev3nP/9RZmam/P391aFDB/Xv318uLi6SZLkZ9Dd2dnaqUKGC6tSpo759+3ITqA20bt1aycnJltdOTk7y8fFRu3bt9Oabb8rNzS1Xnzt17dpVU6dOtUW5AGFeHpnNZo0ZM0bffvutYmNj1bx5c0nSmTNnFB0drfHjxxe4jX379ikuLk79+vUr5WqxatUqTZs2TRMmTFCTJk1069Ytbd26VREREcrMzFTXrl2VlZWlQYMG6dChQ3rzzTfVvHlzubi4aN++fZo1a5Z++OEHffrpp7Kz+99zsdetWycvLy+ZzWZdunRJmzZt0siRI3X58mX16tWrDN/xg+GNN95Q3759Jd1+HPW//vUvTZ061fLf1tq1a5WdnS1J2rx5s6ZNm6YdO3ZY1nd1dS2TuvFgIszLmZycHI0fP17ffPONFixYYDXLXK1atRQfH6+wsLBcU8beqVatWpo1a5batGmjWrVqlXbZD7RVq1bp5Zdf1osvvmhpq127tk6dOqW4uDh17dpVS5Ys0e7du5WQkKBHH33U0q9mzZpq1KiRwsLCtGPHDj3zzDOWZVWqVJGXl5ek2xMU1a1bVxkZGZo+fbrCwsKKNNkQiq5ixYqWz1+S/Pz85O/vr27duulvf/ubevToYVn221Mpf98fsCWumZcjOTk5ioyM1FdffZUryKXbw3bBwcEaP358npPT/N7AgQPl7e2t8ePHM2VsKbO3t9fevXt17do1q/YxY8YoJiZGOTk5WrZsmbp06WIV5L/x8/PT5s2b9fTTTxe4r759++rGjRv69ttvS6p8FEFgYKCaNGmizZs3l3UpgBXCvByZNGmSVq9erT//+c95PuLWzs5OU6ZMUXJysmJiYu66LRcXF02ZMkV79uzRypUrS6tkSHrttde0f/9+tWzZUoMHD9bixYt1+PBhValSRTVr1lRSUpJSUlL0xBNP5LsNf39/qyH2/NSqVUsVKlTQsWPHSvItoAgeffRRPn+UOwyzlxPLly/XjRs31LBhQy1atEidOnXKcxj1j3/8o8LDwxUdHa327durfv36+W6zadOm6tGjhz766CM988wzlsluULLCwsLk4+Ojzz77TN9//722b98uSapXr56ioqJ0/fp1SZKnp6fVep06ddKZM2csrzt27KhJkyYVuL+HHnrIsk3YHp8/yiPOzMuJGzduaPHixZozZ45MJpP+8pe/5Nu3f//+CgwM1Lhx43Tr1q27bvftt99W5cqV9c4775R0yfidxo0ba/bs2dq9e7fWrFmjIUOG6MyZM3rjjTdUuXJlSdKVK1es1omNjdX69eu1fv16BQQE5JryNz/Xr19n5sAylJ6ezuePcocwLyf69++v4OBg+fj4KDIyUtu3b1d8fHyefR0cHPTBBx/o5MmTio2Nvet2K1WqpMmTJ+u7775TQkJCaZT+QDt79qzee+89y3S9Dg4OatiwoYYPH65Zs2bp7Nmzunr1qqpVq6bExESrdX19feXv7y9/f/9C3/l8+vRppaenq169eiX+XlA4Bw8e5PNHuUOYlxMODg6W31944QWFhYUpKioq32tzderU0ZAhQ/TJJ5/ol19+ueu2W7RooW7dumnq1KkMD5YwFxcXrV27Vl988UWuZQ899JDs7Ozk5eWlXr16KSEhQSdOnMjVz2Qy6eLFi4Xa3/Lly+Xm5qZWrVoVu3YU3ZEjR7Rv3z517NixrEsBrHDNvJx67733lJiYqJEjR2rt2rV59hk4cKC2bNmiI0eOFLi9cePG6bvvvtP58+dLutQHWpUqVfTaa69pxowZun79utq1aydXV1cdO3ZMs2bNUteuXeXr66uBAwfqwIED6tGjh4YMGaLQ0FC5urrqp59+0oIFC3Ty5En17t3batsXL16Ug4OD5XvmX375peLi4jRp0iS5ubmV0Tt+cNy4ccMy4nLz5k3t379f06dPV9OmTdWpU6cyrg6wRpiXU5UrV9aUKVM0cOBATZs2Lc8+Tk5O+vDDD/XSSy8VuD13d3dNmjRJgwcPLulSH3gjRoyQv7+/Vq9eraVLlyozM1N+fn7q2rWr5aE9jo6Omjdvnj7//HMlJCQoNjZWN27ckK+vr0JDQxUTE6M//OEPVtvt2rWrpNvfYqhataoCAgIUGxtbqK+wofgWLlyohQsXSrp9uapGjRp6+eWX1a9fP6uRNKA8YApUAAAMjmvmAAAYHGEOAIDBEeYAABgcYQ4AgMER5gAAGBxhDgCAwRHmwH+ZTCYtWrRIXbp0UXBwsJ588kkNHjxYBw4ckCQlJSUpICAg12NZS0NMTIyeffZZy+vvvvtOrVu3VoMGDRQXF6fWrVtr3rx5pV4HAGPge+aApIyMDPXp00eXLl1SeHi4GjVqpPT0dMXFxWnz5s1asGCBatasqTZt2mjZsmV6/PHHS7We9PR0ZWZmWmbO6969uzw8PDRx4kRVrlxZJpNJrq6uqlixYqnWAcAYeAIcIGnWrFk6deqUvvjiC/n4+Fjap06dqgsXLmjy5MkFTmpTkipVqqRKlSpZXl+9elVPPfWUatasabMaABgHw+x44JlMJiUkJKh79+5WQf6byMhIzZgxQ3Z2dlbtly9f1rhx4xQaGqrAwECFhoZq2rRpMpvNkqS0tDS99dZbCgkJUVBQkPr166fDhw9b1k9ISFBYWJjq16+vVq1a6eOPP7as+/th9oCAAJ0+fVpz585VQECAJOUaZt+2bZs6deqkBg0aqH379lq8eLFlW79dHoiNjVXz5s0VFhZW6OlWARgDZ+Z44J05c0ZXr15Vo0aN8lxeq1YtSbdD8ffGjBmjS5cuaf78+apcubJ27typyZMnq0mTJmrbtq0mTpyorKwsrVixQnZ2dpoxY4aGDRumbdu26ciRI4qMjFR0dLTq16+vgwcP6u2335afn5+6dOlitZ9du3bpT3/6k5577jkNGDAgV307duzQ22+/rQkTJigkJET//ve/NWnSJGVkZOitt96y9Nu0aZPi4+N18+ZNOTs7F/NTA1CeEOZ44F29elXS7SlLi6Jly5Zq1qyZ6tSpI0nq1auXFi1apKNHj6pt27Y6ffq0AgICVLNmTbm4uGjSpEk6fvy4zGazzpw5Izs7O/n6+lp+Pv30Uz388MO59uPl5SUHBwdVrFhRXl5euZbHxsaqR48e6t69uyTJz89P6enpeueddzR06FBLv169eumRRx4p0nsEYAyEOR54np6ekm4PmxdFjx499M0332jNmjU6deqUjh49qpSUFMvw9tChQzVmzBht2bJFTZs21VNPPaWOHTvK3t5eLVu2VKNGjdStWzf5+/srNDRU7du3l6+vb5HrP3z4sA4cOKCVK1da2sxms27evKnk5GTL5YHfRhgA3H+4Zo4Hnp+fn6pWraqff/45z+W7d+/W4MGDLXNbS7fDcuDAgZo6daoqVKigzp07Kz4+XjVq1LD0ad++vb777ju9//778vLy0rx58/TCCy8oLS1Nrq6uio+P19q1a9W5c2cdOnRIvXv31pw5c4pcv5OTkwYPHqz169dbfjZs2KAtW7ZY3QPg4uJS5G0DMAbCHA88e3t7de3aVX/729907tw5q2U5OTlasGCBTp48qWrVqlnaDx06pF27dikmJkYjRoxQhw4d5OnpqdTUVOXk5CgrK0vTpk1TcnKyOnbsqA8//FCbNm1SWlqa9uzZo++//15z585VgwYN9Oabb2rlypXq0aOHNm/eXOT6a9eurVOnTsnf39/yc+zYMc2cObPYnw0AY2CYHdDtIfHvv/9ePXv21IgRI9SoUSOlpaVpyZIl+uc//6klS5ZY3c3u5eUlR0dHffnll/Lw8FBqaqpmzpwpk8kkk8kkR0dHHTx4UImJiZowYYKqVKmijRs3ysnJSYGBgTp37pzmzp0rd3d3tWrVSmlpadq9e7eCgoKKXPuQIUM0aNAgPfroo2rXrp1OnTqlyMhIPf3009zoBjwgCHNAt7/XHR8fr4ULF2rOnDk6e/as3N3d1ahRI61atUqPPfaY1d3sPj4++uCDDxQTE6PPPvtMPj4+CgsLk4+Pj+WJcTNmzNAHH3ygQYMGKT09XXXq1NHcuXMtZ88ffPCBFi1apOnTp8vNzU1t27bV6NGji1z7U089paioKC1YsEAff/yxqlSpoi5dumjEiBEl9vkAKN94AhwAAAbHNXMAAAyOMAcAwOAIcwAADI4wBwDA4AhzAAAMjjAHAMDgCHMAAAyOMAcAwOAIcwAADO7/A49ajGGS3Pk3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "sns.set(style=\"darkgrid\")\n",
    "\n",
    "model_results = pd.DataFrame({'classifier':['KNN','KNN','SGD','SGD','DT','DT'],\n",
    "                           'data_set':['train','valid'] * 3,\n",
    "                           'auc':[knn_train_auc, knn_val_auc, sgd_train_auc, sgd_val_auc, \n",
    "                                  tree_train_auc, tree_val_auc]})\n",
    "\n",
    "ax = sns.barplot(x= \"classifier\", y= \"auc\", hue= \"data_set\", data = model_results)\n",
    "ax.set_xlabel('Classifier',fontsize = 15)\n",
    "ax.set_ylabel('AUC', fontsize = 15)\n",
    "ax.tick_params(labelsize=15)\n",
    "\n",
    "plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0., fontsize = 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "white-wages",
   "metadata": {},
   "source": [
    "### Takeaways\n",
    "- The Decision Tree Classifier performed the best on the training data, but the performance did not carry over to the validation data.\n",
    "- The K-nearest neighbor classifier was the worst performing model of the three models\n",
    "- The SGD Logistic Regression classifier performed better than the KNN model and was more consistent that the Decision Tree Classifier, so this classifier will be used for classifying our test data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sacred-membership",
   "metadata": {},
   "source": [
    "## Test Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "downtown-environment",
   "metadata": {},
   "source": [
    "After comparing three different classifiers, our result show that the SGD Logistic Regression classifier performed better and more robustly than the other two models. We will use this model in our test evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "professional-sandwich",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = test_data[feature_cols].values\n",
    "y_test = test_data['output_label'].values\n",
    "X_test_tf = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "digital-groove",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SGD Logistic Regression\n",
      "Training:\n",
      "AUC:  0.6663677304106352\n",
      "accuracy:  0.6197298661954052\n",
      "recall:  0.5568038374147942\n",
      "precision:  0.6369675090252708\n",
      "\n",
      "Validation:\n",
      "AUC:  0.6792550475231378\n",
      "accuracy:  0.6630427488088048\n",
      "recall:  0.5741176470588235\n",
      "precision:  0.18509387445476957\n",
      "\n",
      "Test:\n",
      "AUC:  0.6538727019254345\n",
      "accuracy:  0.6597101060260367\n",
      "recall:  0.5455082742316785\n",
      "precision:  0.17665071770334928\n"
     ]
    }
   ],
   "source": [
    "sgdc_final = SGDClassifier(loss = 'log', alpha = 0.1, random_state = 5)\n",
    "sgdc_final.fit(X_train_tf, y_train)\n",
    "\n",
    "y_train_pred = sgdc_final.predict_proba(X_train_tf)[:,1]\n",
    "y_val_pred = sgdc_final.predict_proba(X_val_tf)[:,1]\n",
    "y_test_pred = sgdc_final.predict_proba(X_test_tf)[:,1]\n",
    "\n",
    "print('SGD Logistic Regression')\n",
    "print('Training:')\n",
    "sgd_train_auc, sgd_train_acc, sgd_train_recall, sgd_train_prec = print_report(y_train, y_train_pred)\n",
    "print()\n",
    "print('Validation:')\n",
    "sgd_val_auc, sgd_val_acc, sgd_val_recall, sgd_val_prec = print_report(y_val, y_val_pred)\n",
    "print()\n",
    "print('Test:')\n",
    "sgd_test_auc, sgd_test_acc, sgd_test_recall, sgd_test_prec = print_report(y_test, y_test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "african-kidney",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbsAAAGsCAYAAABEugk9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAACEi0lEQVR4nOzddXiV5R/H8fc62MbY2AaMMRqku1uQ7hBE6Y4JSDfS3SAlJSWIlEgooXR3NwPW3See3x+H38E55gZs5yy+r+vy8jx5vudmO589dd8miqIoCCGEEBmYqbELEEIIIVKbhJ0QQogMT8JOCCFEhidhJ4QQIsOTsBNCCJHhSdgJIYTI8CTshBBCZHjmxi7gUwQHR6LVftpjgs7OdgQGRqRQRRmLtE3ipG0SJ22TOGmbxKVE25iampAtW5b3LkvXYafVKp8cdv/fj3g/aZvESdskTtomcdI2iUvNtpHTmEIIITI8CTshhBAZXro+jfk+Go2a4GB/1Oq4ZK3v52eKVqtN5arSp7TQNubmlmTL5oKZWYb7URVCGFCG+wYJDvbH2tqWLFlyYGJikuT65uamqNUSdu9j7LZRFIXIyDCCg/3Jnj2n0eoQQqR/Ge40plodR5YsDskKOpG2mZiYkCWLQ7KP0oUQIjEZLuwACboMRP4thRApIUOGnRBCCPFPGe6aXVoyf/5sbt68jlqtwtv7JXnz5gegffuONG3aIln76NbtKzZs2Jrs91Sr1bRt25Q6dT5n6NCR+vnr1q0CoGfPvvp5Bw/u5+rVy4wbNxmAM2dOsXnzj0RFRaPVaqhTpx7du/fB1DTxv4lUKhWzZn3PvXt3sbKyYtKk6Xh65k2w3r59v7Jnzy6ioqJp3rwlnTt3Zf/+Pfzyy8/6dd68eUXDhk0YNmxUsj+vEEIkh4RdKvruO92X9ps3rxk8uO8Hhdb/feg2586d4bPPinPs2B/07++FtbV1srdbuHAO8+cvJU8eT2JjY5g0aSzr1q2id+/+iW63c+d2rK1t2LJlF9euXWHatEmsWbMx3jrXr19j+/afWL16I6ampvTo0Zlq1WrSvHkrmjdvBcCTJ48ZO3Y4PXr0fc+7CCHEp5GwM5J27ZpTrFgJHj68z4oVa/n5521cvnyRsLAwHB0dmT59Ds7O2alRowKnTl1i3bpVBAT48/LlC3x9fWjWrCVdu/ZMsN+DB/dTq1ZdtFqFP/44TLNmLZNVz6ZNP9K9e2/y5PEEwMrKmpEjx/D48RMA9uzZRUBAAL169Yu33dmzp/TzypQpR2hoCD4+PuTIkUO/zrFjR2jduj12dnYALFy4nKxZs8bbz/z5s+jTZyCOjo7Ja0AhhPgAqR52ERERdOzYkR9++IHcuXPHW3b37l3Gjx9PREQEFSpUYMqUKZibp1xJYWdOE3rqr/9cx8TEBEX58C5qstaohUO16h9bGgBVqlTj++9n4u39khcvnvHDDz9iamrK1KkTOXLkEJ06fR1v/UePHrJixVoiIsLp0KEVbdp0wN7eXr88ODiYixfPM3r0BMzMzNi1a3uyw+7hw/sUK1Yi3jxXVzecnFwAaNWq3Xu3Cwjwx9k5u37a2Tk7/v6+8cLO29sbOzt7Bg3qQ0REBM2ataBdu4765Rcvnic2NpZ69eonq1YhhPhQqXqDyvXr1+nUqRPPnj177/IRI0YwYcIEDh8+jKIo/Pzzz+9dL6P6f7jkzu3BoEFD2b9/D0uXLuT27ZtER0clWL9cuQpYWFiQLZsTDg4OREbG7zT16NHfKV++Ag4ODtSsWZvHjx/z4ME9gPded1MURX+3o4mJ6UeF/vs2MTGJ/14ajZqbN68zZ84iliz5gf3793DlyiX98r17d/Pll1998HsLIURypeqR3c8//8ykSZMYOXJkgmWvXr0iJiaGMmXKANCmTRuWLFnCV1+l3JeeQ7XqSR59GfPBaSsrKwDu3bvL5Mnj6NjxK+rW/Rwzs/cHj6Wlpf71+45If/ttP4GB/rRr1xzQ9QC+Z88vjBw5Dnt7e169ehVv/eDgIOztHQAoWvQz7t27Q758+fXLX7x4zrp1a5gw4ftEP4OLiwuBgYHkzu0BQGBgANmzu8Rbx9nZmcKFi2JrawtA5cpVuXv3NuXKVUClUnHt2hX9TTJCCJEaUvXIbvr06VSoUOG9y/z8/HBxefel6OLigq+vb2qWk2Zdu3aZsmXL06pVO/Lmzc+FC+c/uJuu+/fv4efnyy+/HGDXrv3s2rWfOXMWcfToYaKiIilXrgJnzvxNcHAwoDu9/OefR6hQoRIAX33VhfXr1/Dy5QsAoqKiWLx4Pm5uORJ9T4CqVatz6NBvgO5GFEtLq3inMAGqVavFX3+dQKVSERsbw6VLFylS5DMAHj9+iIdHHmxsbD7o8woh0i+/kGj+vv6Kn9YfYfyqn+i53Isl29an6nsa7QaV9x25fOgDxM7Odgnm+fmZYm7+YRn+oet/KDMz0/e+j5mZrtYvvmjE6NHD6dq1I+bm5hQqVAgfn9f69c3NTTE1NUmwj/9vD/D77/tp1qwFWbLY6pdXqlSJPHny8Mcfh2nTph3duvVg6NABAGg0Glq2bE3NmjUBqFGjBtHRkUyaNBatVoNarebzz+vTs2dfTE1N2b17FwEB/vTpE//OzC+/7MTs2dP55psOWFhYMnnyVMzNTbl79w6rV69k4cKlNGzYkDdvvOnRozNqtZrGjZtRpUoVAHx8XuPm5vaf/wampqa4uNgnutxY0mJNaYW0TeIyetuo1FrUGt0f69Gxarz9wolTadl+5D4vfcPQqNTEKgrmOZ9invMFr/66xfODd7HspODVqXuq1WWifMyFmg9Ur149Nm3aFO8GlVevXtGtWzeOHj0KwKVLl1iyZAmbNm1K9n4DAyMSjH/k4/OcHDk8k70PY/f/mJallbb50H9TQ3BxscffP9zYZaRJ0jaJy4hto1UU7jwN4v7LEB68DOGhd2giaypgouCU4wrRHgEAhDzw5/YPZ6lUtxo7N2wjODj6k2oxNTV570EQGPHIzt3dHSsrKy5fvkz58uXZs2cPtWrVMlY5QgghPoBWUXjuE87UjZfizTczNaGsiyn2t88D8KxYOKEusUS/HUD8/3Hmae/B9G7jOJf3FA0aNErRO/Hfx+Bh17t3b7y8vChZsiTz5s1j/PjxREZGUqxYMbp06WLocoQQQnyAwNAYfth3i8evwuLN/666I3Z3L6G6dQ1/31iOVHXAz8lCv9zC1JxS2YvjZJ0N7yMPaNKoCY7WWWnUqKlB6jZI2B07dkz/es2aNfrXRYsWZdeuXYYoQQghxCcIjYhlzOpzxMRp9PM8stvSyPQ5Dud/51msFX+VsyOioB3w7lRi9VyVaFWgKbYWNiiKwpw509m27SdMMWXAAC+D1S89qAghhEhUUFgMO4494uI9P/28sib+1H94GLNHWvwdzVjxpWu8bfLY56ZO7uqUdS2FpZnu6E6r1TJjxhR27drBN990o3//wQb9HBJ2Qggh4tFotRw6/4JTN97g+4+bRopH3sPZ5TYAP7bORrS1Gcrbm+jzOXjyVdG25LJL+LiSRqPh++8nsHfvbnr06MPgwUMNPnyXhJ0QQghA90hYYGgMM3+6THDEu0GTawVewcP8AbsbOPKELPG2qehWjkKO+ajuXjnR/apUKl698qZfv0H07TvQKONUStgJIUQmFh4Vx5ytV1EUhdeB8bsp7BiyC4vieTldPpqLZo4AuNg4M67yd1iYJh0fKpWKuLhYsmSxY+XKtVhYWCa5TWqRwVtT0YABvTh69FC8edHR0TRp8jkhISHv3Wb69MkcPLifgAB/hg9//8XbGjXe3ytNaGgI9epVY9u2n967z39at26Vfow7gN9/P0CvXl3o1u0runbtyM6d25P6eISHhzNixLd07tyOgQN7ExgYkGAdRVHYsGEt3bt/RadObfS9rQBs3bqJr79uz9dft2f79p8SbCuESF3zd1zj2yWneBUQyevAKBzM/HDKcwb7Moewr3iEvV/Yscs9gDdmkQDUyV2dyVVHJTPo4hg1ahgDBvRGrVYbNehAjuxSVZMmzTl69DANGjTSzzt58hjlypVPciib7NldmDdvyQe939Gjh6hWrSb79u2mY8fOyT5VsHfvbvbu/YU5cxaTPXt2wsPDGTZsIFmy2NKkSeKDzK5Zs4JSpcoyd+5iDh36jcWL5/P99zPjrXPkyO9cvHie1as3EhYWSrduX1G9ei1CQ0P49ddd/PTTz2i1Cl9/3Z4aNWrr+9gUQqQcRVEIj1Lxwi+c5z7h3HwcwAPvd48OlIq8w5Oaz1GZm6B6Oy+rpR0FHPNSLVclHCztcbfLmez3i42NZfhwL/7++ySjRo1L9WfoksP4FaSi0zffcOrGm/9cx8Tk/T33J6VGqZxUL/nf//j16jVg+fLFhIWF4uCgG7/t8OGDdOjwFVevXmb16hXExsYQHh5O//5e8Ya4+f+Ar7t27efNm9d8//0EoqOjKV68RGJvx8GD+xk0aCiLFs3lypVLlC9fMVmfZePGdYwfP4Xs2XVD9djb2zNu3BRiYnSnNNau/YHs2bMnGObn7NnTLFu2GoD69RuyYMEc1Gp1vB/sP/88SqdOX2NhYYGzc3ZWrFiLlZUVWq32bV+ZcYCCoihp4hdCiIzmlX8EE9ZdeM8SBQeHJ7jl8eahbTSg++P4qyJtqZyzPObJOHp7n+joaIYNG8TZs6cZP34K7dp9+fHFpyD5dklFtra21KxZm2PH/qBVq7YEBPjz4sVzKleuysSJoxk9egKennm5fPkiixfPS3Q8t4UL59CkSXOaN2/FoUO/sXfv7gTrPHz4gMDAAEqXLku9eg3Ys+eXZIVdSEgIfn6+CUI0b958+u7C/j1g6//9cyw7c3NzsmTJQkhIcLxRD169esmzZ0/ZvHkDKlUcnTt3w8MjD3nyeFK/fkPatWuOoig0b96SHDmS/5ejEOK/Xbjry+ELL3j65l33ZHUCLuOhDsI8rzXbS0WiArzfLivjUpLeJb/55PedNm0S586dYfLk6bRq1faT95dSMnTYVS+Z9NFXavf/2LRpC9asWUmrVm05cuR3GjZsgqmpKRMmTOXMmb85fvyPt+PXJd4n3NWrl5k8eToAX3zRmFmzpiZY5+DB/dSt2wAzMzM+//wLNmxYS1BQIE5OzomezjQ1fdfB9Mcc3SanM2+NRsPjx49YvnwNgYEB9O/fk8KFi/DqlTf3799lz57fURQtw4d/y59/HuXzzxt8eCFCCD2totBr9nH9tJlWQ+2gK9R0iOJe+2LsivQHdNfgcmXJwcAyPXG0yppi79+370Bq1apLw4aNU2yfKUFuUEllpUuXJTAwAF9fHw4f/p2mTXXXwAYO7M3du7cpUqQoXbr0SGLgVBN9h9cmJiYJBmJVq9UcPfo7x48fpV275gwdqru197ff9gFgb+9ARET8zmd1Y9nZ4+CQlVy53Ll370685VevXmb58v++Zuji4kpQUKC+hsjISLJmdYy3jpOTM3XqfI65uTlubjkoXrwkDx7c5/Tpv6lTpx62trZkyWJH/foNuXbtyn++nxDiv0VEq+IFXcdXRxj6cjs5epRifs049kVeBiBHFje6FuvIuMrDUiTowsLC2Lx5PYqikCePZ5oLOpCwM4jGjZuxceM6HBwccHfPTVhYKC9fPqdnz35UrVqDCxfO/ef4dRUqVOLw4YOA7gaXuLi4eMtPn/4LR8ds7N17WD+W3YgRY9m3bw+KolChQkX++OOI/ugxMDCAM2dOUa6c7jTnV199w7Jli/R3U4aEhLBs2aIkbxapUuXdWHbHjh2ldOkyCa67Va9ek2PHjqIoCqGhIdy5c4tChQpTsGAhTp8+hUajG07o/PkzfPZZsQ9oVSEEQEycmk0H7zBo1hG8Fv+tn989YCsnW1qwvL0zv/ocR6PouvkaXn4gEyp/R6Uc5VLk/UNDQ+jXrzuLFy/gwYP7KbLP1JChT2OmFY0aNaV9+xaMGTMRAAeHrDRr1opvvulAlixZKF68FDExMYmeyhw2bCRTp05k377dFC1aDFvb+A91/vbb/gTnxhs0aMSqVcs5f/4sVavW4NGjh/Tp0xVTU1NMTEzp338w+fMXAKBVq3aoVOq3R4SmKIqWli3b0LJla9RqbaI3qPTu3Y/p0yfz9dcdsLe3Y+LEaQCcOnWSU6f+YvToCXz5ZWdWrFjMN998iVaroVu3XuTJ40nu3B68ePGMb77pgJmZGVWr1qBx42Yp0t5CZBYnzz9h4/Fnb6fMcVBFUtomguoFFBZXcgJiMTUxxcHSnpEVvMhqlbJj6QUFBdGvXw+ePXvCggVLKVKkaIruPyUZZDy71CLj2aWutNI2Mp5d+iJtk7hPbRutouAXHI2iKJz++QgHQ3WnIAvF+dCjljvZa9UgWh3DqFNTAMhj786oit+mSO3/FhDgT9++PfD2fsHChcupVq3GJ+0vJX5u0uR4dkIIIZKmUmu59yKY41dece3RPztu0AVdbQ9znKtmZ7X/JXxO/BZv29QKOoBHjx7i7+/HsmWrqFixSqq9T0qRsBNCiDTkr+uv+eOSNw5ZLLjzLDjB8mxxYdQMuoZl4SLkaVmRtQ9/RPVMDYC5iRlFnQpTIntRKrqVTZX64uLisLS0pEqVavz22x/Y26fsqdHUImEnhBBpgKIonLz+mk2HdDd5uKisye9iQ1xEJG5v7lMk4gUFc9mh1KrAVsssvI65Cvev6refX2sq1uZWqVqjrjPnHgwaNISGDZukm6ADCTshhDA6lVpL33kn9NMtnUL57MKmeOs4NWmGY6tWfHtiLMTo5pXMXowauSpT1KnQR/d4klwvXjynT59uREVF4eGRJ1XfKzVI2AkhhBEFhcUwfMUZ/fSQJ9uxfqR7vMiqXFl8KubniOY+PlEX4ISu269cWXIwptIQTE0M8/TY06dP6NOnK2q1mjVrNqbpuy4TI2EnhBBGsnXvFf64G6KfHvVoE6aWlrj1HMSpbMH89uwohL8CIIu5LTmyuJE/qyctCjQyWNAFBQXSs+c3mJiYsGbNJgoWLGSQ901pEnapaP782dy8eR21WoW390vy5s0PQPv2HfU9qSQlIiKC6dMnMXPm/PcuHz9+JC9fvmDjxndD8ly5cokff1yt76QZ4ncsDfDixTOWL1/MmzevAcifvyBDhoxIcjSGbdt+Yv/+X9FqFfr3H0Tt2vUSrHPr1g2WLFlAVFQUBQoUZPz4KVhYWLB+/RoOHNiLvb0DAM2bt6Jt2w7JagchMppJi4/yMtoMgGpB12lZ1RPnISt5pQpizMXFEKpbr4JbGWrnrkb+rHmNUme2bE5880036tT5nHz58hulhpQgYZeKvvtuFPAuaDZs2PrB+wgPD+PhwwfvXRYaGsKDB/dxcnLmxo1rlCpVJln7DAjwZ/DgfowYMZYaNWqhKAqbN69n7NjhrFixNtHt7t69zZEjB1m/fitRUZH07dudsmXL60d0AIiMjGDcuBHMn7+MggULMWnSWA4c2Evr1u24e/cOU6bMoESJUh/UBkJkBMHhsQRGqXh16SY/nAkg5u3X76CSZuSu+zWLrq4m5My5eNtMrz4uRfut/BC3b9/E3NyCIkWK0r17b6PUkJIk7IzA2/sl8+bNJCwsFCsra4YOHUHhwkU5cuQQW7duwtTUlFy5cjFhwlQWLZpLQIA/Y8YMZ+bMefH2c+TIIUqXLkv+/AXZu3d3ssPu1193UalSFWrUqAXo+tvs3LkrOXPmQq1W8+jRA9au/YFFi5bF2+7s2dPUrl0PKysrrKysKFu2PKdP/x2v55OLF89TvHgp/amOoUNHotHobou+f/8OW7Zs5PXrV5QuXZaBA4dgZZW6d48JYUxqjZZfTj7mycsgHr6J/McS3VfvpA55ORB2nDvn3j0fVyp7ccq7lqKcW2mDnar8t+vXrzJwYG/y5s3P5s07kj02ZlqWocPu/JvLnH1z8T/X+djx7KrmrEjlnOU/qq7p0ycxdOhIChcuytOnTxg7djjbtu1mzZqVrF69nmzZnFi9egUvXjxjyJARDB7cN0HQARw8uI8+fQaQP39B1q5dybfffhfvKCsxDx/ep2rV+L0dmJmZ6QeZLVq02HsHjg0I8Oezz4rrp52ds+Pv7xdvHW/vl9jY2DBmzHBev/amVKmyDBo0hKioKAoVKsLAgUPIkSMnM2ZMYcOGtfTtOzBZbSZEevPvG0/MFA2Vg29TOL8rVpXKcFo5yZxnh/TLOxdtR5WcFYwWcP935colBg3qg7OzC/PmLc4QQQcZPOzSoqioKO7evcOMGd/r50VHRxMaGkL16jXp378nNWvWoXbtehQqVER/Te3fHj68j5+fLxUrVsHc3JzChYvw++8H+PLLzglGRQDdMzwmb3+JTExMkhhl4f3et4nJv34xNRoNFy6cY9Wq9bi55WDWrKn89NMGevbsGy9AO3bszMyZ30vYiQxFo9Vy5qYPEb5+7LwSpJ8/KuQIOZo1oWDrCRy4cYJNdzfol33hWZeGnvVS/Rm55Lhw4RxeXv3JmTMnq1atx9XVzdglpZgMHXaVc5ZP8ujL0P0/arVaLC2t4l2/8/PzxcEhK0OGDOfRo5acPXuKqVMn0KNHn0RPTf72237i4lR07Nga0IXo3r27+fLLzm+H9ImIt35wcLD+AdCiRYslGNJHq9Uyfvwohg8fjZOT83vf08XFRT8yAuhGTyhTJn7P6c7OzhQvXoJcudwBqFevPr/88jM+Pj5cunSeZs1aArrglJHJRUZx+2kQ83dcSzA/l2k0EzqXwcq9HnGaOMb9MYeHgU8BqOhWjnaFmmNnmSXBdsaybdtPuLvnZvXq9fqBmTMKGeLHwOzs7Mid20M/ZM/Fi+cYOLAPGo2Gjh1b4+joyDffdKdRo6Y8eHAfMzMzNBpNvH2oVCqOHv2dRYtW6If02blzL4GBAVy5com8efMRFhbK7du3AF2QHTiwhwoVKgHQokVrzp49zdmzpwDdUd+GDWsJDg5KNOgAqlSpxsmTx4iJiSE4OJjLly/q9/l/FStW4f79e/j6+gBw+vQpihT5DCsrK1auXMLr169QFIXdu3+mVq26KdOoQhiBSq1l+uZL9Jh1TB90jnFhlA+5y2DNJea0zce0kU2xcnfnit8Nhp4crw+6AaV70q14xzQTdP8fYmzmzHmsXbspwwUdZPAju7Rq0qRpzJ07g61bN2FubsH338/A3Nycnj37MmTIAKysrLGzs2f8+Mlky+aEm1sOBg/uy9KlqwDd+HVubjkpXryEfp9ZstjRrFkr9u79hXLlKvD997NYsmQ+sbGxxMbGUL58RXr06APorrXNm7eEFSsWs3LlUrRaLYULF9U/3nDv3p333qBSrFgJvviiCb16dUGjUdOrVz9cXFwB6NbtK+bNW4ybWw5GjBjLqFHDiIuLpVChwgwaNARra+u384eiUqkpVao0HTt+bYjmFiJFxcSp+e3sc347+1w/L0dMAHUCr1DAMobcQ4Zh5ZGHKFUUNwPusPbmZtRvx5Krn78GzfI0wSKVezv5EMeOHWXDhnUsW7YaBwcHrK2tjV1SqpAhftLIMDZpUVppGxniJ33JiG2jUmv54/JLdh5/HG9+3mgf2r7+E/u8njg1b0mW4iW4H/qEpdfWJNhHv1LdqPdZ5TTVNocP/87YscMpXrwEy5evNWpflzLEjxBCGImiKMz46TKPX4XFm189+AZVA29gjhbHBg1x6dCR15E+jP97Aiqt7lGb7NZOVM1ViUKO+cmf1TPN3dX422/7mDBhNKVLl2XZslVkyfL+kMgoJOyEEOIfYlUarj8K4PCFlzx98y7kCkW8oInfWWy0sVi65yZrhw5krV2XswHXOHhnBxd9r+jX7VSkDTXc0+4Yb4cPH2T8+FFUqFCJJUtWYmNja+ySUp2EnRBCvBUaGcfQpafizXOLCaTDmz/Jnj8PdrVa41C9JmZZsqBVtCy68gOPQ5/p162eqxJfFW1n4Ko/XIkSpWjRojWjR0/AxsbG2OUYRIYMO90zZWnrlIH4OOn4krJIRzRaLXtPPePAmWcAFHSxps6lHTirQjEB8s2eh4VzdgKiA/np6X5uB9wjUh2l335WjYnYW6b904Dnzp2hUqUquLvnZsqUGcYux6AyXNiZm1sSGRlGliwOEnjpnKIoREaGYW5uaexSRAY2b/vVeCOCeyhhtDurG0vOKo8necZOIFAVyrQT41BpVfr1PnMqjK25DU3zf5Eugm7Tph9ZsGAO48dPpl27jsYux+AyXNhly+ZCcLA/EREhyVrf1NRU/4yJiC8ttI25uSXZsrkYtQaRMUVEq/Ba/Ld+urqrQqkLu8mq1vVhmWfiFKw88rDz4V5Oeuu6/cpq6UDTfA2omKMslmbp54+wtWt/YNmyRXzxRWNatmxr7HKMIsOFnZmZOdmz50z2+hnxNumUIm0jMqqfjz/i0PkX+um+bw6S7ZGudyD7ipVw/eobbse8YNXxUfp1WuZvzBd501dHCIqi8MMPy1i1ajlNm7ZgypQZmbbnosz5qYUQmU5UjJqgsBj+uOzNX9d1fc42jbpBydfXwMyMrLXrkL11O8zs7Ljqd5O1tzbrt51abQxO1tmMVPnHe/nyBevXr6FlyzZMnDgVMzMzY5dkNBJ2QogM7cHLEGZtuZJg/hd+5ykV+wybYsXJNehbTC11pyWfhb3QB12/Ut0omb2YQetNSXnyeLJly04KFCj03g7iMxMJOyFEhnXo/At+Pv4IAA8bDZ7eN3FShZE/6hXuzZvh1Gx0vBvZ7gc9Ysm11QAUdMyXLoNOq9Uyd+4MihT5jFat2lKoUBFjl5QmSNgJITKcV/4RTFh3QT9dOfgWdR/pju5sS5TCffAETP5xSi8gOohJZ2fpp0tmL0a/Ut0MVm9K0Wq1TJs2id27d9K1a09jl5OmSNgJIdI1rVbh0n0/nvuEY2FuyqX7/rwOeDcq+DcvD+IeG4Bj/QZkb90OU6t348bdDXzAr49/41XEG/28IWX7UShbfoN+hpSg0WiYPHkc+/fvoVevfgwc+K2xS0pTJOyEEOnW9j8fcuTiy/cuqxJ8kzqBV7Ep+hnu387A1CL+owJLr67hXvBD/XTt3NVoV6iF0UcK/xj/H4/y998PMGCAF336DDB2SWmOhJ0QIt3RaLWM/uEsgWGxADjaWTKi9Weot60l+t5dALKULUfWzkOxK1U63raPQ56x4MoK/fSwcgPSZEfNH8LU1JT8+Qvi5TVMP5SXiE/CTgiRrvx9/TXrf7+nn57asxJWpw4TNGktAOZOTuT9fjqm1vH7fFRr1Uw9P5+A6ED9vOHlB5Iva9oaPupDxMXF4e39kvz5C9C7dz9jl5OmSdgJIdIFlVrDzhOP+eOSNwDWlmYs6FUOnznTifTRXXPL3rYD2Ro2wuQft9lf8bvBz/f3EK6K0M/rXeIbyriWNOwHSGGxsbF8950XN29eY9++w2TN6mjsktI0CTshRJq35egD/rzsrZ9uXCUPrUpk5emwwfp5BZevinfzCcCcS0t5Hvbuml6lHOXoXLQd5mlopPCPER0dzZAhA7lw4Szjx0+RoEuG9P0vLoTI8N4ERuqDrkX1vDSu4om5Kpbnk8YD4NSkGc6t2sQ7mgPY+WCvPugmVxmFi62zYQtPJVFRkXh59efKlUtMmTKDFi1aG7ukdEHCTgiRJvkGRXH3RTCbDt0HoFvjotQqnYvoJ094PON7AFw6fkW2+l/E2y5Oo2LZtTX6ceaGlx+UYYIOYP36tVy9epnp0+fQuHEzY5eTbkjYCSHSnEv3/Fix51a8eTVK5iTq7h28F8wFIHubdgmCTlEUhp4cp5/+rvwA8mXNk/oFG1Dv3v2pXLkqFSpUMnYp6YqEnRAiTQmPitMH3RcVPWhQwQPbwFc8GToIbaTuYfEcPXvjULW6fhtFUQiMCY7XC8r8Wt9jbW5t2OJTSUhIMPPmzWL48NE4OmaToPsIEnZCiDRBq1WYt/0q916EAFDpM1faV8nF6xWLCLyve9TA3MkJz4nfY2b3brBUraJl453tXPK9pp+3qPZ0LMwsDFl+qgkKCqRfvx48e/aUli3bULFiZWOXlC5J2AkhjMo3OIpVe2/zzOfd2InlC7vQuYg5T4YPQVGpMHdyxq1rd7IUL6FfR6VVc+T5cQ4+Paqf93XR9lTJWSFdPyD+TwEB/vTp053Xr71ZsuQHCbpPIGEnhDAK3+Aoxqw6F29e8XxOfNuuFJHnzvBmnu4h8ay16+D2TTcAQmPDOPL8OPeCH+ET6avfrkDWfHz9WXtcbbMbrP7U5uvrS58+XfHz82PZstVy6vITSdgJIQwqJk7NgAV/xZvn1a4UZQrqgir0r5P4bloPQC6vIdiVKgPApjs7OO9zOd52hbMVpFOR1rjauqR+4UZgY2PLihVrKVu2nLFLSfck7IQQBnP65hvW/XZXP/1V/ULUr+ABgKJW47tpPWFnTgPgPnQ4WYqXQKVVs+nOdq743QCgUd7PaehZF0szy4RvkAH4+fni7JwdNzc3tm7dlekHXU0pEnZCiFRz91kQz3zDefgylNvPglCptQDkdsnC5B6VMH17bS36yRNevn12DsDz++nsDj/PqWOb4u2vR/HOlHeL37FzRvL8+TN69+7K5583YNSo8RJ0KUjCTgiR4k5ce6V/GPyfcjrb8kVFD2qXcQd0jwy8WryQqFs3wMSE7G3ao9Spwtr7v3InSLf9Z06FyW2Xi7oeNclqZW/Qz2FIT548pk+fbmg0Glq3bm/scjIcCTshRIp59CqUCevO88pf9zycazYbvv6iMPlzZsXGyizeXZLhVy7zZsVSAAKymhHc8Qt2Kfd5dfaEfh2vMn0o4lTQoJ/BGB4+vE+fPt0xNTVl7dpNFCiQ8T+zoUnYCSE+maIozPzpCo9ehQKQL6cDbWrlp3g+pwTraiIieDFjKio/X9RmJhxu7skj2ygIf3fzSV2PGrQu0BQzUzODfQZjUani8PLqj7m5OWvWbCBv3vQ3Snp6IGEnhPgkz33CmbLhon66V8sSVPvM9b3rhl84j9+2LajDwzhVxYkr+c2BKABaFmhMtVyVsLPIYoiy0wwLC0umTp2Fm1sOPDwyVtdmaYmEnRDio23/8yFHLr4bQmflsNrkdnfE3//dA+KKRkPYubOEHP+T2GdP8XEyZ8dX78Iwu7UTk6qOxNQkc92Mcf36VR4+fEC7dl/KM3QGIGEnhPgo954H64NuSPvSlCqQcGSB6IcPeTl7un5aW64EO4r6AZArSw66F/+KXHY5DFNwGnL58kUGD+6Li4srzZq1xNo6Y/ThmZalatjt37+flStXolKp6NatG507d463/Pbt20ycOBGVSkXOnDmZO3cuDg4OqVmSECIFLN55neuPAwEY2Lrke4Mu6t5dvOfNBsCxXn3Cv6jC4hurAaiRqzKdirY1XMFpyPnzZ/n22wHkzJmT1as3SNAZSKqdN/D19WXhwoVs3bqVvXv3smPHDh49ehRvnenTp+Pl5cW+ffvIly8f69atS61yhBApICpGRY9Zx/RBV6t0TsoXid97iaIoBP1+UB907kO+406tfCx4G3SVc5SnY5E2hi08jThx4gReXv3InTs3a9duwsXl/dc2RcpLtSO7M2fOUKVKFRwdHQFo2LAhhw4dYtCgQfp1tFotkW+H7IiOjiZr1qypVY4Q4hOpNVoGLfpbP73k25rY2cQfWSD60UPODZqPNiYGUxsbXL/ugq+HPTsvbwbgc49atCmUeQccffjwIXnz5ueHH34kW7Zsxi4nUzFRFEVJjR2vWrWKqKgohg4dCsDOnTu5ceMGU6dO1a9z7do1unfvTpYsWbCxseHnn3+WHwAh0pjoWDWrfr3Bn/+4EWXv3BaYmr57Zk4TG8ut8ZOIePAQgOw1qlNoqBdT/17Kbb8HAHQp05ZmReobtvg0IiwsTH+JJjY2FisrKyNXlPmk2pHd+zL0nw+UxsTEMG7cODZu3EipUqVYv349o0aNYvXq1cl+j8DACLTaT8tqFxf7eHeOiXekbRKXWdpm5Z5bXLznp592cbRmZt+qBAZGALrf89C/T+K/YztKbAx25cpTpG8PHsZEMPHYYh4E6y5d9CrxDWWdSmaKNvu3w4cPMmPGFFat2kDNmpUIC4sD4oxdVpqTEr9TpqYmODvbvXdZqoWdm5sbly5d0k/7+fnh6vru/PSDBw+wsrKiVKlSAHz55ZcsXrw4tcoRQiSTX3AUc7ddJTAsVj/vi4oetK9bALN/9NWoaLU8GtgXRaXCxNISly87kaVuXUZfWcrrcN3wO1kt7elZ4hsKOOY19MdIEw4c2MvEiWMoW7YcHh4exi4nU0u1sKtWrRpLly4lKCgIGxsbjhw5Eu8UpqenJz4+Pjx58oT8+fPz559/UrJkydQqRwiRDKGRcYx+O8acuZkJeXM40KdFMbJntYm3niY6mmdjR6GoVAAUXLoSjYnC2NPTiFTpHhJvW7AZdT1qZpiBVD/Unj2/MGXKeCpWrMzixSuwsbE1dkmZWqoe2Q0dOpQuXbqgUqlo164dpUqVonfv3nh5eVGyZElmzpzJkCFDUBQFZ2dnZsyYkVrlCCGScPaWD2sO3NFPrx5R973rRT24j/f8OaDRYFehEjl79SEoLpSJZ2fp11lSZ2am6OorMWfO/M3kyeOoXr0m8+cvlccL0oBUu0HFEOSaXeqStklcRmobtUbLmFXnCAyLAaBOWXe++aLwe4/I/HdsI/joYQDcuvUka42aqLVqvj0xFoB8DnmY+sVwQoNiDPcB0iCVSsVPP22kc+cuWFq+G3cvI/3cpLR0e81OCJH2/Xsonr4tilO5mFuC9RRF0Q2s+rduhPHc343E9rNiCUYPH15hEJZmFkDmDLtff91FrVp1cHbOTvfuvYxdjvgHCTshMqF1v93h0aswfIN019fMTE1YPaLOe4/mNFGRvFq8kJjHjzAxNyd64mCWvz7Gs2Mb9OuUzl6cXiW/MVT5adKaNStZvnwxPXr0wctrmLHLEf8iYSdEJqIoCkt23dD3gJLHzY5OnxeiSJ6Ez7dGP3yA78b1xPm8IcTOjI3/77z53hb9Op72HvQp1QVHq8zbIYSiKKxcuZTVq1fQrFlLBg781tglifeQsBMik3joHcKcrVfRvL3OPb13ZXI6JxxOJ/LWDV4tWqCfPlfanvPFdXdjZrV0oGrOClTIUZYctq6Z9k7L/1MUhcWL57Nhw1pat27H+PFTMDPLvDfmpGUSdkJkAhqtlpk/XQHA0c6SKT0qYW9rGW8dbUw0L+fNIfbZUwCs8ubDokNLzntvBaCiWzm6Fe9o2MLTuMjISE6ePEb79p0YM2YCpqaZa5ii9ETCTogMLCA0muNXXvH7+RcAeLjaMaVHwrHTFEXh0aD+AFjmzEWO3n2JdcvG2NPTAGjkWY/mBRoZrvA0TqvVotVqsbOzY+PGbdjbO2T6o9y0TsJOiAxq/5ln/PrXk3jzJnStkGC9kL9OEHzwNwDMnZzJO3UGh54dY//pQwAUcsxP0/xfpH7B6YRWq2Xq1ElERkYwc+Y8HBwy7/XK9ETCTogMRFEUrj8OZMmuG/p51UrkoEeTz+J13AygDgnB/+dthF84D4BT85Y4t2jF36/Osv+JLujqedSkVYEmmW4U8cRoNBomTRrLgQN76dNngJy2TEck7ITIICKiVXgtfjcEj7mZKeO+KY9nDvt466lDQvDbvpWISxcAyFKyFK5fd8XC2ZmQ2FC23/8VgK+KtqV6rsqG+wBpnEqlYsKEURw6dJABA7zo02eAsUsSH0DCTogMIComftBN6VEJD9eEPUn4bt5A6MkTANgWK45TsxbYFi6Coigcf3mKA0+OANCqQBMJun+ZOnUihw4d5Ntvh8sD4+mQhJ0QGcD/B1U1MzVhzciEfVoqajUv58wk5sljADzGjMemQEH98uXX13E3SDfunKNVVurkrm6AqtOXNm3aU6xYcTp2/NrYpYiPIGEnRDp3/o6v/vX7gi746GH8d2wDwNTOjtzDRmCdx1O/PEYdow+6SVVG4GrrksoVpx8xMTGcOnWS+vUbUqZMOcqUKWfsksRHkrATIh279iiAVftuA/D9vx4piHn2jFeLF6AJDwPAoVoN3Lr1wORfN1Usu7YOgLoeNSTo/iE6OoohQwZy4cI5fvnlAPnzFzB2SeITSNgJkQ75h0QzdeMlIqJ148mVLuBM7rfX6BStlqCDBwjcsxsAm8JFyNl/IOb2DvH2EREXyahTU/TTrQs0NVD1aV9kZAReXv24evUKU6fOkqDLACTshEhn/EKiGf3DWf109yZFqVkqFwDa2FieDB+CNjoagJx9+mNfKeGNJoqiMPvSEv30kLL9MvX4c/8UHh7OoEF9uHXrBjNmzKVRI/kjICOQsBMiHbl0z48Ve24B0KCCBx0/L6jvuSPs/Dl81vwAgHX+AuQeMRpTC4t42yuKwq3Au/xwY4N+3tK6s+Q5un84c+YUt2/fYvbsBdSv39DY5YgUImEnRDoxa8sVHrwMASC3SxY61S8EgDYmhlfLFhN97y6YmuLUtDnOLVrF674qRh3LkqurCYwJIkIVCehGLOhd8hsJurcURcHExISGDRtTokRJ3N1zG7skkYIk7IRI4y7d82PtgTvEqbUA9G9VgopFdcPthF++iM+P61BiYzAxNyff3AUJrs0BfPfXBP1rTwcPGuf9nJLZixnmA6QDQUGBDBkygCFDRlCuXAUJugxIwk6INOzW00D9aUuAOf2rkj2rbridkON/4rdlMwCu33Qla633D756O/DdSOTL6s6WDov/xd/fj759u/P69Svi4uKMXY5IJRJ2QqRBgaExHL30kiMXXwLQqX4hGlTw0C/32bCOsFN/Y5HdhdwjRmHhnD3BPhRFYdDxUfrpBnneH4aZma+vD717d8Xf359ly1ZToULCESFExiBhJ0Qao9ZoGbHyjH66RfW8+qDTREbyZs0PRN26iYm5Oe5Dvks06L49MVY/PaRsXwplk9vn/ykwMICePb8hODiIlSvXygPjGZyEnRBpSKxKQ//5JwEons+JAa1KYGOl+zUN2PMLQQf2A2BiaUn+uQsxy5JwpPGwuHDmXVqGRtEAML/W91ibWxvoE6Qfjo7ZqFKlGi1btqVkyVLGLkekMgk7IdKQ/wedCTC0Q2lMTUzQqlS8WjiP6Ae6a285+vTDvmLl956S9IvyZ8q5ufrp2TUnSdD9y/PnT7G2tsXNzY3x46ckvYHIECTshEgDnr4JY+rGS/rpNaPqYmpigqJW8+S7IWijIrHI7kKeiZMxs014NAdw2fc6P97eAoCpiSmL68yQxwr+5fHjR/Tp0w1Pz7ysW7dZrmFmIhJ2QqQB/wy6xV41MDUxIfrJE17O+B7QdfnlMXLMe7fVKlpW3djArcB7AHzhWZeWBRqnftHpzIMH9+nbtztmZmaMGzdZgi6TkbATwoheB0Qyfq1upHB7WwsWe9VEExXFg779QaO75maVNx+5h4967/YqjYohJ8fpp+vnqS1B9x53796mX78eWFlZs2bNBjw98xm7JGFgEnZCGImiKEzZcFE/Pbl7JaKfPMZ7/hzQaHQPic+eh3lWx3jb+Eb5cSfoAb883B9vf0vqzJT+Ld9DURTmzZuJrW0WVq/egIdHHmOXJIxAwk4II4iOVbP31FNUai3WlmasGFab4GN/8HLrTwC4dOqMY7368U61HX1+gj2PDybYV+Uc5fn6s/ZyfS4RJiYmzJ27mJiYGHLlcjd2OcJIJOyEMCBFUdh0+D4nr73Wz5vcoxJhZ07j/zboco8cg23hIvG2++3JEQ4++wMAB0t7WhdsSlGnQjhY2huu+HTm0qUL7N69kylTZuDk5GzscoSRSdgJYSB/XvZmy9EH+unPy+emWokccHAXPsf+BCBnv4H6oDv07BhX/K7zKuKNfpv+pbpTIvtnhi08HTp37gxDhgwgVy53wsPDcXJyMnZJwsgk7IRIZQEh0UzZcJHIGDUAnjns6duiOG6O1ngvmEvIvbtgYoLnlGlYvT3N5hvpx/4nhwAo5lyEaFUMvUt2IauVHMkl5dSpvxg2bBCennlZtWq9BJ0AJOyESDXRsWoOnnvOb2ef6+cN7VCakvl1p9SCjxwm+t5dbIsVx91rKCbmul/HYy//1t980iRfA5rma2D44tOpv/46znffeVGwYCFWrlyHo2M2Y5ck0ggJOyFSwaHzL/j5+CP9dBEPR0Z11vW9qCgK/tt+IuTYn5hny0auAYP1Qfc87KU+6Mq6lpKg+0BOTtmpUKESs2cvxMEh4VBHIvOSsBMihW08dE9/A0qh3FkZ1KYk9raWgC7oXi1eQNStm1i4upFn7ARMrXXdee18sJcT3qcB+KpoW6rnqmycD5AOPX78iAIFClKiRElWrlxn7HJEGiT3KguRQuJUGsasPqcPup5NP2PM1+X1QRd66m+ejRlJ1K2bYGJC3mkzMbOzQ1EUzr+5rA+69oVaStB9gH37fqV9+xb8/vsBY5ci0jA5shMiBSiKQr+3nTgDDOtQmhJvr82pw8J4MsxLv8ylU2cc636OiakpT0KfM//ycv2yNgWbUcejuuEKT+d2797J1KkTqVSpKnXqfG7sckQaJmEnRApY//s9/evVI+pgbqY7aaIoCi+mTQbAplBhcg36Nt6wPAuvrAQgV5Yc9CzRmRxZ3AxXdDq3Y8dWZs78nurVazJ//lKsrWV0B5E4CTshPpG3fwSnbuiehVv5XW190Gnj4ng0WNfHpX2VquTs1TfedkNPjEOraLEys2Rc5WEGrzs9e/LkMbNmTaVOnXrMmbMIS0tLY5ck0jgJOyE+0cR1FwBoX6cAVhbv+qZ8Mf170GiwzOWeIOh2PthLnFYFwLRqYxEfJn/+AixfvpqKFStjYSFBJ5ImN6gI8ZGiYtT0nnNcP924iifwtrPmTeuJe+UNZmZ4Tp4ab7s7gff1N6OMquiFrYWt4YpO59atW8W5c2cAqFatpgSdSDYJOyE+kKIonLzizaBFf6HRKpgAM/tWAXSnLh8PGUToXycxtbEh/5wFmJi++zULjglh+XXdrfH9SnUjj31uY3yEdEdRFJYvX8zSpQv5888jxi5HpENyGlOID+AbFMWY1ef00w62FiwYVANTUxPUoSE8Gfmdfhy6/PMWYWplFW/7NTc3A1DBrQwlsxczXOHpmKIoLFo0j40b19G6dXvGjJlo7JJEOiRhJ0Qy3X4axPwd1/TT03pVJld23Z2V4Zcu8uYH3SME9lWqkqNnn3jD8yiKwnX/WzwPfwlA9+JfGa7wdExRFObOncnWrZvo0OErRo8ej6mpnJASH07CTohkCIuM0wddo8p5GNihLP7+4QAE/X6QgF9+BiBboya4tOuQYPt5l5fzLOwFAK0LNjVM0RmAoihERkbw9ddd+e670fH+gBDiQ0jYCZEMQ5aeAiC3ix0d6hbUz4+4fk0fdLkGDsaubPkE24bGhuuDbkSFQXjaexig4vRNo9EQEhKMs3N2Jk2ahomJiQSd+CRyPkCI/6BSa5m8/oJ++vuelfSvtSoVr1csBSDPuInvDTqVRsXY07q7MdsXaklehzzypZ0EtVrNxIlj6NKlI+Hh4ZiamkqbiU8mYSfEf9h06B4vfCMAmNGnSrxlfls3g0aDU5NmWOfLn2Db52EvGXJynH5augFLmkqlYty4kfz22z5atWqHvb2M3ydShpzGFCIRT9+EcfqWDwCLvGrgYPvumS6/YycI+/svMDHBuXXbBNseevYn+58cBsDN1pUJlb8zTNHpmEoVx6hR33Hs2FGGDBlBt249jV2SyEAk7IR4j1cBkUzdeAmA4R3LxAu6V0sXEXn9GgCeU6YnOMX2JtJXH3StCjShgWcdg9Sc3i1fvoRjx44yYsRYOnfuYuxyRAYjYSfEvxy74s1PRx4A4OFqR7G8TvplAb/+QuT1a5hYWOA5eRqWbvE7bn4Z/opZFxcD0K5QC+p61DBc4elc9+69KFr0Mxo1krtVRcqTsBPiH14HROqDrmyh7AxuWwoAbWwsb1atIPLGdSyyu1B+xWKCwuL02ymKwoIrK3gS+hwAC1MLCbpkiI6OYt261fTu3Z+sWR0l6ESqSfIGlcjISKZMmULXrl0JCQlh4sSJREZGGqI2IQxKq1UYv/Y8AC1r5HsXdKo4Hg3qR+SN65g5OJB71FjM/tEzik+kL4OOj9IHXfvCLVlUZ7rhP0A6ExkZwcCBffjxx9VcvXrZ2OWIDC7JsJs2bRoODg4EBgZiZWVFREQEEydKdz0i47l03w8AUxMTWtbIB+iO2F4vWQSKglXefBRYsASLbNn022gVLVPPz9dPz6g+njq55a7LpISHh9O/fy+uX7/KzJnzqFKlmrFLEhlckmF39+5dhg4dirm5OTY2NsybN4+7d+8aojYhDCY2TsMPe28DMLFbBQA00dG8mDqZqLt3sC5QkDxjJ8TbJkoVxeDjowFwtMrK8npzyGrlYNjC06HQ0BD69u3OnTu3mTt3EQ0bNjF2SSITSPKa3b/7odNoNNI3nchwTl57Beiu03m42uG39SdC/zqBolZjX7kKbt90izd6wYOAJ4z/e65++rvyAwxec3rl7++Hv78vCxYsoVatusYuR2QSSYZdxYoVmTt3LjExMfz9999s2bKFSpUqJbWZEOmGoihsP/YIgM71CuA9bzbR9+9ham1Nzn4DsStTNt76Gq2G8Sd0Qedm68LEKiMMXnN6FBUViY2NLQULFmb//qNYW1sbuySRiSR5iDZ8+HBsbW2xt7dn4cKFFClShNGjRxuiNiEMouds3QCsBXLaETpvKtH375GlbDkKLFkRL+jC4yL45eF+vE6MAcDTwUOCLpn8/Hzp3Lk9P/64GkCCThhckkd2J0+eZODAgQwcOFA/b8+ePbRq1So16xLCIDYduqd/3fz0WlTaOByqVSdHj976+VGqaH68vYW7QQ/08zwccjK8/EBE0nx83tC7dzcCA/0pU6acscsRmVSiYXfs2DHUajVz5sxBURQURQF0nbQuXLhQwk6ke68DIjlx7TUAU0qpiH6ke27un0F35vVFttzbqZ9ukrc+NXNXpYB7Lv0QPyJxr15506dPN0JDQ1i5ch2lS5dNeiMhUkGiYXf37l3OnTtHYGAgmzZtereBuTk9e0qfdSJ9i1Np9M/U1coB0bu3AVBg4VL9Outvb+WS7zUAquWsSJtCzbExl9NvyRUTE0Pv3l0JDw/nhx/WU6JESWOXJDKxRMPu/6cut2zZQufOnT9q5/v372flypWoVCq6deuWYD9Pnjxh0qRJhIaG4uLiwoIFC8iaNetHvZcQyeUbFMWY1ecA+Mw2lqqndgCQb9ZczN72sv86wkcfdF2LdaRSDjn99qGsra0ZOHAIBQoUpGjRz4xdjsjkTJT/n59MRFxcHCdPntT3mqLRaHjx4gVDhw79zx37+vrSqVMndu/ejaWlJR07dmTBggUULKgb+FJRFBo1asS4ceOoVasW8+bNQ1EURoxI/gX/wMAItNr/LD9JLi72cjoqERmxbW49CWTBz9cBsDE3weveRkwAz++nY5XLHYDDz46x78khADoXbU+1XBUT7Ccjtk1KCQp6zb17j6lWraaxS0lz5OcmcSnRNqamJjg72713WZI3qAwdOpSXL1/i7+9PsWLFuH79erIePThz5gxVqlTB0dERgIYNG3Lo0CEGDRoEwO3bt7G1taVWrVoA9OvXj7CwsOR+JiE+2P0Xwfqg+zy/NRWP6O4MzD1yjD7ofCJ99UFXLWel9wadSNz9+/fo378HNja27NlzEAsLy6Q3EsIAkgy7u3fvcuTIESZPnkz37t1RFIUpU6YkuWM/Pz9cXFz0066urty4cUM//eLFC7Jnz86oUaO4c+cOhQsXZsKECe/bVaISS/AP5eIiA0QmJqO0zbM3YczeehWAkrlsqHBkFQCl5szEvkhhAM57X2X+eV0AtivelA4lmv3nPjNK26SUGzdu0LdvN2xsbNi+fRu5cjkbu6Q0SX5uEpeabZNk2Lm6umJubk7evHl58OABjRs3Jjo6Oskdv+/s6D/H/VKr1Vy4cIGffvqJkiVLsmjRImbNmsWsWbOSXbycxkxdGaVtNFotszdeBODzfFZUPKoLOpeOnYlxykmMfzjhcRHMP60Lui8Lt6Kma9X//OwZpW1Sys2b1+nfvxf29vbs3v0LNjbZpH3eQ35uEpfapzGTfKjc1taW/fv3U7RoUX7//Xfu379PSEhIkm/q5uZGQECAftrPzw9XV1f9tIuLC56enpQsqbtDq1mzZvGO/IRICRfv+dF7zgm8/XXXnEv9sQEAl6++Jlv9BgD4Rvox+tT3AJR1KUmt3NUSDMgq/tvRo4fIli0bP/74E3ny5DF2OUIkkGTYTZw4kbt371K9enVMTU35+uuvk/XoQbVq1Th79ixBQUFER0dz5MgR/fU5gLJlyxIUFMS9e7qHeo8dO0bx4sU/4aMIEZ9Wq7Byzy0AnIhh+OOfsFJU5Js9j2z16uvX2/v4dwCq56pE9+JfGaXW9EqtVgMwZMgINm/eQc6cuYxckRDvl+RpzLx58zJy5EgAFi1aBMCjR4+S3LGbmxtDhw6lS5cuqFQq2rVrR6lSpejduzdeXl6ULFmS5cuXM378eKKjo8mRIwdz5sz5tE8jxFuBoTGMWHkGAHdNKN883QtAweWrMH07Fl2UKpqDz45yPUA32sFXRdsZp9h06ty5M8yaNZXly9fg7p4bR8dsSW8khJEkGnbe3t4sXLgQR0dHhg8fjo2NDRERESxdupQtW7Zw69atJHfevHlzmjdvHm/emjVr9K9Lly7Nrl27PqF8Id7v/8/RZYsLo53379h+Vgz3ocP1IxeExoYx9vQ0/frtCrUwSp3p1d9/n+S77waTN28+bGxsjV2OEElKNOzGjh1L4cKF8fPzY9WqVdSoUYOhQ4fi4ODA2rVrDVmjEB9kxIrTqDVaAPq+2INt8RLkHjo83jrjz8wAoKBjPgaW7omlmdwin1zHj//JiBFDKFSoECtXrpMjOpEuJBp2Pj4+bNq0iZiYGNq0acPPP/9Mt27d6NGjB+bmSZ79FMIofjv7jMCwWAAGP/0Zm8JFEgTdsRd/oVV0YTi0XH+D15ienT17mhEjvuWzz4qzfPkaHBxksFqRPiSaWra2ulMT1tbWhIaGMmfOHGrUqGGwwoT4EBHRKqZtuoRfsO6xmF7P95KjYlly9OwTb72fH+zlpPdpADoWaWPwOtO7EiVK0bZtBwYPHoadXco85yqEISTrEM3Z2VmCTqRpXov/1r9u7HuG7KpQ3Lr2iPcIQUhsqD7oxlYairtdToPXmV79/fdJKlSohL29PWPGTDR2OUJ8sEQfPfjnl4SZmZlBihHiY5y/46t/PfrRJkqHPyLPhMmYWlgAEKOO4eDTo4w7PR2ACm5lJOg+wC+//IyXVz/9wKtCpEeJHtndv3+fcuV0Pb3HxMToXyuKgomJCVeuXDFMhUIkIlalYfqmy3j7RwDQ9vUxAHIOGIy1Z15A93jBiL8n6bfxsMslz9J9gO3btzBr1lRq1KhNr179jF2OEB8t0bA7evSoIesQ4oM8eBnCrC3v/uBq//oPCkS9Ju/0WVi65QBAq2jjBd2M6uPJaiU3VCTX5s3rmT9/NnXqfM6cOQuxtJQ7VkX6lWjYubu7G7IOIZJt5Z5bXLznB4BnDnu+enMUJeo1zq3a6INOURQGHx+t32Z5Pemw4EOEhoawfv1aGjRoxIwZc7F4e0pYiPRKniEQ6cqZW2/0QTe4TUmyrp+D2t8fm6Kf4dT0XQcGp16f079eWHtagv2I9/t/B+5ZszqyefMO3NxyyKNGIkOQn2KRbkREq1h74C4AvZsUwfXwFiL9/QFwH/St/qaqkNhQtt//FYAxFYfIA+PJpCgKy5YtQqvV4uU1DHf33MYuSYgUk2RH0EKkBRqtVv94QctqnrhumE3k9WtYeXhQaNU6TK2tAd0X9v/vuqycozy57aVj4uRQFIWFC+ewbt0qwsJC3ztElxDpWZJh5+/vT58+fWjYsCEBAQH07NkTPz8/Q9QmhF7/+X/pX9e18EUTFoZT85Z4TpqKydtHY7SKlkVXf9Cv16XYlwavMz1SFIU5c6azadN6OnbszPjxUzA1lb+DRcaS5E/0lClTqF+/PlZWVmTNmpWiRYsyfvx4Q9QmMrnwqDj6zT9Bj1nH9H1drhleG591uue9nBo2erduXASDj4/mUchTACZVGWH4gtOpWbOmsm3bT3zzTTdGjRovY/mJDCnJsHv16hUdOnTA1NQUCwsLRowYwZs3bwxRm8jEwqLi+HbJKeJUupArmd+ZqT0r8bifbixFm0KFMbW2AXTX6P4/+CrAlKqjcbV1MXzR6VS5chXo2bMvw4aNkqATGVaSN6iYmJig1Wr10xEREfGmhUhpt58GMX/HNf30j6PrAfB4+BD9vNzDRwG63lH+f43O3NScxXVmGKzO9EytVnP37m1KlixNw4ZNaNiwibFLEiJVJRl2X3zxBcOHDyc8PJzt27ezc+dOGjdubIjaRCajKAp9551ArdHdHFGjVE6++aIwAKqgIDQhIQAUXLkGEzMz4jQqvvtL109jkWwF8Srb5737FfGpVCrGjh3B8eN/snv3AfLk8TR2SUKkuiTDrl+/fuzZswetVsuZM2f48ssvad++vSFqE5nMst039UHXvXFRapbW3UmpiYjg6chhAOTyGoqJuTkPgh+x+Oq7vhoHl+lt+ILTobi4OEaNGsbx43/w3XejJOhEppFk2G3fvp1mzZrRqlUrA5QjMqvzd3y5+jAAgFXD62Bh/u5y8vPvdV1+OTVrjl2p0gz/ayLR6hj98qV1Z8m1pmSIjY3lu++8OHXqJKNHj6djx6+NXZIQBpPkDSoXLlygfv36jBkzhmvXrhmgJJHZXH8UwKp9twHo0qhIvKALv3IZdVAg1vnyk71VWw48OaIPusFlerOs7mxMTeQ2+eTYv38Pp06dZPz4KRJ0ItNJ8shuwYIFhIaGcuDAAaZNm0ZsbCzt2rWja9euhqhPZHBxKg2Ld90AoF45d+qUedcnq8+Pawk7cwoA12+68ibSl9+f/QHAlKqjyG7jbPiC07G2bTtQqFBhSpcua+xShDC4ZP1JnDVrVr788kv69u2LjY0Na9asSe26RCYQE6em3/yTALg62vD1F0X0y4IOHtAHnceocVw292Ha+fkAeNp7SNAlU0REBCNGDOH582eYmJhI0IlMK8kju9u3b7N7924OHTpEsWLF6NWrF/Xq1TNEbSIDexMYybg15/XTM/pW0b9WBQUSsHsXAAWX/UC0mZatf+v+wGqc93Oa5W9o2GLTqbCwMAYN6s3t27do3LgZnm/H+BMiM0oy7AYOHEjbtm3ZuXMnuXJJP4Pi0ymKog86a0szlg+tpb/BJM7HhzerVgCQy2sIptbWLL24GIAyLiUl6JIpNDSE/v178uDBA+bOXUy9evWNXZIQRpVk2J04ccIAZYjMZMWeWwCYm5myYlht/XxNRATPJo0DjYZsjZqQpWRpJp6ZRWBMEAC9SshNFckRHBxM377defbsCQsWLKVWrTrGLkkIo0s07Dp16sS2bdsoW7ZsvNu6FUXBxMSEK1euJLapEIkKCInm8n3dsDyLBteItyzo0EHQaHAfNoIsxYpzN/CBPuhGVfSSxwuSycrKEmdnZ4YMGU61ajWS3kCITCDRsFu8WHfq6MCBAwmWyfAf4kNptQqztl7hkXcoAF9U9MDW+t2PX/TjR4T8cQQzBweyFCuORqth2fW1gC7o8tjL2GpJ8ff3w9bWlixZ7FixYq38cSDEPyR6N6arqysAkyZNwt3dPd5/w4YNM1iBImMYuPAvfdB1/LwQX9YrqF8W9eA+L2dOQ1Grce3YGYDNd38GIIetqwRdMvj4vKFHj28YPfo7AAk6If4l0SM7Ly8vnj59ysuXL2nevLl+vlqtlrGuxAe5/iiAWJUGSNg7CkD4ubMA5OjTD/tKldlydxcXfa8C0K9Ud8MWmw69euVN795dCQsLY/r02cYuR4g0KdGwGzlyJK9evWLChAlMmDBBP9/MzIxChQoZpDiRvqnUWn796wmHLrwA4PuelRIEnf/P2wn96wQ2hYsQUSwv0/+eQoQqEoCxlYbiYivP0/2XFy+e06dPN6Kioli9ej3FipUwdklCpEmJhl3u3LnJnTs3hw8fllMi4oPFxKkZsODd6OKtauQjt4tdvHUirl4h+MghAKy7f83kCwv0y8ZUHIK7XU7DFJtOKYrCmDHDiY2NYc2ajRQpUtTYJQmRZiV5N2a5cuXkbkzxwVbuua1/vcirBg62lvGWa2Oieb18CQD55y9m2JVZAFTJWYHORdtJf5fJYGJiwvTps1Gr1RQsWNjY5QiRpn3U3ZhC/JepGy/x9E0YAOtG1U1wZkAdEsyT4UMByFq7DpteHkCj6K7pfV20vZxJSML9+/f444/DDBjgRd68+Y1djhDpQpJ3Yzo5OeHv74+7uzt//PEHy5Ytky8jkajfzz/XB9237Uol+FmJuH5NH3TWBQpCm6Zc9rsOwITK38nPVhJu375J795d2b9/D8HBwcYuR4h0I8lzRWPGjOHPP//kxo0bbNq0iVy5csW7YUWI/3sTGMnO448BmNWvKqULZo+3PPaVN6+XLgIgR+9+5Bkznj9f6q7rff1ZB3JkcTNovenNjRvX6Nu3O3Z2dqxbtxknJydjlyREupFk2L18+ZLvvvuO48eP07p1awYPHkxISIgBShPpzZQNFwHo0eQzXB1t4i3TxsXxcuY0QDdcj0PlKtwLesjp17o+MqvkKG/YYtOZK1cu0a9fD7Jlc+bHH3/C3V2ePRTiQyQZdiqVCoBTp05RpUoVNBoNUVFRqV6YSF8OX3hBnEoLQLWSORIsf7V4AdqYGJxbtsaxdl3iNHEsvaYbyaCiWzk5fZmE0NBQ3N1zs27dJnLkkLtUhfhQSXYEXa5cOZo0aYKZmRnlypWja9euVKtWzRC1iXTi7rMgdhx7BMDk7hUx/Vdwxbx4TvT9e5iYm+PUVNdBwdCT4wEo6JiPbsU7GrbgdCQoKAgnJyfq1v2cmjVrY26e5K+sEOI9kvzNmTBhAlevXqVo0aKYmprSs2dPatWqZYjaRDpw93kwc7dfA2Bg6xLkcbOPt1xRq/WnL92HjQATE5ZfW6df/m3ZvgarNb35668TjBo1jLlzF1GjRi0JOiE+QZK/PWZmZvj5+fHLL7+gUqmoXr26dBcmAF0PKXO36br1KuLhSPkirvGWq0NDeDpmJIpKhWP9BtgWLsL621u5E3QfgNEVv5Xn6RJx7NhRRo4cRuHCRShZspSxyxEi3Usy7NatW8e+ffto3bo1iqKwYcMGfHx86N+/vyHqE2nY/29Iye2ShVGdyyVYHvDLTpS4OOzKlselQyfuBj7gku81AKZVG0s2a0cDVpt+HD78O2PHDqd48RIsX74We3v7pDcSQvynJMNuz549bNu2DTs7XVdP7dq1o0OHDhJ2mdy0TZd4HaDrw3JC14rxlimKgt/mDYSdOY1ljpzkGjgYRVH0Q/a0L9RSgi4R9+7dZcyY7yhduizLlq0iSxa7pDcSQiQpWRcB/h90APb29nLtIJNb+ssNnrzWPTg+tVflBJ07v1m1gohLuqO+XF5DCYgOYtLZWfrldTyqG67YdKZIkaKMHTuJpk2bY2Nja+xyhMgwkrxg4u7uzsaNG1GpVKhUKjZs2ECuXLkMUZtIg568DuPqwwBA1+ele/Ys8Zb7bd+iD7pCq39k2ctd8YJuUZ0Zhis2Hdmz5xcePXqIiYkJ7dp9KUEnRApLMuymTJnCH3/8QZkyZShTpgxHjhxh0qRJhqhNpDF7/n7CtE2XAOjXsni8zp0VrZbn06YQ8sdRAJzGjGTp9bU8Dn0GQOuCTZlTczIWpnJW4N+2bdvM5Mnj2Lx5vbFLESLDSvKbx83Njc2bNxMdHY1WqyVLlixJbSIyoJPXXrHv9DMAapXORaXP4nft5b9zB7HPnmKePTt5J09l2d3N3A/WPXs3uuIQPOzlbMD7bNr0IwsWzKFevQaMHz/Z2OUIkWElemT37Nkz2rZtS7ly5RgwYABRUVESdJnUS78INh7SPS4wu19VujWOP25a2IVzhBw9jImFBflmzCFQG6UPuuX15kjQJWLt2h9YsGAOX3zRmNmzF2BhYZn0RkKIj5Jo2H3//fe0bt2anTt34unpyZw5cwxZl0gjImNUTPrxAgAtqufF5V99XipaLT5rVgHgMXocMdpYJp+bDUBDz3qGLTYdUavVXLhwjqZNWzBjxlwsLCyMXZIQGVqipzEDAgL4+uuvARg+fDgtW7Y0WFHC+BRFYdvhe2w9ojuis7OxoFXNhGOnBR86CIqCU5NmWHvm5ecHewGwNrOiRYFGBq05PVAUhZiYGGxsbFi8eCWWlpaYmZkZuywhMrxEj+z++XiBmZmZPG6Qyfx9440+6JwdrFnybc0E6yhqNYEH9unWadkagLNvdHdizq4pNzH9m6IozJ8/iz59uhEdHY2NjY0EnRAGkmiCKYoSb1p6pc884lQaNvx+D4A5/auSPavNe9d7Mf17lLg4HOt9jomZGZGqKOI0cdia22Aud13Go9VqmT17Gjt2bKVTp2+wtrY2dklCZCqJfiP5+Pgwbdq0RKfHjx+fupUJo5m99QoART2zJRp0kbdvEfvyBZiY4NKxMxqthpF/Twbg8zy1DVVquqDVapk2bRK7d++kS5ceDB06Qv54FMLAEg27zp07/+e0yJgu3vPj6ZtwAOYMrklAQESCdRRFwf/n7QDkm7OAaRcX4hPpq1/+hWcdg9SaXixdupDdu3fSq1c/Bg78VoJOCCNINOwGDRpkyDpEGqAoCiv33ALg23alEv1SDjn2B3GvvHFu2Zotrw/rg655/kbUyV1dRjL4lzZt2uPk5Mw333QzdilCZFpyYUXozfjpMgClCjhTumD2964TfPQw/ju2ATAz62WifWMAmFptDE7W2QxTaDqgUqnYv38PrVu3w8MjjwSdEEYmYScAePw6lMevdJ0792le7L3rRD24rw+6230aEB1xHYDxlb+ToPuHuLg4Ro4cyokTf5I7tweVKlUxdklCZHpyvkkAsO7AXQAGtymJrXXCB5wjb9/Ce85MMDPDY8o0/ngbdJOqjCRnFrcE62dWsbGxDBs2mBMn/mT06PESdEKkEUmGnVarZe3atYwaNYqIiAhWrVqFRqMxRG3CQPadeopPUBQAZQu7JFgece0qrxbOAyDXgMGc1z4HoGT2z3C1ff/pzswoOjoaL6/+nD79F+PHT6Fjx6+NXZIQ4q0kw27OnDk8ePCAGzduAPD3338zc+bMVC9MGM6eU08BGPdN+QTLYl9583rZYgBcu3QjtrAnOx/qeknpULiVwWpMD+7fv8v161eYPHk67dp9aexyhBD/kOQ1u7Nnz/Lrr7/Spk0b7Ozs+PHHH6XrsAxk/5lnABRwd6CAe9Z4y0KuXef5pO8BcOvek6zVa/Lt8TEA5M+aV67TvaXRaDAzM6NMmXIcOHCU7NkTHh0LIYwrySM7c3NzTE3frWZpaSldh2UQQWEx/PrXEwDa1ioQb1nUg/vcfht0OXr2IWv1mtwKuIta0Z3C/q78AMMWm0aFhYXRvXtnfvtN122aBJ0QaVOSYVe4cGG2bNmCRqPhyZMnTJw4kaJFiya1GQD79++nSZMmNGjQgC1btiS63okTJ6hXT3rIN7RNh3V9X3b8vBBFPeMfpflt3giA+9DhOFStRnBMCCtv6AYX7VOyq2ELTaNCQoLp27cbd+7cxtZWhr8SIi1LMuzGjRvH7du3CQwMpFOnTkRGRjJ27Ngkd+zr68vChQvZunUre/fuZceOHTx69CjBegEBAcyePfvjqhcfLSgshhuPAwGoXz53vGWBB/YR9+Y1OZs2IUvxEjwNfc74MzMA+DxPLUq7FDd4vWlNQEAAffp04/HjRyxatJy6dT83dklCiP+Q5PlIOzs7ZsyY8cE7PnPmDFWqVMHR0RGAhg0bcujQoQQ9s4wfP55BgwYxf/78D34P8XE0Wi3DV5wBoEEFD0xN3/WUEn75IoF7dgOQ5+tOBEdqmHd5OQAFsualTcFmhi84jYmOjqJr1468ePGCJUt+oEqVasYuSQiRhCTD7p+dP/9TUh1B+/n54eLy7vqFq6ur/o7O/9u0aRPFihWjdOnSyalVpJDec07oX3eqX0j/Os7XhzcrdcGWvf2XPIv2Y+wx3VG3rbkNw+Q6HQA2Nra0atWKwoVLUKFCJWOXI4RIhiTD7v9HZqDrAunUqVOULVs2yR3/e4ggiD9M0IMHDzhy5AgbNmzAx8cnmeXG5+xs91Hb/ZuLi32K7Cc9mL3pov717tnNsTDXnclWR0VzYZLuD5iiY0ejKuaJ18F3Y9J9//l3uDhmnnZ6n1evXhESEkLx4sX59ttvjV1OmpaZfqc+lLRN4lKzbZIMu3+fduzbty99+/ZNcsdubm5cunRJP+3n54erq6t++tChQ/j7+9O2bVtUKhV+fn589dVXbN26NdnFBwZGoNUmDNUP4eJij79/+CftI734Ye8tLtz1A2D+wOqEBEcCuj9MXnw/CUWtJnubdjxzsWP226DLY5+bURW9QEWmaaf38fZ+Se/eXTE3t+DXX38jZ85smbo9/ktm+p36UNI2iUuJtjE1NUn0IOiDuwuztbXFz88vyfWqVavG2bNnCQoKIjo6miNHjlCrVi39ci8vLw4fPszevXtZvXo1rq6uHxR04sOoNVp90I3rUp5s9lb6Zb4b1xP78gUWOXLwqlJ+Zl9cAkBex7dBl8k9f/6Mnj2/ISoqkjlzFsijN0KkQ0n+1k6dOlV/+lFRFG7fvk3+/PmT3LGbmxtDhw6lS5cuqFQq2rVrR6lSpejduzdeXl6ULFny06sXyfbjb7q+L7+o6EGBXO8eHleHhBB26i/MnZxwGDeSeWdnAfCZU2GmNBia6f8KffLkMX36dEOj0bBmzSYKFy5i7JKEEB8hybDLli3+81ctWrSgRYsWydp58+bNad68ebx5a9asSbBe7ty5OXbsWLL2KT6cVqtw7o5uzLl2deI/PO637ScAHJs0Z8LboGvkWY/mBRoZtsg06scfVwOwdu0mChQoaORqhBAfK8mwe/HiBXPmzDFELSIVaLUKveYcByCPmx3mZu/OXEc/eULE5UtY5snDBO3v+vnN8jc0eJ1p1YQJ3xMQ4I+7e+6kVxZCpFlJXrO7f//+e++sFOnDrpOP9a8ndq2of60OCeblDF13YDvqvrugu6j29ERHKM8sbt26Sd++3QkLC8XKykqCTogMIMkju+zZs9O0aVNKly5NlizvukRK6jk7YXyxKg2Hzr8AYOWw2vEeHn85V/f8XEyr+rxQ6Z5/nFdrChZmCceyy0yuXbvCoEF9yJrVkcjISBwcsia9kRAizUs07OLi4rC0tKRs2bLJeq5OpC2KojB1o+7Rj/KFXbCyNNMv89uxDZWvDyb587LKVhd03Yt1wsbcxii1phWXL19k0KC+uLi4sHr1BnLkyGnskoQQKSTRsPvyyy/59ddfEzxnJ9KHaZsu8zpA9xxdnxbv+rL02fgjYX//hWWRwswtHwJAjixuVMiRuf+guXz5IgMH9iZnzlysWrUeV1cZfV2IjCTRa3ZynS79Co+K4+mbMACWfFtT30uK39bNuqDLkZPbbSvo1x9XaahR6kxL3N1zU6lSFdau3SRBJ0QGlOiRXWxsLHfu3Ek09IoXl57v06KoGDXfLjkF6B4zsLPRXYOLuHaVkGN/ApBn/CQOPdgGwPxa32Nq8sF9C2QYd+7cokiRz8iRIydLlvxg7HKEEKkk0bB7+fIlgwcPTrSPyz///DNVCxMfZ9CivwCwtjSjceU8+vn+O3ThlmfiFEytrbkZcJeslvZYm1sbpc604M8/jzBq1DAGDPCiR48+xi5HCJGKEg27ggULsmfPHgOWIj7VQ+8Q/evlQ2vpHyFQh4Wh8vfDukBBTN1zMvXcPABsLWyNUWaacPjwQcaOHUHx4iVp376TscsRQqQy6eQvAzly8SWg6/vyn8/K+azT9QLi3LY9Q0++e2SkX6luBq0vrThwYC8TJ46hbNlyLFnyA1mypMzoGUKItCvRsKtQoUJii0Qa9Dogksv3/QHIn9NBP99vxzaibt/CPJsTK8OO6ucvrjMDc9PM97dOUFAg06dPoUKFSixevAIbm8x7dCtEZpLot508NJ6+jF97HoBWNfLpj+pCT58i5OhhABwG9+Ppo7WA7qaUzBh0AE5OzqxevZ5ChYpgbZ15r1cKkdlk3tvwMpDnPu9GJmhRIx8AYefO4LteF2755y1k0tuga+hZL1PelLJlyyZ++eVnAEqWLC1BJ0QmI2GXzmkVhSkbdKOPd25QGABFrcZnre46neekqYRavbujtkUmHM1gw4Z1zJ07g7NnT8vzo0JkUhJ26ZiiKPSbd0I//Xl5XYfF/rt2AGBXvgIWud2Z+HbonrYFmxm8RmNbs2YlixbNpVGjJsycOS/Td3ItRGaVOS/cZABqjZY+c0/op1cNrw1AzIvnhPxxFBMrK3L26c+uh/v169TLU+vfu8nQVqxYwurVK2jWrCVTpszAzMws6Y2EEBmShF06FBYZx5Clp/TTy4bUxMLcDFVgAC++nwSA58QpmJiZccL7NADzan1vlFqNydramtat2zF+/BQJOiEyOQm7dGjXiXdj1K0dWRdTUxMirl7h9fIlAGQpVRq1U1aGHhsJQCHH/NhkkptSFEXh9etXuLvnpkePPiiKIqcuhRByzS49OnXzDQDrRumCThsbqw8696HDyTn4W0b8PUm/fo8SnY1Sp6FptVpmzpzKl1+25s2b1wASdEIIQMIu3dl54hEAuV3s9F/kr5fpgi5bw0YEeGRl8PHR+vWX1Z2Ng6W94Qs1MK1Wy9Spk/j55620bdtBxqITQsQjYZeOqDVafj+nG3m8V7PPAPDfuYOou7exzJGT7O2+ZMOdbfr1l9SZmSmObDQaDRMnjuHXX3fSu3d/hgwZnik+txAi+eSaXTqhKIr+7ssqxdzI42ZPrPdLgg//DuhGMzAxMcEvKgBrMyvm155qxGoNa/v2LRw4sJcBA7zo02eAscsRQqRBEnbpxM5/3JTSq1kxFEXh+eQJwNtheywt2flgLwAFHfMZpUZj6dChI66urjRokPkemBdCJI+cxkwnDp3Xnb5cPrQWpqYmBO7ZDYBjvfpY5/FEpVXrHzPoULi10eo0lLi4OObNm0lwcDAWFpYSdEKI/yRhlw54+0UA4Olmj42VOVEP7hP0237Ms2fHpeNXAOx+eACAYs5FcLbJZrRaDSEmJoahQwfy008bOX/+jLHLEUKkA3IaM42LilEz8ccLAHSqXwhFo+HN6pUA5OjaAxNTU6763eSvV7ov/X4luxmrVIOIjo5iyJCBXLhwjkmTptGoUVNjlySESAck7NI43+AoAErkc6KwhyPBfx5FExKCa+cu2H5WjCt+N1h36ycAquWsiJlpxu0pJDIyAi+vfly9eoWpU2fRrFlLY5ckhEgnJOzSuD8ueQPQuIonAAG/7ATAoVp1XoR564Oucd76NMv/hXGKNJDo6GiCg4OZOXMeDRs2MXY5Qoh0RMIuDYuOVXP2tg8ARTwciX35EiUuDrvyFdBamDH7tO5h8o5F2lDTvYoxS01V4eHh2NjYkD27Czt27MHCwsLYJQkh0hm5QSUNG7jwLwAqfeaquwPzgO7RAudmLfn2xFgA3GxdMnTQBQcH07t3V6ZN03V/JkEnhPgYEnZplF9ItP513xbFiXvzmojLlwD4JfK8ftn4yt8ZvDZDCQoKpE+frjx58kgeLRBCfBIJuzRq3YE7AAxpXxoTExN8NvwIgLlXb06/1t2dObnKKExNMuY/ob+/H716deHlyxcsXbqK6tVrGrskIUQ6Jtfs0qDbT4N46B2KiQmUzO9E1N07xDx+RJay5ZgRoDuVWSd3dVxsnY1caerQarV4efXjzZs3LFu2mgoVKhm7JCFEOidhl8ZotQrzd1wDoHfzYmjCwvCePweAp3U/Ax9vXGycaV844952b2pqyvDhYzAzM6NMmXLGLkcIkQFI2KUhiqLQa85xQDeET6WCTjwa0AeAV3VLssvnKADflR9otBpTk7f3Sy5dukCrVm0pX76iscsRQmQgEnZpyNGLL/WvJ3arQOCuHQBYN/icXS43Afjcoxb2lnZGqS81PX/+lN69u6FSxVG37udkzepo7JKEEBlIxry7IR26/yKY7cd0A7NO710ZU1UswUcPg5kZfxTX/TPZmNvQplAzY5aZKh4/fkTPnl1QqVSsXr1Bgk4IkeIk7NKA648CmL31KgCdGxQmp3MW/LbrBmG1atKAy37XMTUxZXaNicYsM1U8fHifXr26ALB27SYKFSpi5IqEEBmRnMZMAxbvugFA1eJufF4+N9EPHxJ2SvdA+Rz7KwAUyVYwQ/Z7efXqFSwtLVm9ej2enplrHD4hhOFI2BnZ7adB+te9mxdHHRrKy9nTAXjZvSHE6o74BpXpZZT6UktsbCxWVlZ06NCJxo2bYW9vb+yShBAZmJzGNLK/b7wGYGSnsgC8nDsTgGxNmrL7bdB9X3WMcYpLJVevXqFZswbcvHkdQIJOCJHqJOyM6IVvOBfu+gFQJI8jsa+8Ufn4YJY1K9vzhQBgYWqRoQZjvXTpAgMG9MLW1hZXVzdjlyOEyCQk7Ixo76mnAPRs+hkoCi9mTANA3bsT94IfAjCzxnij1ZfSzp07w6BBfciZMxdr127CzS2HsUsSQmQSEnZGEqvScPVhAADVS+Yk4tpVlNgYrDzycFbzDIDvyg/AxtzGiFWmnLt3b+Pl1Q8PjzysXbsRFxdXY5ckhMhEJOyMZPqmywBUKOICgN/mDQA4DR7ERV/dHZj5HDyNUltqKFSoCF9/3Y01azbi5JQx+/QUQqRdEnZG8MwnDG//CAD6typBxPVraMLDMbGwYMH9DQDksXfHxMTEiFWmjL/+OoG/vx/m5uZ4eQ3D0THjXH8UQqQfEnZG8P0G3bh0/VuVwMTERNdTCmA5+lv8onWnNkdW8DJafSnl998PMHToQJYtW2TsUoQQmZyEnYFdexSgf12xqCtxvj5E37uLdf78zHm8CYBeJb5J90d1+/b9yrhxIylTphyjRo0zdjlCiExOws7Afj/3HND1fwngu3kjAOENqurXKeta0vCFpaDdu3cyadJYKlaswrJlq7G1zWLskoQQmZyEnYE99A4FIKdzFiJuXCf63l3MsjmxIlw3fE+Xz740ZnmfLC4uji1bNlGtWk2WLFmJjU3GuJtUCJG+SXdhBvTQOwQAzxy6HkNCj/8JwLEOn0H4Q/I55KFyzvLGKu+TabVaLC0tWbNmI3Z2dlhaWhq7JCGEAOTIzqAW/KzrHqt1zXxE3rlN5M0b2FWtyrVw3QPkA0r3NGZ5n2T9+jWMHDkEtVqNk5OTBJ0QIk2RsDOQ6Fg1sXEaAErmd8bnxzVgYsL+chYAFHLMj61F+jzlt3r1ChYvno+5uZwoEEKkTfLtZCArftWNNN66Zj6iH9xHExJCaIUi3Ai+B0C/Ut2NWd5HURSF5csXs3btDzRr1pIpU2ZgZpbxhiESQqR/cmRnAL7BUdx+FgxAo8qe+G7ewNUiNmworJvXPH8jrM2tjFniR1m5cilr1/5A69bt+f77mRJ0Qog0S47sUplPUBRjV58DoHHlPIQfOcjrmAD+qucEwDefdaBKzgrGLPGjVa9ek5iYGIYMGY6pqfzdJIRIu+QbKpXN2KzrA7Nq8Ry0KufC48O72dpEF3RN8zVId0Gn1Wo5e/Y0AKVLl2XYsJESdEKINE++pVLR7WdBRESrsDQ3pXfzYoSfOcXmZrpOkN3tctIkXwMjV/hhNBoN338/gf79e3L16hVjlyOEEMkmYZeKNv6uu/lkfBfd0dv+BwfRmJlgiiljKw01ZmkfTK1WM3HiGPbs+YU+fQZQpkxZY5ckhBDJJtfsUolWqxAQGgNAblc7Hj66zIWSum6zxlceZszSPphKpWL8+FEcPnyQgQOH0Lt3P2OXJIQQHyRVj+z2799PkyZNaNCgAVu2bEmw/I8//qBly5a0aNGCAQMGEBoamprlGNQzn3AAKn3miqJWs+jFDgDaOFTFLUv6Grj04sVzHD58kKFDR0jQCSHSpVQLO19fXxYuXMjWrVvZu3cvO3bs4NGjR/rlERERTJ48mdWrV7Nv3z6KFCnC0qVLU6scg/vpyH0A6pRxZ94fM/Tz65VvZaSKPl61ajX5+ee9dO2afnt4EUJkbqkWdmfOnKFKlSo4Ojpia2tLw4YNOXTokH65SqVi8uTJuLm5AVCkSBHevHmTWuUYVGSMSn9kF33rMM8sdQO1zqk6Id0M3RMTE0OvXr24ePE8AIULFzFyRUII8fFSLez8/PxwcXHRT7u6uuLr66ufzpYtG/Xr1wd0X6yrV6/WT6d31x7qxqzrUCc/p3x0t+n3LtCeLDb2xiwr2aKjo/Dy6sehQ4d4/fqVscsRQohPlmo3qCiKkmDe+45qwsPDGTBgAEWLFqV169Yf9B7OznYfXd8/ubikbAhduHcDgDI+V/ktjzUAn5evg6lJ2r/5NSIigr59+3Pp0kUWL15M27ZtjV1SmpXSPzcZibRN4qRtEpeabZNqYefm5salS5f0035+fri6xr8xw8/Pj549e1KlShXGjh37we8RGBiBVpswVD+Ei4s9/v7hn7SPf4qOVXPzse7I7u79U2grmuFm60JgQGSKvUdqiYqKpF+/nty+fZOZM+fRtm3bFG2bjCSlf24yEmmbxEnbJC4l2sbU1CTRg6BUO9SoVq0aZ8+eJSgoiOjoaI4cOUKtWrX0yzUaDf369aNx48aMGzcu3VzLSsrinbphfErnsWdXRV1fkR2LfNgRq7FYWVmTP38B5s5dRMOGTYxdjhBCpJhUPbIbOnQoXbp0QaVS0a5dO0qVKkXv3r3x8vLCx8eHO3fuoNFoOHz4MAAlSpRg+vTpqVVSqvP2i+DB25HIW8VcYQ5gY2JJ4WwFjVtYEoKDg4mLi8XNLQeTJ6ff9hdCiMSk6kPlzZs3p3nz5vHmrVmzBoCSJUty79691Hx7g/v9/HMAWlXJzaVzP0NeB5oU+MLIVf23wMAA+vbtjomJCdu3/yojFwghMqS0f8dEOvHgZQhnb+vuNq0WdINbBXUDsZZ1LWXMsv6Tn58vvXp1wdvbmxEjxkrQCSEyLAm7FDJri65j5FqlchJ48AB+ThaYmpiSzdrRuIUlwsfnDT17dsHX14cVK9ZQqVIVY5ckhBCpRsIuBVx54A+AlaUZ7XLF8MrVAoCqOSsas6z/NGfODIKDA1m5ch3lyqWvYYaEEOJDSUfQKeD0TV3PLyM6lsFn1hj2N8kKQD2PmsYs6z9NmPA9Pj6v+eyz4sYuRQghUp0c2aWAqw8DsLEyw3rvZjRRUcRZ6po1Rxrr8PnZsydMmTIelSqObNmySdAJITINCbtP9PiV7lEDByszIi5fYlvLHABUz1XJmGUl8OjRQ3r27MLJk8fx8fExdjlCCGFQEnaf6Pxd3R2Yzd78RZy5Cf5ZtAC0L9TSmGXFc//+PXr37oKpqQnr1m3GwyOPsUsSQgiDkrD7BIqi8MclbwAsQx6zsoOu4+t6HjWxMLMwZml6d+7confvrlhaWrF27Wby5ctv7JKEEMLgJOw+klZR6Dn7OACmipatTbMBYGFqTpuCzYxZWgI5c+bixx9/wtMzr7FLEUIIo5Cw+0h/X3+tf9087hf9TSmL6sxIE/18+vjo7hAtVqwE27b9grt7biNXJIQQxiNh9xHUGi0bD+lGIu/pcJ8jNXXDUgwp28+YZeldvHieVq2asGvXdgBMTeWfWQiRucm34Ec4eU13VJcvm8LWok8ByGppT6Fsxr8edvbsaQYP7kuuXO7UqfO5scsRQog0QcLuI5y5pTtFaG95WD9vevXxxipH7++/T/Ltt/3Jkycva9duInt2l6Q3EkKITEDC7gOdufWGp2/CsTCDh566eUvqzDT6dTo/P1+GD/eiYMFCrFmzAScnJ6PWI4QQaYl0F/aB9vytO21ZKOcpngK5rV0xMzX+aAGurm7MnDmPChUq4+DgYOxyhBAiTZEjuw+g1SoEhMbgYKXhaa4IAPqX723Umn7//QBnzpwCoF69BhJ0QgjxHhJ2H+Du82AAHC3uAFAnewUcrbIarZ59+35l7NgRbN26GUVRjFaHEEKkdRJ2H2DXiccABBbV9ZpSJV91o9Xyyy8/M2nSWCpXrsrcuYuMfs1QCCHSMrlm9wGe+4ZjYROE1tQEayzwsHc3Sh3bt29h1qyp1KhRm/nzl2BlZWWUOoQQIr2QI7tkuvYoAADLIpcB+KZER6PUoSgKDx7co06dz1mwYKkEnRBCJIMc2SXT1qMPMMkSitZSA0BplxIGryE8PBx7e3vGj5+CRqPBwiJtdDYthBBpnRzZJcNzn3ACQmOwyP0AgAGlexr0GpmiKKxatZwOHVoSGBiAqampBJ0QQnwACbtkWH/wLgBmWQMBKJKtgMHeW1EUli1bxMqVS6lYsTKOjtkM9t5CCJFRyGnMJIRHxfHCLwLM4wAo51oKc1PDNJuiKCxcOIdNm9bTtm0Hxo2bLJ06CyHER5BvziT89XYonyqqMwAUzVbIYO+9ZcsmNm1aT8eOnRk/fooEnRBCfCQ5skvCscu6Z+rulYkCTClkwFOYLVq0AqBz5y7yHJ0QQnwCOVT4D77BUQRHxGFlFUisla6pXG2zp+p7ajQaNm/eQGxsLA4OWfn6664SdEII8YnkyO4/jFl1DoCsHtcIB3qW+DpV30+tVjNx4hgOHtyPk5MzTZs2T9X3E0KIzEKO7BIRHB4LgJmiwd5O9zdBOddSqfZ+KpWKMWOGc/DgfgYPHipBJ4QQKUjCLhH33nb63NjvLK8toynuXDTV3isuLo6RI4dy9OghvvtuFD179k219xJCiMxITmMm4tCFFwBYZ/EHbDAzSb0x63x83nDt2mVGjx5Px46pe6pUCCEyIwm79/ALjuKln268uos1sgJx1MpdNcXfR6WKw9zcgjx5PNm79xAODsYbLkgIITIyOY35Hv8ft65B8F/4WcZhbWbNZ06FU/Q9oqOjGDCgNytXLgWQoBNCiFQkYfceGw/dB8DEWTfSQV2PlB23LiIiggEDenP58kU8PfOl6L6FEEIkJKcx/yUiWqV/7VPECYihdu6UC7uwsDAGDuzNnTu3mDVrAV980SjF9i2EEOL95MjuX05cfQVAvaAzPLSPAcDe0i5F9q3RaBg4sDd3795h7tzFEnRCCGEgcmT3L8eu6LoHu1onBDClas6KKbZvMzMzOnbsjL29A7Vq1Umx/QohhPhvEnb/olJrsdHGEGOtO+jtXLTdJ+8zIMCfhw8fULVqdZo2bfHJ+xNCCPFhJOz+JTJGTQnu8Bionbv6J/dL6evrS9++3QgODuK33/7Ezi5lTokKIYRIPgm7f/ALidb9P184AGVcin/S/t68eU2fPt0IDAxg+fI1EnRCCGEkEnb/8Ng7FIBwN13oFc5W8KP39eqVN717dyUsLIwffviRUqXKpESJQgghPoKE3T+8eB2MmauumzB3u5yftK/9+/cQERHBqlU/Urx4yZQoTwghxEeSsPuHGB8fLPPeAaBbsU4ftQ9FUTAxMaFv34G0aNGaXLncU7JEIYQQH0Ges/uHC8GBAOSwcSGXXY4P3v7Rowd89VVbXr58gYmJiQSdEEKkEXJk90+FrwHQqlDTD9703r279OvXHQsLC9RqdQoXJoQQ4lPIkd1bT988BKs4AEpmL/ZB296+fZM+fbphbW3DunU/kS9f/tQoUQghxEeSsHtr953DABQNKftB2927d5e+fbtjZ2fHunWbyZPHMzXKE0II8Qkk7N56rH4DQN385T9oOw8PD2rWrMO6dZtxd8+dGqUJIYT4RBJ2gF9oKCYWKoizpESZ5I1bd/PmDaKjo8iSxY6ZM+eRM2euVK5SCCHEx5KwA346sxeAQn4OyVr/zJlT9Or1DQsWzEnNsoQQQqQQCTvgdaQvAN2b9Epy3b/+OsG33/bH0zMf/ft7pXZpQgghUoCEHRDt5A9A1uxO/7nesWNHGTZsMIUKFWbNmg04Of33+kIIIdKGTB924W/8UBQTzGNs/nO9mJgYZs+eTrFixfjhh/VkzepomAKFEEJ8skz/UPnqLacx8VCwi/rv3k6sra1ZvXo92bO7kCWLjF4ghBDpSaY/sjsbGAlAhVIe712+Z88vLFo0D0VR8PTMJ0EnhBDpUKYOO7VGi8pWN5yPm51zguW7dm1n8uRx3L9/F7VaZejyhBBCpJBMfRrzkXcIFh73AcifNW+8Zdu2bWb27OnUrFmbefOWYGFhaYQKhRBCpIRMHXZ3nvpgaqM7jelm66Kfv3nzeubPn029eg2YPXu+BJ0QQqRzmfo0pk/oawBqa/JiYmKin58zZy6aNGnO7NkLJOiEECIDyNRHdmpNGAAFHfKgKAqPHz+iYMFC1K/fkPr1Gxq5OiGEECklVY/s9u/fT5MmTWjQoAFbtmxJsPzu3bu0bduWhg0bMm7cOIOPAxeC7mHyLI4uLF26gI4dW3Pnzi2D1iCEECL1pVrY+fr6snDhQrZu3crevXvZsWMHjx49irfOiBEjmDBhAocPH0ZRFH7++efUKue9QkxDURSFPb8c4scf19C6dTuKFv2wseyEEEKkfakWdmfOnKFKlSo4Ojpia2tLw4YNOXTokH75q1eviImJoUyZMgC0adMm3nJD0JrG8OSXW2z7eRudOn3D2LGTMDXN1JcxhRAiQ0q1b3Y/Pz9cXN7d4ejq6oqvr2+iy11cXOItNwTf2974nHlK1649GTlybLybVIQQQmQcqXaDiqIoCeb9M0ySWp4czs6f1ptJvZI1KGyWg+nTp0jQJcLFxd7YJaRZ0jaJk7ZJnLRN4lKzbVIt7Nzc3Lh06ZJ+2s/PD1dX13jLAwIC9NP+/v7xlidHYGAEWm3C0Eyu3s264NLdHn//8I/eR0bm4iJtkxhpm8RJ2yRO2iZxKdE2pqYmiR4EpdppzGrVqnH27FmCgoKIjo7myJEj1KpVS7/c3d0dKysrLl++DMCePXviLRdCCCFSSqqFnZubG0OHDqVLly60atWKZs2aUapUKXr37s3NmzcBmDdvHjNnzqRx48ZER0fTpUuX1CpHCCFEJmaivO/iWTrxqacxQU4r/Bdpm8RJ2yRO2iZx0jaJS7enMYUQQoi0QsJOCCFEhidhJ4QQIsOTsBNCCJHhSdgJIYTI8CTshBBCZHgSdkIIITI8CTshhBAZnoSdEEKIDE/CTgghRIYnYSeEECLDS7UhfgzB1DRlxqBLqf1kRNI2iZO2SZy0TeKkbRL3qW3zX9un646ghRBCiOSQ05hCCCEyPAk7IYQQGZ6EnRBCiAxPwk4IIUSGJ2EnhBAiw5OwE0IIkeFJ2AkhhMjwJOyEEEJkeBJ2QgghMrxME3b79++nSZMmNGjQgC1btiRYfvfuXdq2bUvDhg0ZN24carXaCFUaR1Jt88cff9CyZUtatGjBgAEDCA0NNUKVxpFU2/zfiRMnqFevngErM76k2ubJkyd88803tGjRgp49e8rPzT/cvn2btm3b0qJFC/r27UtYWJgRqjSeiIgImjVrhre3d4JlqfZdrGQCPj4+St26dZXg4GAlMjJSad68ufLw4cN46zRt2lS5evWqoiiKMmbMGGXLli1GqNTwkmqb8PBwpXr16oqPj4+iKIqyaNEiZerUqcYq16CS83OjKIri7++vNGrUSKlbt64RqjSOpNpGq9UqX3zxhXLy5ElFURRl7ty5ypw5c4xVrkEl5+emU6dOyokTJxRFUZSZM2cqCxYsMEapRnHt2jWlWbNmSvHixZWXL18mWJ5a38WZ4sjuzJkzVKlSBUdHR2xtbWnYsCGHDh3SL3/16hUxMTGUKVMGgDZt2sRbnpEl1TYqlYrJkyfj5uYGQJEiRXjz5o2xyjWopNrm/8aPH8+gQYOMUKHxJNU2t2/fxtbWllq1agHQr18/OnfubKxyDSo5PzdarZbIyEgAoqOjsba2NkapRvHzzz8zadIkXF1dEyxLze/iTBF2fn5+uLi46KddXV3x9fVNdLmLi0u85RlZUm2TLVs26tevD0BMTAyrV6/WT2d0SbUNwKZNmyhWrBilS5c2dHlGlVTbvHjxguzZszNq1CiaN2/OpEmTsLW1NUapBpecn5vRo0czbtw4atSowZkzZ+jYsaOhyzSa6dOnU6FChfcuS83v4kwRdsp7BnYwMTFJ9vKMLLmfPTw8nN69e1O0aFFat25tiNKMLqm2efDgAUeOHGHAgAGGLCtNSKpt1Go1Fy5c4Ouvv2b//v14eHgwa9YsQ5ZoNEm1TUxMDOPGjWPjxo2cOnWKr776ilGjRhmyxDQrNb+LM0XYubm5ERAQoJ/28/OLdwj97+X+/v7vPcTOiJJqm//P++qrryhatCjTp083dIlGk1TbHDp0CH9/f9q2bUufPn307ZQZJNU2Li4ueHp6UrJkSQCaNWvGjRs3DF6nMSTVNg8ePMDKyopSpUoB8OWXX3LhwgWD15kWpeZ3caYIu2rVqnH27FmCgoKIjo7myJEj+msJAO7u7lhZWXH58mUA9uzZE295RpZU22g0Gvr160fjxo0ZN25cpjnihaTbxsvLi8OHD7N3715Wr16Nq6srW7duNWLFhpNU25QtW5agoCDu3bsHwLFjxyhevLixyjWopNrG09MTHx8fnjx5AsCff/6p/6Mgs0vV7+IUuc0lHdi3b5/StGlT5YsvvlBWr16tKIqi9OrVS7lx44aiKIpy9+5dpW3btkqjRo2UYcOGKbGxscYs16D+q22OHDmiFClSRGnRooX+v7Fjxxq5YsNJ6ufm/16+fJmp7sZUlKTb5tq1a0rbtm2VJk2aKD169FACAgKMWa5BJdU2J06cUJo3b640a9ZM6dq1q/LixQtjlmsUdevW1d+NaYjvYhmpXAghRIaXKU5jCiGEyNwk7IQQQmR4EnZCCCEyPAk7IYQQGZ6EnRBCiAzP3NgFCGEsRYoUoXDhwpiavvubr0SJEv/54Pzu3bs5fPgwq1at+uT3X7p0KVu2bMHNzQ0TExM0Gg3Ozs5MmjSJfPnyffD+fH19+fbbb9m+fTsvX75kzpw5LF26NN78T+Xt7U2DBg0oXLiwfl5UVBQ5cuRgxowZeHh4/Of2y5Yto2jRopmmyzmRdkjYiUxt48aNODk5Ge39mzRpwsSJE/XTmzdv5rvvvmP37t0fvC83Nzd9oL1+/ZqnT58mmJ8SrK2t2bt3r35aURSmTZvGwoULWbBgwX9ue/78eQoWLJhitQiRXHIaU4j32LVrF+3bt6dVq1bUrVv3vT2jHDlyhNatW9OmTRvat2/PxYsXAV0/oqNHj6ZNmzY0b96cGTNmJHtMrqpVq+pDysfHh379+tG8eXOaNWvG2rVrAV2/k5MmTaJ58+a0adMGLy8vIiMj8fb2pmzZsmg0GsaPH8+LFy/o2bOnfr5Wq6V27drcvHlT/35Dhw7Vf7aVK1fSunVrWrZsyYABA5LdAW9sbCz+/v5kzZoVgKdPn9K9e3e+/PJL6tatS//+/YmNjWXLli3cunWLOXPmcPToUeLi4pgxYwatW7emRYsWjB49moiIiGS9pxAfSsJOZGpdu3alZcuW+v8CAwOJjIxk586drF69mj179rBw4ULmzp2bYNs5c+YwadIkdu/ezbfffsv58+cBmDFjBsWLF2f37t3s2bOH4OBg1q9fn2QtarWaXbt2UblyZQCGDx9O5cqV2b9/P9u2bWPfvn389ttvXLt2jQsXLrBv3z52796Nh4cH9+/f1+/HzMyMadOmkSdPHtatW6efb2pqStu2bfn1118BCA0N5cyZMzRv3pw9e/bw4MEDdu7cyd69e6lduzbjx49/b50xMTG0bNmS5s2bU61aNVq3bk2+fPkYPnw4oBvCpVWrVuzYsYMjR47g7e3NiRMn6Ny5MyVKlGDkyJE0aNCA1atXY2Zmxu7du9m3bx+urq7Mmzcvmf9yQnwYOY0pMrXETmP+8MMPnDx5kmfPnnHv3j2ioqISrNO0aVMGDRpE7dq1qV69Or179wZ0o5bfvHmTXbt2AbpwSMzBgwf1/QCqVCqKFy/O1KlTiYqK4sqVK/z4448A2Nvb06ZNG/766y/GjRuHmZkZ7du3p0aNGjRs2JBSpUq9d9Tnf2vbti3t2rVj9OjRHDhwgLp162Jvb8/x48e5efMmbdu2BXTjrUVHR793H/88jfn3338zYsQIqlevTpYsWQAYMWIEp0+fZs2aNTx79gw/P7/3tt+JEycIDw/nzJkz+s/v7Oyc5GcQ4mNI2AnxLz4+Pnz55Zd06NCB8uXL06hRI44fP55gvaFDh9KuXTtOnTrF7t27Wb16Nbt370ar1bJ48WIKFCgAQFhYWKIdaP/7mt3/RUREJBjuRKvVolarcXBwYO/evVy5coVz584xZMgQunTpkqybPtzd3SlWrBgnTpxg9+7djB07Vr/vXr166UdtiIuLIzQ0NMn91axZk+7duzNs2DB+//137O3tGTZsGBqNhsaNG1OnTh3evHnz3qFbtFotY8eOpXbt2gBERkYSGxub5HsK8THkNKYQ/3Lr1i2cnJwYMGAANWvW1AedRqPRr6NWq6lXrx5RUVF06tSJSZMm8fjxY9RqNTVq1GDDhg0oikJcXBz9+/fnp59++qAa7OzsKF26NFu2bAF01wH37NlDtWrVOH78ON26daNs2bIMHjyYVq1a6UcX+D8zMzNUKtV7992hQwfWrFlDTEwM5cuXB6BGjRrs2rVLf81s8eLFjBw5Mlm19ujRAwcHB5YsWQLAqVOnGDhwIE2aNMHExITr16/r287MzEx//bJGjRps2bKFuLg4tFotEyZMSPIGFyE+lhzZCfEv1atXZ9euXf9r7w5RFggCAAq/xVVYy4LNAxgsNotNk0GD4HqCTZoEgygGFYUFwZt4CG+izSu4KH8TbP51eV+dGRimPBgGhn6/TxRFtFotarUat9vtMycMQ1arFYvFgjAMCYKA4/FIpVJhvV5zOBwYDofkeU6n0yFN03/v43Q6sdvtuFwuPJ/Pz4OU9/vN9XplMBhQrVaJ45j9fv+1ttFoUCqVGI/HnM/nr7Fer8d2u/1cuwIkScLj8WAymRAEAfV6/efPVsvlMpvNhjRNSZKE+XzObDYjjmOiKKLdbnO/3wHodrtkWUae50ynU7IsYzQa8Xq9aDabLJfLf5+T9At/PZAkFZ7XmJKkwjN2kqTCM3aSpMIzdpKkwjN2kqTCM3aSpMIzdpKkwjN2kqTC+wP50fgTROQNGAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 504x504 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_curve \n",
    "\n",
    "fpr_train, tpr_train, thresholds_train = roc_curve(y_train, y_train_pred)\n",
    "auc_train = roc_auc_score(y_train, y_train_pred)\n",
    "\n",
    "fpr_valid, tpr_valid, thresholds_valid = roc_curve(y_val, y_val_pred)\n",
    "auc_valid = roc_auc_score(y_val, y_val_pred)\n",
    "\n",
    "fpr_test, tpr_test, thresholds_test = roc_curve(y_test, y_test_pred)\n",
    "auc_test = roc_auc_score(y_test, y_test_pred)\n",
    "\n",
    "plt.figure(figsize=(7,7))\n",
    "plt.plot(fpr_train, tpr_train, 'r', label = ('Train AUC: ' + str(round(auc_train, 2))))\n",
    "plt.plot(fpr_valid, tpr_valid, 'b',label = ('Valid AUC: ' + str(round(auc_valid, 2))))\n",
    "plt.plot(fpr_test, tpr_test, 'g',label =('Test AUC: ' +  str(round(auc_test, 2))))\n",
    "plt.plot([0,1],[0,1],'k--')\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "present-kennedy",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "coated-prayer",
   "metadata": {},
   "source": [
    "The objective of this modeling task was to find a model that could predict the probability that a patient would be readmitted into the hospital after 30 days. Once again, it is important to note that guessing 'not readmitted' for every sample would result in a 89% accuracy rate, yet the recall would be 0%, since we would never predict a readmission. Guessing randomly would result in a 50% accuracy rate. Our best model, the SGD Logistic Regression classifer, had a 67% test accuracy on held-out data and AUC score of 0.66. There is room for improvement by performing more detailed data research, tuning the hyperparameters of the models, and using new models such as multi-layer perceptron binary classifiers."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}