kaggle_starter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# KEN3450, Data Analysis 2020 \n",
    "\n",
    "**Kaggle Competition 2020**<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import scipy as sp\n",
    "from sklearn import preprocessing\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "#import your classifiers here\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "# %matplotlib notebook\n",
    "\n",
    "pd.set_option('display.max_columns', None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Diagnosing the Maastricht Flu \n",
    "\n",
    "You are given the early data for an outbreak of a dangerous virus originating from a group of primates being kept in a Maastricht biomedical research lab in the basement of Henri-Paul Spaaklaan building, this virus is dubbed the \"Maastricht Flu\".\n",
    "\n",
    "You have the medical records of $n$ number of patients in `flu_train.csv`. There are two general types of patients in the data, flu patients and healthy (this is recorded in the column labeled `flu`, a 0 indicates the absences of the virus and a 1 indicates presence). Notice that the dataset is unbalanced and you can expect a similar imbalance in the testing set.\n",
    "\n",
    "**Your task:** build a model to predict if a given patient has the flu. Your goal is to catch as many flu patients as possible without misdiagnosing too many healthy patients.\n",
    "\n",
    "**The deliverable:** submit your final solution via Kaggle competition using the `flu_test.csv` data.\n",
    "\n",
    "Maastricht Gemeente will use your model to diagnose sets of future patients (held by us). You can expect that there will be an increase in the number of flu patients in any groups of patients in the future.\n",
    "\n",
    "Here are some benchmarks for comparison and for expectation management. Notice that because the dataset is unbalanced, we expect that there is going to be a large difference in the accuracy for each class, thus `accuracy` is a metric that might be misleading in this case (see also below). That's why the baselines below are based on the expected accuracy **per class** and also they give you an estimate for the AUROC on all patients in the testing data. This is the score you see in the Kaggle submission as well.\n",
    "\n",
    "**Baseline Model:** \n",
    "- ~50% expected accuracy on healthy patients in training data\n",
    "- ~50% expected accuracy on flu patients in training data\n",
    "- ~50% expected accuracy on healthy patients in testing data (future data, no info on the labels)\n",
    "- ~50% expected accuracy on flu patients in testing data (future data, no info on the labels)\n",
    "- ~50% expected AUROC on all patients in testing data (future data, no info on the labels)\n",
    "\n",
    "**Reasonable Model:** \n",
    "- ~70% expected accuracy on healthy patients in training data\n",
    "- ~55% expected accuracy on flu patients, in training data\n",
    "- ~70% expected accuracy on healthy patients in testing data (future data, no info on the labels, to be checked upon your submission)\n",
    "- ~57% expected accuracy on flu patients, in testing data (future data, no info on the labels, to be checked upon your submission)\n",
    "- ~65% expected AUROC on all patients, in testing data (future data, no info on the labels, to be checked from Kaggle)\n",
    "\n",
    "**Grading:**\n",
    "Your grade will be based on:\n",
    "1. your model's ability to out-perform the benchmarks (they are kind of low, so we won't care much about this)\n",
    "2. your ability to carefully and thoroughly follow the data analysis pipeline\n",
    "3. the extend to which all choices are reasonable and defensible by methods you have learned in this class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Read the data, clean and explore the data\n",
    "\n",
    "There are a large number of missing values in the data. Nearly all predictors have some degree of missingness. Not all missingness are alike: NaN in the `'pregnancy'` column is meaningful and informative, as patients with NaN's in the pregnancy column are males, where as NaN's in other predictors may appear randomly. \n",
    "\n",
    "\n",
    "**What do you do?:** We make no attempt to interpret the predictors and we make no attempt to model the missing values in the data in any meaningful way. We replace all missing values with 0.\n",
    "\n",
    "However, it would be more complete to look at the data and allow the data to inform your decision on how to address missingness. For columns where NaN values are informative, you might want to treat NaN as a distinct value; You might want to drop predictors with too many missing values and impute the ones with few missing values using a model. There are many acceptable strategies here, as long as the appropriateness of the method in the context of the task and the data is discussed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Train\n",
    "df = pd.read_csv('data/flu_train.csv')\n",
    "df = df[~np.isnan(df['flu'])]\n",
    "df.head()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Test\n",
    "df_test = pd.read_csv('data/flu_test.csv')\n",
    "df_test.head()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#What's up in each set\n",
    "\n",
    "x = df.values[:, :-1]\n",
    "y = df.values[:, -1]\n",
    "\n",
    "x_test = df_test.values[:, :-1]\n",
    "\n",
    "print('x train shape:', x.shape)\n",
    "print('x test shape:', x_test.shape)\n",
    "print('train class 0: {}, train class 1: {}'.format(len(y[y==0]), len(y[y==1])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Lets count the amount of Nan\n",
    "nan_count = df.isna().sum()\n",
    "nan_percentage = [count / x.shape[0] for count in nan_count]\n",
    "columns = df.columns\n",
    "\n",
    "threshold = 0.6\n",
    "\n",
    "# And let's plot those\n",
    "fig1 = plt.figure(figsize=(30,15))\n",
    "ax = fig1.add_subplot(111)\n",
    "plt.axhline(y=threshold,linewidth=1, color=\"k\")\n",
    "ax.bar(columns, nan_percentage)\n",
    "plt.xticks(rotation=90)\n",
    "ax.set_xlabel('Column')\n",
    "ax.set_ylabel('NaN Count')\n",
    "ax.set_title('Columns with missing values')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that there are many columns that have missing values. Of course it is not wise to just blatantly drop every observation that has a NaN value somewhere, however some columns have so many missing values, some over 75% of the data! For now we will drop these columns because they won't be able to indicate sickness if they don't have information about the majority of the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "too_much_cols = []\n",
    "for i in range(len(columns)):\n",
    "    if nan_percentage[i] >= threshold:\n",
    "        too_much_cols.append(columns[i])\n",
    "df_cut = df.drop(too_much_cols, axis=1)\n",
    "print(\"So we went from {} features to {} features, based on too much NaN\".format(df.shape[1], df_cut.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do another count\n",
    "nan_count = df_cut.isna().sum()\n",
    "nan_percentage = [count / x.shape[0] for count in nan_count]\n",
    "columns = df_cut.columns\n",
    "\n",
    "# And let's plot those\n",
    "fig1 = plt.figure(figsize=(30,15))\n",
    "ax = fig1.add_subplot(111)\n",
    "plt.axhline(y=threshold,linewidth=1, color=\"k\")\n",
    "ax.bar(columns, nan_percentage)\n",
    "plt.xticks(rotation=90)\n",
    "ax.set_xlabel('Column')\n",
    "ax.set_ylabel('NaN Count')\n",
    "ax.set_title('Columns with missing values')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's have a look at the distribution of the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sick_df = df_cut[(df['flu'] == 1)]\n",
    "okay_df = df_cut[(df['flu'] == 0)]\n",
    "\n",
    "plt.bar(['Sick', 'Healthy'], [sick_df.shape[0], okay_df.shape[0]])\n",
    "plt.show()\n",
    "print(\"So we have {}% sick people and {}% healthy people\".format(np.round((sick_df.shape[0]/df_cut.shape[0])*100, \n",
    "                                                                          decimals=2), \n",
    "                                                                 np.round((okay_df.shape[0]/df_cut.shape[0])*100, \n",
    "                                                                          decimals=2)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cut.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets see what values our columns have so we can try to quantify those for our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for column in df_cut.select_dtypes(include='object').columns:\n",
    "    print(\"{} has values:\\t\\t\\t{}\\n\".format(column, df_cut[column].unique()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nothing weird to see here, columns with simple 'male'/'female' or 'yes'/'no' can easily be changed to numerical values. Some more categorical columns need some more thought on how to feed those to our model\n",
    "\n",
    "The other more varied text values can be added as dummy columns. Whether they improve the model/ have any relation with sickness needs to be tested.\n",
    "\n",
    "Gender has values:\t\t\t['male' 'female']\n",
    "            Might be relevant, or could cause noise-> Test\n",
    "            \n",
    "Race1 has values:\t\t\t['White' 'Mexican' 'Black' 'Other' 'Hispanic']\n",
    "            Might be relevant -> Test\n",
    "\n",
    "Education has values:\t\t\t['High School' 'Some College' nan 'College Grad' '9 - 11th Grade'\n",
    " '8th Grade']\n",
    "            Most probably relevant -> implement\n",
    "\n",
    "MaritalStatus has values:\t\t\t['Married' 'LivePartner' nan 'Divorced' 'NeverMarried' 'Separated'\n",
    " 'Widowed']\n",
    "            Might be relevant -> Test\n",
    "\n",
    "HHIncome has values:\t\t\t['25000-34999' '35000-44999' '75000-99999' '55000-64999' 'more 99999'\n",
    " '65000-74999' nan '15000-19999' '10000-14999' ' 0-4999' '45000-54999'\n",
    " ' 5000-9999' '20000-24999']\n",
    "            Most probably relevant -> implement\n",
    "\n",
    "HomeOwn has values:\t\t\t['Own' 'Rent' 'Other' nan]\n",
    "            Might be relevant -> Test\n",
    "\n",
    "Work has values:\t\t\t['NotWorking' nan 'Working' 'Looking']\n",
    "            Might be relevant -> Test\n",
    "\n",
    "BMI_WHO has values:\t\t\t['30.0_plus' '12.0_18.5' '18.5_to_24.9' '25.0_to_29.9' nan]\n",
    "            Most probably relevant -> implement\n",
    "        \n",
    "Diabetes has values:\t\t\t['No' 'Yes' nan]\n",
    "            Most probably relevant -> implement\n",
    "\n",
    "HealthGen has values:\t\t\t['Good' nan 'Vgood' 'Fair' 'Excellent' 'Poor']\n",
    "            Most probably relevant -> implement\n",
    "\n",
    "LittleInterest has values:\t\t\t['Most' 'Several' nan 'None']\n",
    "\n",
    "Depressed has values:\t\t\t['Several' nan 'None' 'Most']\n",
    "            Most probably relevant -> implement\n",
    "\n",
    "SleepTrouble has values:\t\t\t['Yes' nan 'No']\n",
    "            Most probably relevant -> implement\n",
    "\n",
    "PhysActive has values:\t\t\t['No' nan 'Yes']\n",
    "            Most probably relevant -> implement\n",
    "\n",
    "TVHrsDay has values:\t\t\t[nan 'More_4_hr' '4_hr' '2_hr' '1_hr' '3_hr' '0_to_1_hr' '0_hrs']\n",
    "            Might be relevant -> Test\n",
    "\n",
    "CompHrsDay has values:\t\t\t[nan 'More_4_hr' '0_hrs' '1_hr' '3_hr' '2_hr' '0_to_1_hr' '4_hr']\n",
    "            Might be relevant -> Test\n",
    "\n",
    "Alcohol12PlusYr has values:\t\t\t['Yes' nan 'No']\n",
    "            Most probably relevant -> implement\n",
    "\n",
    "Smoke100 has values:\t\t\t['Yes' nan 'No']\n",
    "            Most probably relevant -> implement\n",
    "\n",
    "Smoke100n has values:\t\t\t['Smoker' nan 'Non-Smoker']\n",
    "            Most probably relevant -> implement\n",
    "            Two above can be merged if the same\n",
    "\n",
    "Marijuana has values:\t\t\t['Yes' nan 'No']\n",
    "            Most probably relevant -> implement\n",
    "\n",
    "RegularMarij has values:\t\t\t['No' nan 'Yes']\n",
    "            Most probably relevant -> implement\n",
    "\n",
    "HardDrugs has values:\t\t\t['Yes' nan 'No']\n",
    "            Most probably relevant -> implement\n",
    "\n",
    "SexEver has values:\t\t\t['Yes' nan 'No']\n",
    "            Might be relevant -> Test\n",
    "\n",
    "SameSex has values:\t\t\t['No' 'Yes' nan]\n",
    "            Might be relevant -> Test\n",
    "\n",
    "SexOrientation has values:\t\t\t['Heterosexual' nan 'Bisexual' 'Homosexual']\n",
    "            Might be relevant -> Test\n",
    "\n",
    "To be tested:\n",
    "SexOrientation\n",
    "SameSex\n",
    "SexEver\n",
    "CompHrsDay\n",
    "TVHrsDay\n",
    "HomeOwn\n",
    "Work\n",
    "MaritalStatus\n",
    "Race Gender\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Smoke100n and Smoke100 neem to have similar true false values\n",
    "If the counts are the same it is likely they are identical\n",
    "If the amount of NaN values is different, we can merge the columns to complete them"
   ]
  },
  {
<<<<<<< Updated upstream
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df_cut['Smoke100n'].isna().sum())\n",
    "print(df_cut['Smoke100'].isna().sum())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No need to merge, so we will drop Smoke100n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cut = df_cut.drop(columns='Smoke100n')"
   ]
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "Changing all text entries to numeric values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cut['Gender'].replace(['female','male'],[0,1],inplace=True)\n",
    "\n",
    "dummies = ['Race1', \n",
    "           'Education', \n",
    "           'MaritalStatus', \n",
    "           'HomeOwn', \n",
    "           'BMI_WHO', \n",
    "           'HealthGen', \n",
    "           'LittleInterest', \n",
    "           'Depressed', \n",
    "           'TVHrsDay', \n",
    "           'SexOrientation'] \n",
    "           'CompHrsDay', \n",
    "\n",
    "for column in dummies:\n",
    "    df_cut = pd.concat([df_cut, pd.get_dummies(df_cut[column], prefix=column)], axis=1)\n",
    "    \n",
    "df_cut.drop(dummies, axis=1)\n",
    "    \n",
    "print(df_cut.columns)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "yes_no_cols = ['HardDrugs','Diabetes', 'SleepTrouble', 'PhysActive', 'Alcohol12PlusYr', \n",
    "              'Smoke100', 'Marijuana', 'RegularMarij', 'HardDrugs', 'SexEver', 'SameSex']\n",
    "yes_vals = []\n",
    "no_vals = []\n",
    "\n",
    "for column in yes_no_cols:\n",
    "    yes_vals.append(df_cut[df_cut[column] == 'Yes']['flu'].sum())\n",
    "    no_vals.append(df_cut[df_cut[column] == 'No']['flu'].sum())\n",
    "\n",
    "ind = np.arange(len(yes_no_cols))    # the x locations for the groups\n",
    "width = 0.35       # the width of the bars: can also be len(x) sequence\n",
    "\n",
    "p1 = plt.bar(ind, no_vals, width)\n",
    "plt.figure(figsize=(10,5))\n",
    "p2 = plt.bar(ind, yes_vals, width,\n",
    "             bottom=no_vals)\n",
    "plt.ylabel('Amount')\n",
    "plt.title('Ratio between \\'Yes\\' and \\'No\\'')\n",
    "\n",
    "plt.xticks(ind, (yes_no_cols), rotation=50, ha='right')\n",
    "\n",
    "plt.show()"
    "plt.legend((p1[0], p2[0]), ('No', 'Yes'))\n",
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LinReg for unclear difference, else just choose the dominant answer for NaN values in columns, where it suggests a sick person\n",
    "# LinReg might create some bias/ some relation that is coincidental\n",
    "# Dominant answer for sick people will yield some false positives, preferable over sick people classified as healthy\n",
    "# df_cut['HardDrugs'].fillna('No')\n",
    "df_cut['HardDrugs'].replace(['No','Yes'],[0,1],inplace=True)\n",
    "\n",
    "# df_cut['Diabetes'].fillna('No')\n",
    "df_cut['Diabetes'].replace(['No','Yes'],[0,1],inplace=True)\n",
    "\n",
    "# df_cut['Alcohol12PlusYr'].fillna('Yes')\n",
    "df_cut['Alcohol12PlusYr'].replace(['No','Yes'],[0,1],inplace=True)\n",
    "\n",
    "# df_cut['SexEver'].fillna('Yes')\n",
    "df_cut['SexEver'].replace(['No','Yes'],[0,1],inplace=True)\n",
    "\n",
    "df_cut['SameSex'].fillna('No')\n",
    "df_cut['SameSex'].replace(['No','Yes'],[0,1],inplace=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To be tested:\n",
    "# SexOrientation\n",
    "# SameSex\n",
    "# SexEver\n",
    "# TVHrsDay\n",
    "# CompHrsDay\n",
    "# HomeOwn\n",
    "# MaritalStatus\n",
    "# Work\n",
    "# Race \n",
    "# Gender\n",
    "\n",
    "df_min = df_cut.drop(columns='SexOrientation', 'SameSex', 'SexEver', 'CompHrsDay', 'TVHrsDay', 'HomeOwn', 'Work', 'MaritalStatus', 'Race', 'Gender')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a dataframe for each potentially useful column. This way we can check if a column is relevant for health, or if it is noise."
   ]
  },
    "# df_cut['SameSex'].fillna('No')\n",
    "df_cut['SameSex'].replace(['No','Yes'],[0,1],inplace=True)"
    "df_cut['SameSex'].fillna('No')\n",
    "df_cut['SameSex'].replace(['No','Yes'],[0,1],inplace=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To be tested:\n",
    "# SexOrientation\n",
    "# SameSex\n",
    "# SexEver\n",
    "# CompHrsDay\n",
    "# TVHrsDay\n",
    "# HomeOwn\n",
    "# Work\n",
    "# MaritalStatus\n",
    "# Race \n",
    "# Gender\n",
    "\n",
    "df_min = df_cut.drop(columns='SexOrientation', 'SameSex', 'SexEver', 'CompHrsDay', 'TVHrsDay', 'HomeOwn', 'Work', 'MaritalStatus', 'Race', 'Gender')\n"
   ]
  },
  {
  {
  },
   ]
    "# print(y)"
    "# print(X)\n",
    "y = np.array(df_num_only['flu']).reshape(-1,1)\n",
    "X = np.array(df_num_only.loc[:, df_num_only.columns != 'flu'])\n",
    "df_num_only.fillna(0, inplace=True)\n",
    "\n",
    "\n",
    "used_columns = df_cut.select_dtypes(exclude='object').columns\n",
    "df_num_only = df_cut[df_cut.select_dtypes(exclude='object').columns]\n",
    "\n",
    "       + ['flu']]\n",
    "df_cut = df_cut[[c for c in df_cut if c not in ['flu']] \n",
    "\n",
   "source": [
   "outputs": [],
   "metadata": {},
   "execution_count": null,
   "cell_type": "code",
  {
   ]
  },
    "#     print(\"{} has range\\t\\t\\t[{}, {}]\".format(column, df_cut[column].min(), df_cut[column].max()))\n"
    "# for column in df_cut.select_dtypes(exclude='object').columns:\n",
   "source": [
   "outputs": [],
   "metadata": {},
   "execution_count": null,
   "cell_type": "code",
  {
  },
   ]
    "from sklearn.linear_model import LinearRegression\n"
    "from sklearn.model_selection import KFold\n",
    "from sklearn import preprocessing\n",
    "from datetime import datetime\n",
    "from scipy import stats\n",
   "source": [
   "outputs": [],
   "metadata": {},
   "execution_count": null,
   "cell_type": "code",
  {
  },
   "source": [
    "# Hier moeten nog de yes en no geinterpoleerd worden door linreg"
   ]
   "metadata": {},
   "cell_type": "markdown",
  {
  },
   ]
   "source": [
    "df_cut_dummies = pd.get_dummies(df_cut['SexOrientation'], prefix=['SexOrientation'])\n",
    "df_SexOrientation = df_min.join(df_cut_dummies)\n",
    "\n",
    "df_cut_dummies = pd.get_dummies(df_cut['Gender'], prefix=['Gender'])\n",
    "df_Gender = df_min.join(df_cut_dummies)\n"
   "outputs": [],
   "metadata": {},
   "execution_count": null,
   "cell_type": "code",
  {
  },
   ]
    "Create a dataframe for each potentially useful column. This way we can check if a column is relevant for health, or if it is noise."
   "source": [
   "metadata": {},
   "cell_type": "markdown",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now this is where we find some weird ranges in our data.\n",
    "* Weight has a range of 2.8 to 223 (Babies v.s. obese?)\n",
    "* Height has a range of 83.6 to 200.4 (Midgets v.s. Basketballers?)\n",
    "* Alcohol per Day has highest value of 82 (glasses of wine!?)\n",
    "* SexNumPartnLife probably has Hugh Hefner in the top with 2000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
    "from imblearn.over_sampling import BorderlineSMOTE\n",
    "X_resampled, y_resampled = BorderlineSMOTE().fit_resample(X, y.ravel())\n",
    "print(X.shape)\n",
    "print(y.shape)\n",
    "print(X_resampled.shape)\n",
    "print(y_resampled.shape)\n",
    "\n",
    "print(y.sum())\n",
    "print(y_resampled.sum())"
   ]
   "source": [
   "outputs": [],
   },
    "scrolled": true
   "metadata": {
   "execution_count": null,
   "cell_type": "code",
  {
  },
   ]
    "X_train_0, X_test_0, y_train_0, y_test_0 = train_test_split(X, y.ravel(), test_size=0.3)\n"
    "from sklearn.model_selection import train_test_split\n",
   "source": [
   "outputs": [],
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
    "sick_df = y.sum()\n",
    "okay_df = y_resampled.sum()\n",
    "\n",
    "plt.bar(['Sick before','Healthy before', 'Sick after', 'Healthy'], \n",
    "        [y.sum(),(len(y)-y.sum()),\n",
    "        y_resampled.sum(), (len(y_resampled)-y_resampled.sum())],\n",
    "        color=['purple', 'green'])\n",
    "plt.show()"
   ]
   "source": [
   "outputs": [],
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Model Choice\n",
    "\n",
    "The first task is to decide which classifier to use (from the ones that we learned this block), i.e. which one would best suit our task and our data. Note that our data are heavily unbalanced, thus you need to do some exploration on how different classifiers handle inbalances in the data (we will discuss some of these techniques during week 3 lecture).\n",
    "\n",
    "It would be possible to do brute force model comparison here - i.e. tune all models and compare which does best with respect to various benchmarks. However, it is also reasonable to do a first round of model comparison by running models (with out of the box parameter settings) on the training data and eliminating some models which performed very poorly.\n",
    "\n",
    "Let the best model win!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def expected_score(model, x_test, y_test):\n",
    "    overall = 0\n",
    "    class_0 = 0\n",
    "    class_1 = 0\n",
    "    for i in range(100):\n",
    "        sample = np.random.choice(len(x_test), len(x_test))\n",
    "        x_sub_test = x_test[sample]\n",
    "        y_sub_test = y_test[sample]\n",
    "        \n",
    "        overall += model.score(x_sub_test, y_sub_test)\n",
    "        class_0 += model.score(x_sub_test[y_sub_test==0], y_sub_test[y_sub_test==0])\n",
    "        class_1 += model.score(x_sub_test[y_sub_test==1], y_sub_test[y_sub_test==1])\n",
    "\n",
    "    return pd.Series([overall / 100., \n",
    "                      class_0 / 100.,\n",
    "                      class_1 / 100.],\n",
    "                      index=['overall accuracy', 'accuracy on class 0', 'accuracy on class 1'])\n",
    "\n",
    "score = lambda model, x_test, y_test: pd.Series([model.score(x_test, y_test), \n",
    "                                                 model.score(x_test[y_test==0], y_test[y_test==0]),\n",
    "                                                 model.score(x_test[y_test==1], y_test[y_test==1])], \n",
    "                                                index=['overall accuracy', 'accuracy on class 0', 'accuracy on class 1'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "models = [\n",
    "#            rfc, \n",
    "            abc, \n",
    "#           gnb, \n",
    "#           gbc, \n",
    "#           bgc, \n",
    "#           knc,\n",
    "          dtc\n",
    "         ]\n",
    "for model in models:\n",
    "    print(\"Training model\", model)\n",
    "    model.fit(X_train, y_train)\n",
    "    print(\"Calculating expected score\")\n",
    "    print(expected_score(model, X_test_0, y_test_0))\n",
    "    \n",
    "    \n",
    "print(\"done\")\n",
    "model = abc    "
    "X_train, X_test, y_train, y_test = train_test_split(X_resampled, y_resampled.ravel(), test_size=0.2)\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "knc = KNeighborsClassifier()\n",
    "bgc = BaggingClassifier()\n",
    "gbc = GradientBoostingClassifier()\n",
    "gnb = GaussianNB()\n",
    "abc = AdaBoostClassifier(n_estimators=10, learning_rate=0.1)\n",
    "rfc = RandomForestClassifier()\n",
    "dtc = DecisionTreeClassifier(splitter='random', class_weight='balanced')\n",
    "\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.ensemble import GradientBoostingClassifier, BaggingClassifier\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "# Let's start with some simple classifiers to see how they do on the train and test data\n",
    "\n",
    "### fancy models that solve the problem\n",
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that all these 'simple' classifiers without any parameters set all perform very bad on the 'flu' class. Here we also see how important it is to not only look at accuracy because they all score around 90% overall accuracy, which is totally not representative how the model performs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
   ]
    "\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "metadata": {},
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## On evaluation\n",
    "\n",
    "### AUROC\n",
    "\n",
    "As mentioned abbove, we will use the accuracy scores for each class and for the whole dataset, as well as the AUROC score from Kaggle platform. You can coimpute AUROC locally (e.g. on your train/validation set) by calling the relevant scikit learn function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "###AUROC locally\n",
    "\n",
    "#score = roc_auc_score(real_labels, predicted_labels)\n",
    "\n",
    "#real_labels: the ground truth (0 or 1)\n",
    "#predicted_labels: labels predicted by your algorithm (0 or 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accuracy (per class)\n",
    "\n",
    "Below there is a function that will be handy for your models. It computes the accuracy per-class, based on a model you pass as parameter and a dataset (split to x/y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extended_score(model, x_test, y_test):\n",
    "    overall = 0\n",
    "    class_0 = 0\n",
    "    class_1 = 0\n",
    "    for i in range(100):\n",
    "        sample = np.random.choice(len(x_test), len(x_test))\n",
    "        x_sub_test = x_test[sample]\n",
    "        y_sub_test = y_test[sample]\n",
    "        \n",
    "        overall += model.score(x_sub_test, y_sub_test)\n",
    "        class_0 += model.score(x_sub_test[y_sub_test==0], y_sub_test[y_sub_test==0])\n",
    "        class_1 += model.score(x_sub_test[y_sub_test==1], y_sub_test[y_sub_test==1])\n",
    "\n",
    "    return pd.Series([overall / 100., \n",
    "                      class_0 / 100.,\n",
    "                      class_1 / 100.],\n",
    "                      index=['overall accuracy', 'accuracy on class 0', 'accuracy on class 1'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#same job as before, but faster?\n",
    "\n",
    "score = lambda model, x_val, y_val: pd.Series([model.score(x_val, y_val), \n",
    "                                                 model.score(x_val[y_val==0], y_val[y_val==0]),\n",
    "                                                 model.score(x_val[y_val==1], y_val[y_val==1])], \n",
    "                                                index=['overall accuracy', 'accuracy on class 0', 'accuracy on class 1'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Solution extraction for Kaggle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure that you extract your solutions (predictions) in the correct format required by Kaggle"
   ]
  },
  {
<<<<<<< Updated upstream
=======
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test = pd.read_csv('data/flu_test.csv')\n",
    "\n",
    "df_test['Gender'].replace(['female','male'],[0,1],inplace=True)\n",
    "#df_test['HardDrugs'].fillna('No')\n",
    "df_test['HardDrugs'].replace(['No','Yes'],[0,1],inplace=True)\n",
    "\n",
    "#df_test['Diabetes'].fillna('No')\n",
    "df_test['Diabetes'].replace(['No','Yes'],[0,1],inplace=True)\n",
    "\n",
    "#df_test['Alcohol12PlusYr'].fillna('Yes')\n",
    "df_test['Alcohol12PlusYr'].replace(['No','Yes'],[0,1],inplace=True)\n",
    "\n",
    "#df_test['SexEver'].fillna('Yes')\n",
    "df_test['SexEver'].replace(['No','Yes'],[0,1],inplace=True)\n",
    "\n",
    "#df_test['SameSex'].fillna('No')\n",
    "df_test['SameSex'].replace(['No','Yes'],[0,1],inplace=True)\n",
    "\n",
    "dummies = ['Race1', \n",
    "           'Education', \n",
    "           'MaritalStatus', \n",
    "           'HomeOwn', \n",
    "           'BMI_WHO', \n",
    "           'HealthGen', \n",
    "           'LittleInterest', \n",
    "           'Depressed', \n",
    "           'TVHrsDay', \n",
    "           'CompHrsDay', \n",
    "           'SexOrientation'] \n",
    "\n",
    "for column in dummies:\n",
    "    df_test = pd.concat([df_test, pd.get_dummies(df_test[column], prefix=column)], axis=1)\n",
    "\n",
    "df_test = df_test[used_columns[:-1]]\n",
    "print(len(df_test.columns))\n",
    "    \n",
    "\n",
    "df_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.array(df_test.fillna(0))\n",
    "predictions = model.predict(X)\n",
    "\n",
    "file = open('predictions.csv', 'w')\n",
    "with file:\n",
    "\n",
    "    writer = csv.writer(file)\n",
    "    writer.writerow(['ID', 'Prediction'])\n",
    "    for i in range(len(predictions)):\n",
    "\n",
    "        writer.writerow([df_test.iloc[i]['ID'].astype(int), int(np.round(predictions[i]))])\n",
    "        "
   ]
  },
  {
>>>>>>> Stashed changes
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Conclusions\n",
    "\n",
    "Highlight at the end of your notebook, which were the top-3 approaches that produced the best scores for you. That is, provide a table with the scores you got (on the AUROC score you get from Kaggle) and make sure that you judge these in relation to your work on the training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}