examples/cloudml-bank-marketing/bank_marketing_classification_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ApXzF2S5lXAf"
   },
   "source": [
    "# How to build a machine learning marketing model for banking using Google Cloud Platform and Python\n",
    "\n",
    "This notebook shows you how to build a marketing model for banking using [Google Cloud Platform](https://cloud.google.com/) (GCP). Many financial institutions use traditional on-premise methods and tooling to build models for marketing. This notebook takes you through the steps to build a Machine Learning model using open source and Google Cloud Platform. Some of the advantages of this approach are:\n",
    "\n",
    "* Flexibility: The open source models can easily be ported. \n",
    "* Scalability: It's easy to scale using the power of Google Cloud.\n",
    "* Transparency: Lime will give you more insights in the model.\n",
    "* Ease of use: Pandas and Scikit-learn are easy to use.\n",
    "\n",
    "We will go into some important topics for modeling within banking, like data exploration and model explanation. This notebook is created so that it can be re-used for migrating workloads to open source and the cloud. We will take you through the following steps:\n",
    "\n",
    "* Fetching data from [Google BigQuery](https://cloud.google.com/bigquery/)\n",
    "*   Data Exploration using [Pandas profling](https://github.com/pandas-profiling/pandas-profiling)\n",
    "*   Data partitioning using [Scikit-learn](http://scikit-learn.org/)\n",
    "*   Data Engineering\n",
    "*   Building and evaluating different models\n",
    "*   Explaining the models using [Lime](https://github.com/marcotcr/lime)\n",
    "*   Use [Cloud Machine Learning Engine](https://cloud.google.com/ml-engine/) to deploy to model as an API\n",
    "*   Get predictions\n",
    "\n",
    "\n",
    "#### Type of model\n",
    "\n",
    "The goal of this model is to predict if the banking client will subscribe a term deposit, which is variable `y` in our dataset. This class of models are called \"propensity to buy\" models and this type of problem is binary classification. \"Propensity to buy\" models can help us predict the success of our marketing campaign.\n",
    "\n",
    "<br>\n",
    "\n",
    "#### What this notebook will not do\n",
    "\n",
    "* Teach you the basics of Machine Learning. We focus on how to train and deploy a ML model using the power of Google Cloud. \n",
    "* There is not one Cloud solution for all of your business problems. Because we choose Pandas and Scikit-Learn, they both have limitations. We choose to use Pandas + Scikit-learn because it helps making the transition from on-prem solutions to open source and Google Cloud easier. We have other solutions that can help you scale things even further using Google Cloud. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "G62K-mIIL5rT"
   },
   "source": [
    "## Prerequisites\n",
    "Before we get started we need to go through a couple of prerequisites. \n",
    "\n",
    "First, install two packages that your environment may not have (lime and pandas profiling). After running the cell, restart the Kernel by clicking 'Kernel > Restart Kernel' on the top menu."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pandas-profiling\n",
    "!pip install lime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*(Only if you are using a colab notebook)* We need to authenticate to Google Cloud and create the service client. After running the cell below, a link will appear which you need to click on and follow the instructions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EHixUTA3S_uC"
   },
   "outputs": [],
   "source": [
    "# ONLY RUN IF YOU ARE IN A COLAB NOTEBOOK.\n",
    "from google.colab import auth\n",
    "auth.authenticate_user()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5nUIZjy7WPah"
   },
   "source": [
    "Next, we need to set our project. Replace 'PROJECT_ID' with your GCP project ID."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3Abr7mSOV3sb"
   },
   "outputs": [],
   "source": [
    "%env GOOGLE_CLOUD_PROJECT=PROJECT_ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "C8kDQyTgUxn9"
   },
   "outputs": [],
   "source": [
    "!gcloud config set project $GOOGLE_CLOUD_PROJECT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "U2ehCXf6S6XK"
   },
   "source": [
    "Third, you need to activate some of the GCP services that we will be using. Run the following cell if you need to activate API's. Which can also be done via the GUI via APIs and Services -> Enable APIS and Services."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "S0VuViuwL5rU"
   },
   "outputs": [],
   "source": [
    "!gcloud services enable ml.googleapis.com\n",
    "!gcloud services enable bigquery-json.googleapis.com"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DaBNb7V0SAUq"
   },
   "source": [
    "The data that we will be using for this demo is the [UCI Bank Marketing Dataset](https://archive.ics.uci.edu/ml/datasets/bank+marketing). \n",
    "\n",
    "The first step we will need to take is to create a BigQuery dataset and a table so that we can store this data. Make sure that you replace `your_dataset` and `your_table` variables with any dataset and table name you want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ghr_crNYXH9w"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "your_dataset = 'your_dataset'\n",
    "your_table = 'your_table'\n",
    "project_id = os.environ[\"GOOGLE_CLOUD_PROJECT\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OQFkRqGrDL0V"
   },
   "outputs": [],
   "source": [
    "!bq mk -d {project_id}:{your_dataset}\n",
    "!bq mk -t {your_dataset}.{your_table}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dyI-AWNsqnJQ"
   },
   "source": [
    "There is a public dataset avaliable which has cleaned up some of the rows in the `UCI Bank Marketing Dataset`. We will download this file in the next cell and save locally as `data.csv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "r950lGFCcvQd"
   },
   "outputs": [],
   "source": [
    "!curl https://storage.googleapis.com/erwinh-public-data/bankingdata/bankingdata --output data.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cHvOu8R7rzRL"
   },
   "source": [
    "We will now upload the `data.csv` file into our BigQuery table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2ltxd27fZeC3"
   },
   "outputs": [],
   "source": [
    "!bq load --autodetect --source_format=CSV {your_dataset}.{your_table} data.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EOCzgdftcTXx"
   },
   "source": [
    "## 1) Fetching data \n",
    "\n",
    "In this chapter we will get data from BigQuery and create a Pandas dataframe that we will be using for data engineering, data visualization and modeling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "n0UKQ403briV"
   },
   "source": [
    "###  Data from BigQuery to Pandas\n",
    "We are going to use the datalab.bigquery library to fetch data from bigquery and load a Pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "n9Pd0hZA2KDZ"
   },
   "outputs": [],
   "source": [
    "#import pandas and bigquery library\n",
    "\n",
    "import pandas as pd\n",
    "from google.cloud import bigquery as bq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "C9ITEzJvTtcF"
   },
   "source": [
    "We doing two things in this cell:\n",
    "\n",
    "\n",
    "1.   We are executing an SQL query\n",
    "2.   We are converting the output from BQ into a pandas dataframe using `.to_dataframe()`\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "itAmFXV1bria"
   },
   "outputs": [],
   "source": [
    "# Execute the query and converts the result into a Dataframe\n",
    "\n",
    "client = bq.Client(project=project_id)\n",
    "df = client.query('''\n",
    "  SELECT\n",
    "    *\n",
    "  FROM\n",
    "    `%s.%s`\n",
    "''' % (your_dataset, your_table)).to_dataframe()\n",
    "\n",
    "df.head(3).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rlTX30n1Uv2-"
   },
   "source": [
    "We will now expore the data we got from BQ"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "v1l1lW6KU8EG"
   },
   "source": [
    "## 2) Data exploration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ecG7ZyjhU-7c"
   },
   "source": [
    "We will use [Pandas profiling](https://github.com/pandas-profiling/pandas-profiling) to perform data exploration. This will give us information including distributions for each feature, missing values, the maximum and minimum values and many more. These are all out of the box. Run the next cell first if you haven't installed pandas profiling. (Note if after you haved installed pandas profiling, you get an import error, restart your kernel and re-run all the cells up until this section)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "e0RZYFeyL5rv"
   },
   "outputs": [],
   "source": [
    "import pandas_profiling as pp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OLhnLGe-bril",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Let's create a Profile Report using the dataframe that we just created. \n",
    "pp.ProfileReport(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0nBOsN-Abrir"
   },
   "source": [
    "Some interesting points from the pandas profiling:\n",
    "* We have categorical and boolean columns which we need to convert to numeric values\n",
    "* The predictor value is very skewed (only 5289 defaulted compared to a massive 39922 users not defaulting) so we need to ensure that our training and testing splits are representative of this skew\n",
    "* No missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dLbmS1Xzbris"
   },
   "source": [
    "## 3) Data partitioning (split data into training and testing)\n",
    "\n",
    "As our dataset is highly skewed, we need to be very careful with our sampling approach. Two things need to be considered: \n",
    "\n",
    "1.   Shuffle the dataset to avoid any form of pre-ordering.\n",
    "2.   Use [stratified sampling](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedShuffleSplit.html) (SE). SE makes sure that both datasets (test, training) do not significantly differ for variables of interest. In our case we use SE to achieve a similar distribution of `y` for both datasets.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "21qr6opmbrit"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import StratifiedShuffleSplit\n",
    "\n",
    "#Here we apply a shuffle and stratified split to create a train and test set.\n",
    "split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=40)\n",
    "for train_index, test_index in split.split(df, df[\"y\"]):\n",
    "    strat_train_set = df.loc[train_index]\n",
    "    strat_test_set = df.loc[test_index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hCxaFLEbL5r2"
   },
   "outputs": [],
   "source": [
    "# check the split sizes\n",
    "\n",
    "print(strat_train_set.size)\n",
    "print(strat_test_set.size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6yfveEsqL5r8"
   },
   "outputs": [],
   "source": [
    "# We can now check the data\n",
    "strat_test_set.head(3).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OgqODGuHbrjI"
   },
   "source": [
    "## 4) Data preparation (feature engineering)\n",
    "\n",
    "Before we can create Machine Learning models, we need to format the data so that it is in a form that the models can understand.\n",
    "\n",
    "We need to do the following steps:\n",
    "\n",
    "1.   For the numeric columns, we need to normalize these columns so that one column with very large values does not bias the computation.\n",
    "2.   Turn categorical values into numeric values replacing each unique value in a column with an integer. For example, if a column named \"Colour\" has three unique strings \"red\", \"yellow\" and \"blue\" they will be assigned the values 0, 1 and 2 respectively. So each instance of yellow in that column will be replaced with 0. Note: [one hot encoding](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html) is an alternative method to convert categorical values to integers.\n",
    "3.   For True/False values we simply convert these to 1/0 respectively.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "k97YOu5ybrjJ"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gSEnTh95aM-P"
   },
   "source": [
    "Now we are going to ceate a function to split the label we want to predict and the feature that we will use to predict this value. In addition, we convert the label to 1/0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0N9-64iYbrjL"
   },
   "outputs": [],
   "source": [
    "def return_features_and_label(df):\n",
    "    \"\"\"returns features and labal given argument dataframe\"\"\"\n",
    "    \n",
    "    # Get all the columns except \"y\". It's also possible to exclude other columns\n",
    "    X = df.drop(\"y\", axis=1)\n",
    "    \n",
    "    Y = df[\"y\"].copy ()\n",
    "    # Convert our label to an integer\n",
    "    Y = LabelEncoder().fit_transform(Y) \n",
    "    return X, Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qYppEq_ebrjP"
   },
   "outputs": [],
   "source": [
    "train_features, train_label = return_features_and_label(strat_train_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lHhgCTeJbVUe"
   },
   "source": [
    "Our training dataset, `train_features`, contains both categorical and numeric values. However, we know that machine learning models can only use numeric values. The function below converts categorical variables to integers and then normalizes the current numeric columns so that certain columns with very large numbers would not over-power those columns whose values are not so large."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9XhZTGl0L5sG"
   },
   "outputs": [],
   "source": [
    "def data_pipeline(df):\n",
    "    \"\"\"Normalizes and converts data and returns dataframe \"\"\"\n",
    "  \n",
    "    num_cols = df.select_dtypes(include=np.number).columns\n",
    "    cat_cols = list(set(df.columns) - set(num_cols))\n",
    "    # Normalize Numeric Data\n",
    "    df[num_cols] = StandardScaler().fit_transform(df[num_cols])\n",
    "    # Convert categorical variables to integers\n",
    "    df[cat_cols] = df[cat_cols].apply(LabelEncoder().fit_transform)\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HvUC4An8L5sH"
   },
   "outputs": [],
   "source": [
    "train_features_prepared = data_pipeline(train_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tHg7Jb3Hbrjf"
   },
   "outputs": [],
   "source": [
    "train_features_prepared.head(3).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GRGEDxnjlL30"
   },
   "source": [
    "Some columns in our training dataset may not be very good predictors. This means that we should perform feature selection to get only the best predictors and reduce our time for training since our dataset will be much smaller."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EZ-SZH2nllkj"
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SelectKBest, f_classif  \n",
    "\n",
    "predictors = train_features_prepared.columns\n",
    "\n",
    "# Perform feature selection where `k` (5 in this case) indicates the number of features we wish to select\n",
    "selector = SelectKBest(f_classif, k=5)\n",
    "selector.fit(train_features_prepared[predictors], train_label) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5U1HSzDEeVxo"
   },
   "source": [
    "To visualize the selection, we can plot a graph to look at the scores for each feature. Note that the `duration` feature had 0 as its p-value and so it could not be shown in the logarithmic scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "n2tUoYMdL5sO"
   },
   "outputs": [],
   "source": [
    "# Get the p-values from our selector for each model and convert to a logarithmic scale for easy vizualization\n",
    "importance_score = -np.log(selector.pvalues_)\n",
    "\n",
    "# Plot each column with their importance score\n",
    "plt.rcParams[\"figure.figsize\"] = [14,7]\n",
    "plt.barh(range(len(predictors)), importance_score, color='C0')\n",
    "plt.ylabel(\"Predictors\")\n",
    "plt.title(\"Importance Score\")\n",
    "plt.yticks(range(len(predictors)), predictors)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ky9P0pDcL5sR"
   },
   "source": [
    "It's also possible to use a Tree classifier to select the best features. It's often a good option when you have a highly imbalanced dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ju1b_riCL5sR"
   },
   "outputs": [],
   "source": [
    "# Example of how to use a Tree classifier to select best features. \n",
    "\n",
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "from sklearn.feature_selection import SelectFromModel    \n",
    "\n",
    "predictors_tree = train_features_prepared.columns\n",
    "\n",
    "selector_clf = ExtraTreesClassifier(n_estimators=50, random_state=0)\n",
    "selector_clf.fit(train_features_prepared[predictors], train_label) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uV3BSz7zL5sU"
   },
   "outputs": [],
   "source": [
    "# Plotting feature importance\n",
    "\n",
    "importances = selector_clf.feature_importances_\n",
    "std = np.std([tree.feature_importances_ for tree in selector_clf.estimators_],\n",
    "             axis=0)\n",
    "\n",
    "plt.rcParams[\"figure.figsize\"] = [14,7]\n",
    "plt.barh(range(len(predictors_tree)), std, color='C0')\n",
    "plt.ylabel(\"Predictors\")\n",
    "plt.title(\"Importance Score\")\n",
    "plt.yticks(range(len(predictors_tree)), predictors_tree)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MJC5Y0OjoI6c"
   },
   "source": [
    "The plot sometimes may be difficult to know which are the top five features. We can display a simple table with the top selected features and their scores. We are using `SelectKBest` with `f_classif`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "k9PGxlvYllnc"
   },
   "outputs": [],
   "source": [
    "# Plot the top 5 features based on the Log Score that we calculated earlier.  \n",
    "train_prepared_indexs = [count for count, selected in enumerate(selector.get_support()) if selected == True]\n",
    "\n",
    "pd.DataFrame(\n",
    "    {'Feature' : predictors[train_prepared_indexs],\n",
    "     'Original Score': selector.pvalues_[train_prepared_indexs],\n",
    "     'Log Score' : importance_score[train_prepared_indexs]\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4yt7qIGMn_yr"
   },
   "source": [
    "Let us now create a training dataset that contains the top 5 features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8a7OKGddoAPn"
   },
   "outputs": [],
   "source": [
    "# Here we are creating our new dataframe based on the selected features (from selector)\n",
    "train_prepared_columns = [col for (selected, col) in zip(selector.get_support(), predictors) if selected == True]\n",
    "train_prepared = train_features_prepared[train_prepared_columns]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9a8bjys6brjl"
   },
   "source": [
    "## 5) Building and evaluation the models\n",
    "\n",
    "In this section we will be building models using [Scikit-Learn](http://scikit-learn.org/stable/). We show how [hyper parameter tuning / optimization](https://en.wikipedia.org/wiki/Hyperparameter_optimization) and model evaluation can be used to select the best model for deployment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_AqfURjDbrjl"
   },
   "outputs": [],
   "source": [
    "# Importing libraries needed\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier\n",
    "\n",
    "from sklearn.metrics import make_scorer\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import precision_score\n",
    "from sklearn.metrics import recall_score\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import auc\n",
    "from sklearn.metrics import roc_curve\n",
    "\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "R9SBpOghmJo1"
   },
   "source": [
    "The following code defines different hyperparameter combinations. More precisely, we define different model types (e.g., Logistic Regression, Support Vectors Machines (SVC)) and the corresponding lists of parameters that will be used during the optimization process (e.g., different kernel types for SVM)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-eXA8ULs09Zj"
   },
   "outputs": [],
   "source": [
    "# this function will create the classifiers (models) that we want to test\n",
    "def create_classifiers(): \n",
    "    \"\"\"Create classifiers and specify hyper parameters\"\"\"\n",
    "    \n",
    "    log_params = [{'penalty': ['l1', 'l2'], 'C': np.logspace(0, 4, 10)}]\n",
    "\n",
    "    knn_params = [{'n_neighbors': [3, 4, 5]}]\n",
    "\n",
    "    svc_params = [{'kernel': ['linear', 'rbf'], 'probability': [True]}]\n",
    "\n",
    "    tree_params = [{'criterion': ['gini', 'entropy']}]\n",
    "\n",
    "    forest_params = {'n_estimators': [1, 5, 10]}\n",
    "\n",
    "    mlp_params = {'activation': [\n",
    "                    'identity', 'logistic', 'tanh', 'relu'\n",
    "                  ]}\n",
    "\n",
    "    ada_params = {'n_estimators': [1, 5, 10]}\n",
    "\n",
    "    classifiers = [\n",
    "        ['LogisticRegression', LogisticRegression(random_state=42),\n",
    "         log_params],\n",
    "        ['KNeighborsClassifier', KNeighborsClassifier(), knn_params],\n",
    "        ['SVC', SVC(random_state=42), svc_params],\n",
    "        ['DecisionTreeClassifier',\n",
    "         DecisionTreeClassifier(random_state=42), tree_params],\n",
    "        ['RandomForestClassifier',\n",
    "         RandomForestClassifier(random_state=42), forest_params],\n",
    "        ['MLPClassifier', MLPClassifier(random_state=42), mlp_params],\n",
    "        ['AdaBoostClassifier', AdaBoostClassifier(random_state=42),\n",
    "         ada_params],\n",
    "        ]\n",
    "\n",
    "    return classifiers\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8rHVWMQ0omEL"
   },
   "source": [
    "After defining our hyperparameters, we use sklearn's [grid search](http://scikit-learn.org/stable/modules/grid_search.html) to iterate through the different combinations of hyperparameters and return the best parameters for each model type. Furthermore, we use [crossvalidation](http://scikit-learn.org/stable/modules/cross_validation.html), pruning the data into smaller subsets (see [K-fold cross validation](https://www.cs.cmu.edu/~schneide/tut5/node42.html))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "x4y0Nfr7brjq"
   },
   "outputs": [],
   "source": [
    "# this grid search will iterate through the different combinations and returns the best parameters for each model type. \n",
    "# Running this cell might take a while\n",
    "\n",
    "def grid_search(model, parameters, name,training_features, training_labels):\n",
    "    \"\"\"Grid search that returns best parameters for each model type\"\"\"\n",
    "    \n",
    "    clf = GridSearchCV(model, parameters, cv=3, refit = 'f1',\n",
    "                       scoring='f1', verbose=0, n_jobs=4)\n",
    "    clf.fit(training_features, training_labels)\n",
    "    best_estimator = clf.best_estimator_\n",
    "  \n",
    "    return [name, str(clf.best_params_), clf.best_score_,\n",
    "            best_estimator]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "X1cRRxe_sJWD"
   },
   "source": [
    "Finally, we define a process enabling us to return the best configuration for each model using cross-validation (the best model is selected based on its F1-score)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "37gGBJOHbrjs"
   },
   "outputs": [],
   "source": [
    "# Now we want to get the best configuration for each model. \n",
    "\n",
    "def best_configuration(classifiers, training_features, training_labels):\n",
    "    \"\"\"returns the best configuration for each model\"\"\"\n",
    "    \n",
    "    clfs_best_config = []\n",
    "\n",
    "    for (name, model, parameters) in classifiers:\n",
    "        clfs_best_config.append(grid_search(model, parameters, name,\n",
    "                                training_features, training_labels))\n",
    "    return clfs_best_config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oxyY4Kcybrjw"
   },
   "outputs": [],
   "source": [
    "# Here we call the Grid search and Best_configuration function (note we only use 100 rows to decrease the run time). \n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "classifiers = create_classifiers()\n",
    "clfs_best_config = best_configuration(classifiers, train_prepared[:100], train_label[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1EBvAeuF4KQT"
   },
   "source": [
    "### Evaluation of model performance\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZSsQU86TUJuv"
   },
   "source": [
    "In order to choose the best performing model, we shall compare each of the models on the held-out test dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JKtRPARxUKLB"
   },
   "outputs": [],
   "source": [
    "# Prepare the test data for prediction\n",
    "test_features, test_label = return_features_and_label(strat_test_set)\n",
    "test_features_prepared = data_pipeline(test_features)\n",
    "test_prepared = test_features_prepared[train_prepared_columns]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2tSPx-zJMiYs"
   },
   "source": [
    "### Model comparison\n",
    "\n",
    "To compare the performance of different models we create a table with different metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4Ovb2mPk28P6"
   },
   "outputs": [],
   "source": [
    "f1_score_list = []\n",
    "accuracy_list = []\n",
    "precision_list = []\n",
    "recall_list = []\n",
    "roc_auc_list = []\n",
    "model_name_list = []\n",
    "\n",
    "# Iterate through the different model combinations to calculate perf. metrics.\n",
    "\n",
    "for name, params, score, model in clfs_best_config:\n",
    "    pred_label = model.predict(test_prepared) # Predict outcome.\n",
    "    f1_score_list.append(f1_score(test_label,pred_label)) # F1 score.\n",
    "    accuracy_list.append(accuracy_score(test_label, pred_label)) # Accuracy score.\n",
    "    precision_list.append(precision_score(test_label, pred_label)) # Precision score.\n",
    "    recall_list.append(recall_score(test_label, pred_label)) # Recall score.\n",
    "    roc_auc_list.append(roc_auc_score(test_label,\n",
    "                        model.predict_proba(test_prepared)[:, 1]))  # Predict probability.\n",
    "    model_name_list.append(name)\n",
    "\n",
    "# Sum up metrics in a pandas data frame.\n",
    "pd.DataFrame(\n",
    "    {'Model' : model_name_list,\n",
    "     'F1 Score' : f1_score_list,\n",
    "     'Accurary': accuracy_list,\n",
    "     'Precision': precision_list,\n",
    "     'Recall': recall_list,\n",
    "     'Roc_Auc': roc_auc_list \n",
    "    },\n",
    "    columns = ['Model','F1 Score','Precision','Recall', 'Accurary', 'Roc_Auc']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KAAFrXCNMA_0"
   },
   "source": [
    "### Graphical comparison\n",
    "\n",
    "For the graphical representation of model performance we use roc curves to highlight the True Positive Rate (TPR), also known as recall, and the False Positive Rate (FPR)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "N31vyvUV8Gek"
   },
   "outputs": [],
   "source": [
    "# Create a function that plots an ROC curve\n",
    "def roc_graph(test_label, pred_label, name):\n",
    "  \"\"\"Plots the ROC curve's in a Graph\"\"\"\n",
    "  \n",
    "  fpr, tpr, thresholds = roc_curve(test_label, pred_label, pos_label=1)\n",
    "  roc_auc = auc(fpr, tpr)\n",
    "  plt.plot(fpr, tpr, lw=2, label='%s ROC (area = %0.2f)' % (name, roc_auc))\n",
    "\n",
    "plt.clf()\n",
    "\n",
    "# Iterate though the models, create ROC graph for each model.\n",
    "for name, _, _, model in clfs_best_config:\n",
    "    pred_label = model.predict_proba(test_prepared)[:,1]\n",
    "    roc_graph(test_label, pred_label, name)\n",
    "\n",
    "plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.0])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('ROC Curves ')\n",
    "plt.legend(loc=\"lower right\", fontsize='small')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qdAit7p8yMYe"
   },
   "source": [
    "Now that we have all these evaluation metrics, we can select a model based on which metrics we want out models to maximize or minimize."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xHQHiFAy2CkB"
   },
   "source": [
    "## 6) Explaning the model\n",
    "\n",
    "We use the python package [LIME](https://github.com/marcotcr/lime) to explain the model and so we will move from our use of pandas to numpy matrices since this is what LIME accepts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cDeqSUof2BIy"
   },
   "outputs": [],
   "source": [
    "import lime.lime_tabular\n",
    "import lime\n",
    "import sklearn\n",
    "import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4Kh_-IdSuMaD"
   },
   "source": [
    "The first thing we will do is to get the unique values from our label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "N1jegoxzCCXJ"
   },
   "outputs": [],
   "source": [
    "class_names = strat_train_set[\"y\"].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3DYgVsUYuxPJ"
   },
   "source": [
    "We need a dataset with our top 5 features but with the categorical values still present. This will allow LIME to know how it should display our features. E.g. using our column example earlier, it will know to display \"yellow\" whenever it sees a 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JOKMP7ASuwm5"
   },
   "outputs": [],
   "source": [
    "train = train_features[train_prepared_columns].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6l1VzbPkxNSI"
   },
   "source": [
    "LIME needs to know the index of each catergorical column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Pp7xRp_uB8Vy"
   },
   "outputs": [],
   "source": [
    "num_cols = train_features._get_numeric_data().columns\n",
    "cat_cols = list(set(train_features.columns) - set(num_cols))\n",
    "\n",
    "categorical_features_index = [i for i, val in enumerate(train_prepared_columns) if val in cat_cols]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZeiLEj3cxc6v"
   },
   "source": [
    "In addition, LIME requires a dictionary which contains the name of each column and the unique values for each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oXblckKvB8Yb"
   },
   "outputs": [],
   "source": [
    "categorical_names = {}\n",
    "for feature in categorical_features_index:\n",
    "    # We still need to convert catergorical variables to integers\n",
    "    le = sklearn.preprocessing.LabelEncoder()\n",
    "    le.fit(train[:, feature])\n",
    "    train[:, feature] = le.transform(train[:, feature])\n",
    "    categorical_names[feature] = le.classes_\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZTMSS3K5dyLK"
   },
   "source": [
    "Create a function that will return the probability that the model (in our case we chose the logistic regression model) selects a certain class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MtwJyTXmthz6"
   },
   "outputs": [],
   "source": [
    "predict_fn = lambda x: clfs_best_config[0][-1].predict_proba(x).astype(float)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2bGLdgPGd6sS"
   },
   "source": [
    "Use the LIME package to configure a variable that can be used to explain predicitons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0UiCdC9YB8op"
   },
   "outputs": [],
   "source": [
    "explainer = lime.lime_tabular.LimeTabularExplainer(train, feature_names=train_prepared_columns,class_names=class_names,\n",
    "                                                   categorical_features=categorical_features_index, \n",
    "                                                   categorical_names=categorical_names, kernel_width=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sNyCyk-IeFzL"
   },
   "source": [
    "When you would like to understand the prediction of a value in the test set, create an explanation instance and show the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "beyQg2KzCz0m"
   },
   "outputs": [],
   "source": [
    "i = 106\n",
    "exp = explainer.explain_instance(train[i], predict_fn)\n",
    "pprint.pprint(exp.as_list())\n",
    "fig = exp.as_pyplot_figure()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JxwAq863NLlL"
   },
   "source": [
    "## 7) Train and Predict with Cloud Machine Learning Engine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5EjqbpmGyujl"
   },
   "source": [
    "We just saw how to train and predict our models locally. However, when we want more compute power or want to put our model in production serving 1000s of requests, we can use [Cloud Machine Learning Engine](https://cloud.google.com/ml-engine/docs/scikit/) to perform these tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cAfB1M0KzAdP"
   },
   "source": [
    "Let us define some environment variables that CMLE uses. Do not forget to replace all the variables in square brackets (along with the square brackets) with your credentials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mxW8DWZ8L5tM"
   },
   "outputs": [],
   "source": [
    "%env GCS_BUCKET=<GCS_BUCKET>\n",
    "%env PROJECT_ID=<PROJECT_ID> \n",
    "%env REGION=us-central1\n",
    "%env LOCAL_DIRECTORY=./trainer/data\n",
    "%env TRAINER_PACKAGE_PATH=./trainer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QWQ8kPKc9rYk"
   },
   "source": [
    "ML Engine needs a Python package with our code to train models. We need to create a directory and move our code there. We also need to create an `__init__.py` file, this is a unique feature of python. You can read the [docs](https://docs.python.org/3/tutorial/modules.html#packages) to understand more about this file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4TMA6-l5L5tO"
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "mkdir trainer\n",
    "touch trainer/__init__.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GuvKzNKMuFQL"
   },
   "source": [
    "In order for Cloud Machine Learning Engine to access the training data we need to upload a trainging file into Google Cloud Storage (GCS). We use our `strat_train_set` dataframe and convert it into a csv file which we upload to GCS. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "27EgFUSfuFfS"
   },
   "outputs": [],
   "source": [
    "strat_train_set.to_csv('train.csv', index=None)\n",
    "!gsutil cp train.csv $GCS_BUCKET"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JyPTUKus-Q97"
   },
   "source": [
    "This next cell might seem very long,  however, most of the code is identical to earlier sections. We are simply combining the code we created previously into one file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zVj9A-GML5tQ"
   },
   "source": [
    "Before running this, cell substitute `<BUCKET_NAME>` with your GCS bucket name. Do not include the `'gs://'` prefix.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ykFjpT7UL5tQ"
   },
   "outputs": [],
   "source": [
    "%%writefile trainer/task.py\n",
    "\n",
    "import datetime\n",
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import subprocess\n",
    "\n",
    "from google.cloud import storage\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.externals import joblib\n",
    "from sklearn.feature_selection import SelectKBest, f_classif\n",
    "from sklearn.pipeline import FeatureUnion\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import LabelEncoder, StandardScaler\n",
    "\n",
    "from sklearn.metrics import make_scorer\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier\n",
    "\n",
    "# TODO: REPLACE '<BUCKET_NAME>' with your GCS bucket name\n",
    "BUCKET_NAME = <BUCKET_NAME>\n",
    "\n",
    "# Bucket holding the training data\n",
    "bucket = storage.Client().bucket(BUCKET_NAME)\n",
    "\n",
    "# Path to the data inside the  bucket\n",
    "blob = bucket.blob('train.csv')\n",
    "# Download the data\n",
    "blob.download_to_filename('train.csv')\n",
    "# [END download-data]\n",
    "\n",
    "# [START scikit-learn code]\n",
    "# Load the training dataset\n",
    "with open('./train.csv', 'r') as train_data:\n",
    "    df = pd.read_csv(train_data)\n",
    "\n",
    "def return_features_and_label(df_tmp):\n",
    "    # Get all the columns except the one named \"y\"\n",
    "    X = df_tmp.drop(\"y\", axis=1)\n",
    "    Y = df_tmp[\"y\"].copy()\n",
    "    # Convert label to an integer\n",
    "    Y = LabelEncoder().fit_transform(Y)\n",
    "    return X, Y\n",
    "\n",
    "def data_pipeline(df_tmp):\n",
    "    num_cols = df_tmp._get_numeric_data().columns\n",
    "    cat_cols = list(set(df_tmp.columns) - set(num_cols))\n",
    "    # Normalize Numeric Data\n",
    "    df_tmp[num_cols] = StandardScaler().fit_transform(df_tmp[num_cols])\n",
    "    # Convert categorical variables to integers\n",
    "    df_tmp[cat_cols] = df_tmp[cat_cols].apply(LabelEncoder().fit_transform)\n",
    "    return df_tmp\n",
    "\n",
    "def create_classifiers():\n",
    "    log_params = [{'penalty': ['l1', 'l2'], 'C': np.logspace(0, 4, 10)}]\n",
    "\n",
    "    knn_params = [{'n_neighbors': [3, 4, 5]}]\n",
    "\n",
    "    svc_params = [{'kernel': ['linear', 'rbf'], 'probability': [True]}]\n",
    "\n",
    "    tree_params = [{'criterion': ['gini', 'entropy']}]\n",
    "\n",
    "    forest_params = {'n_estimators': [1, 5, 10]}\n",
    "\n",
    "    mlp_params = {'activation': [\n",
    "                    'identity', 'logistic', 'tanh', 'relu'\n",
    "                  ]}\n",
    "\n",
    "    ada_params = {'n_estimators': [1, 5, 10]}\n",
    "\n",
    "    classifiers = [\n",
    "        ['LogisticRegression', LogisticRegression(random_state=42),\n",
    "         log_params],\n",
    "        ['KNeighborsClassifier', KNeighborsClassifier(), knn_params],\n",
    "        ['SVC', SVC(random_state=42), svc_params],\n",
    "        ['DecisionTreeClassifier',\n",
    "         DecisionTreeClassifier(random_state=42), tree_params],\n",
    "        ['RandomForestClassifier',\n",
    "         RandomForestClassifier(random_state=42), forest_params],\n",
    "        ['MLPClassifier', MLPClassifier(random_state=42), mlp_params],\n",
    "        ['AdaBoostClassifier', AdaBoostClassifier(random_state=42),\n",
    "         ada_params],\n",
    "        ]\n",
    "\n",
    "    return classifiers\n",
    "\n",
    "\n",
    "def grid_search(model, parameters, name, X, y):\n",
    "    clf = GridSearchCV(model, parameters, cv=3, refit = 'f1',\n",
    "                       scoring='f1', verbose=0, n_jobs=4)\n",
    "    clf.fit(X, y)\n",
    "    best_estimator = clf.best_estimator_\n",
    "  \n",
    "    return [name, clf.best_score_, best_estimator]\n",
    "\n",
    "\n",
    "def best_configuration(classifiers, training_values, testing_values):\n",
    "    clfs_best_config = []\n",
    "    best_clf = None\n",
    "    best_score = 0\n",
    "\n",
    "    for (name, model, parameters) in classifiers:\n",
    "        clfs_best_config.append(grid_search(model, parameters, name,\n",
    "                                training_values, testing_values))\n",
    "    \n",
    "    for name, quality, clf in clfs_best_config:\n",
    "        if quality > best_score:\n",
    "            best_score = quality\n",
    "            best_clf = clf\n",
    "    \n",
    "    return best_clf\n",
    "\n",
    "\n",
    "train_features, train_label = return_features_and_label(df)\n",
    "train_features_prepared = data_pipeline(train_features)\n",
    "\n",
    "predictors = train_features_prepared.columns\n",
    "# Perform feature selection\n",
    "selector = SelectKBest(f_classif, k=5)\n",
    "selector.fit(train_features_prepared[predictors], train_label)\n",
    "\n",
    "train_prepared_columns = [col for (selected, col) in zip(selector.get_support(), predictors) if selected == True]\n",
    "train_features_prepared = train_features_prepared[train_prepared_columns]\n",
    "\n",
    "\n",
    "x = train_features_prepared.values\n",
    "y = train_label\n",
    "classifiers = create_classifiers()\n",
    "clf = best_configuration(classifiers, x[:100], y[:100])\n",
    "# [END scikit-learn]\n",
    "\n",
    "\n",
    "# [START export-to-gcs]\n",
    "# Export the model to a file\n",
    "model = 'model.joblib'\n",
    "joblib.dump(clf, model)\n",
    "\n",
    "# Upload the model to GCS\n",
    "bucket = storage.Client().bucket(BUCKET_NAME)\n",
    "blob = bucket.blob('{}/{}'.format(\n",
    "    datetime.datetime.now().strftime('model_%Y%m%d_%H%M%S'),\n",
    "    model))\n",
    "blob.upload_from_filename(model)\n",
    "# [END export-to-gcs]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yStJtBfqFlGF"
   },
   "source": [
    "To actully run the `train.py` file we need some parameters so that CMLE knows how to set up the environment to run sucessfully."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NJmrxV6tNdtB"
   },
   "outputs": [],
   "source": [
    "%%bash \n",
    "\n",
    "JOBNAME=banking_$(date -u +%y%m%d_%H%M%S)\n",
    "\n",
    "echo $JOBNAME \n",
    "\n",
    "gcloud ml-engine jobs submit training model_training_$JOBNAME \\\n",
    "        --job-dir $GCS_BUCKET/$JOBNAME/output \\\n",
    "        --package-path trainer \\\n",
    "        --module-name trainer.task \\\n",
    "        --region $REGION \\\n",
    "        --runtime-version=1.9 \\\n",
    "        --python-version=3.5 \\\n",
    "        --scale-tier BASIC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EdG8UV2RL5tV"
   },
   "source": [
    "The above cell submits a job to CMLE which you can view by going to the Google Cloud Console's sidebar and select `ML Engine > Jobs` or search `ML Engine` in the search bar. ONLY run the cells below after your job has completed sucessfully. (It should take approximately 8 minutes to run).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_COTlRphmMA-"
   },
   "source": [
    "\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oWGm6c6KIs2O"
   },
   "source": [
    "Now that we have trained our model and it is saved in GCS we need to perform prediction. There are two options available to use for prediction:\n",
    "\n",
    "1.   Command Line\n",
    "2.   Python\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tJdbJg2jNd05"
   },
   "outputs": [],
   "source": [
    "test_features_prepared = data_pipeline(test_features)\n",
    "test_prepared = test_features_prepared[train_prepared_columns]\n",
    "test = test_prepared.as_matrix().tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8AlU8Bo2KJS8"
   },
   "source": [
    "Just like training in CMLE, we set some environment variables when we run our command line commands. Note that `<MODEL_DIRECTORY>` can be found in your bucket, it will be of the form `model_YYYYMMDD_HHMMSS` (e.g. `model_190114_134228`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pVXCz7X8L5tc"
   },
   "outputs": [],
   "source": [
    "%env VERSION_NAME=v1\n",
    "%env MODEL_NAME=cmle_model\n",
    "%env JSON_INSTANCE=input.json\n",
    "%env MODEL_DIR=gs://<MODEL_DIRECTORY>/model_YYYYMMDD_HHMMSS\n",
    "%env FRAMEWORK=SCIKIT_LEARN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jwKMqPziKf-b"
   },
   "source": [
    "Create a model resource for your model versions as well as the version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "M57aiibWNd3l"
   },
   "outputs": [],
   "source": [
    "! gcloud ml-engine models create $MODEL_NAME --regions=us-central1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oZ35NchhOBQs"
   },
   "outputs": [],
   "source": [
    "! gcloud ml-engine versions create $VERSION_NAME \\\n",
    "        --model $MODEL_NAME --origin $MODEL_DIR \\\n",
    "        --runtime-version 1.9 --framework $FRAMEWORK \\\n",
    "        --python-version 3.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lotdJpT8LIFS"
   },
   "source": [
    "For prediction, we will upload a file with one line in our GCS bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hgkh9wN_OBTo"
   },
   "outputs": [],
   "source": [
    "import json\n",
    "with open('input.json', 'w') as outfile:\n",
    "  json.dump(test[0], outfile)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HHAZLchKOBWh"
   },
   "outputs": [],
   "source": [
    "!gsutil cp input.json $GCS_BUCKET"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nY3S-jxqLjVz"
   },
   "source": [
    "We are now ready to submit our file to get a preditcion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MusDyG42OBZm"
   },
   "outputs": [],
   "source": [
    "! gcloud ml-engine predict --model $MODEL_NAME \\\n",
    "        --version $VERSION_NAME \\\n",
    "        --json-instances $JSON_INSTANCE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mTrHlVsHLo1B"
   },
   "source": [
    "We can also use python to perform predictions. See the cell below for a simple way to get predictions using python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "X_IisRM3OTCs"
   },
   "outputs": [],
   "source": [
    "import googleapiclient.discovery\n",
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "PROJECT_ID = os.environ['PROJECT_ID']\n",
    "VERSION_NAME = os.environ['VERSION_NAME']\n",
    "MODEL_NAME = os.environ['MODEL_NAME']\n",
    "\n",
    "# Create our CMLE service\n",
    "service = googleapiclient.discovery.build('ml', 'v1')\n",
    "name = 'projects/{}/models/{}'.format(PROJECT_ID, MODEL_NAME)\n",
    "name += '/versions/{}'.format(VERSION_NAME)\n",
    "\n",
    "# Iterate over the first 10 rows of our test dataset\n",
    "results = []\n",
    "for data in test[:10]:\n",
    "    # Send a prediction request \n",
    "    responses = service.projects().predict(\n",
    "        name=name,\n",
    "        body={\"instances\": [data]}\n",
    "    ).execute()\n",
    "\n",
    "    if 'error' in responses:\n",
    "        raise RuntimeError(response['error'])\n",
    "    else:\n",
    "        results.extend(responses['predictions'])\n",
    "\n",
    "for i, response in enumerate(results):\n",
    "    print('Prediction: {}\\tLabel: {}'.format(response, test_label[i]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CieZLu6DL5tv"
   },
   "source": [
    "Copyright 2019 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "How to build a machine learning marketing model for banking using Google Cloud Platform and Python.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}