Intro_to_ML_notebook.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Intro to ML notebook.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Neofly4023/intro-to-ML-workshop/blob/master/Intro_to_ML_notebook.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "ErAGgD_8g5KI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Introduction\n",
        "\n",
        "Before we start, there's an easter egg, if you can find the origin of this paragraph, congrats :D\n",
        "\n",
        "“ People of tomorrow, the key was hidden from the eyes of the world and is now found again by the purest of heart.  The sacred guardians of the key were appointed by nature itself to protect the machine and guard its sacred purpose.  Until the time came when human kind needed its power once more ... ”\n",
        "\n",
        "\n",
        "We'll start with an overview of how machine learning models work and how they are used. This may feel basic if you've done statistical modeling or machine learning before.\n",
        "\n",
        "The workshop will have you build models for the following scenario:\n",
        "\n",
        "Your cousin has made millions of dollars speculating on real estate. He's offered to become business partners with you because of your interest in data science. He'll supply the money, and you'll supply models that predict how much various houses are worth.\n",
        "\n",
        "You ask your cousin how he's predicted real estate values in the past. and he says it is just intuition. But more questioning reveals that he's identified price patterns from houses he has seen in the past, and he uses those patterns to make predictions for new houses he is considering.\n",
        "\n",
        "Machine learning works the same way. We'll start with a model called the Decision Tree. There are fancier models that give more accurate predictions. But decision trees are easy to understand, and they are the basic building block for some of the best models in data science.\n",
        "\n",
        "For simplicity, we'll start with the simplest possible decision tree.\n",
        "\n",
        "![Decision tree](https://i.imgur.com/7tsb5b1.png)\n",
        "\n",
        "It divides houses into only two categories. The predicted price for any house under consideration is the historical average price of houses in the same category.\n",
        "\n",
        "We use data to decide how to break the houses into two groups, and then again to determine the predicted price in each group. This step of capturing patterns from data is called **fitting** or **training** the model. The data used to **fit** the model is called the **training data**.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "BLJ744wJl1nT",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Improving the Decision Tree\n",
        "\n",
        "Which of the following two decisions trees is more likely to result from fitting the real estate training data?\n",
        "\n",
        "![Improving trees](http://i.imgur.com/prAjgku.png)\n",
        "\n",
        "The decision tree on the left (Decision Tree 1) probably makes more sense, because it captures the reality that houses with more bedrooms tend to sell at higher prices than houses with fewer bedrooms. The biggest shortcoming of this model is that it doesn't capture most factors affecting home price, like number of bathrooms, lot size, location, etc.\n",
        "\n",
        "You can capture more factors using a tree that has more \"splits.\" These are called \"deeper\" trees. A decision tree that also considers the total size of each house's lot might look like this:\n",
        "\n",
        "![Deeper tree](http://i.imgur.com/R3ywQsR.png)\n",
        "\n",
        "You predict the price of any house by tracing through the decision tree, always picking the path corresponding to that house's characteristics. The predicted price for the house is at the bottom of the tree. The point at the bottom where we make a prediction is called a **leaf**.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "oSfG9NtsmaTU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Using Pandas to Get Familiar With Your Data\n",
        "\n",
        "The first step in any machine learning project is familiarize yourself with the data. You'll use the Pandas library for this. Pandas is the primary tool data scientists use for exploring and manipulating data. Most people abbreviate pandas in their code as pd. We do this with the command :\n"
      ]
    },
    {
      "metadata": {
        "id": "akKrNN9ag4HD",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import pandas as pd"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "akLJOacrmhW9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The most important part of the Pandas library is the DataFrame. A DataFrame holds the type of data you might think of as a table. This is similar to a sheet in Excel, or a table in a SQL database.\n",
        "\n",
        "Pandas has powerful methods for most things you'll want to do with this type of data.\n",
        "\n",
        "As an example, we'll looking at data about home prices in Melbourne, Australia. In the hands-on exercises, you will apply the same processes to a new dataset, which has home prices in Iowa.\n",
        "\n",
        "The example (Melbourne) data is at the file path located at the [github repo](https://github.com/Neofly4023/intro-to-ML-workshop) I created for this workshop \n",
        "\n",
        "We load and explore the data with the following commands:"
      ]
    },
    {
      "metadata": {
        "id": "IHFonr5T9owf",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# create folders to hold the datasets in this environment\n",
        "\n",
        "!mkdir -p input/home-data-for-ml-course input/melbourne-housing-snapshot\n",
        "!wget https://raw.githubusercontent.com/Neofly4023/intro-to-ML-workshop/master/input/melbourne-housing-snapshot/melb_data.csv -O ./input/melbourne-housing-snapshot/melb_data.csv\n",
        "!wget https://raw.githubusercontent.com/Neofly4023/intro-to-ML-workshop/master/input/home-data-for-ml-course/train.csv -O ./input/home-data-for-ml-course/train.csv \n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "W_o6hMw7nZ__",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# save filepath to variable for easier access\n",
        "melbourne_file_path = './input/melbourne-housing-snapshot/melb_data.csv'\n",
        "# read the data and store data in DataFrame titled melbourne_data\n",
        "melbourne_data = pd.read_csv(melbourne_file_path) \n",
        "# print a summary of the data in Melbourne data\n",
        "melbourne_data.describe()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4OSBQK4sqVaU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Interpreting Data Description\n",
        "\n",
        "The results show 8 numbers for each column in your original dataset. The first number, the **count**, shows how many rows have non-missing values.\n",
        "\n",
        "Missing values arise for many reasons. For example, the size of the 2nd bedroom wouldn't be collected when surveying a 1 bedroom house. We'll come back to the topic of missing data.\n",
        "\n",
        "The second value is the **mean**, which is the average. Under that, **std** is the standard deviation, which measures how numerically spread out the values are.\n",
        "\n",
        "To interpret the **min, 25%, 50%, 75% ** and **max** values, imagine sorting each column from lowest to highest value. The first (smallest) value is the min. If you go a quarter way through the list, you'll find a number that is bigger than 25% of the values and smaller than 75% of the values. That is the 25% value (pronounced \"25th percentile\"). The 50th and 75th percentiles are defined analogously, and the max is the largest number.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "y4i5L54xw3sh",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Exercise 1\n",
        "\n",
        "Get started with your first coding exercise.\n",
        "\n",
        "This exercise will test your ability to read a data file and understand statistics about the data.\n",
        "\n",
        "In later exercises, you will apply techniques to filter the data, build a machine learning model, and iteratively improve your model.\n",
        "\n",
        "The course examples use data from Melbourne. To ensure you can apply these techniques on your own, you will have to apply them to a new dataset (with house prices from Iowa).\n",
        "\n",
        "The exercises use a \"notebook\" coding environment.  In case you are unfamiliar with notebooks, we have a [90-second intro video](https://www.youtube.com/watch?v=4C2qMnaIKL4).\n"
      ]
    },
    {
      "metadata": {
        "id": "3SBc1J6crcoK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Step 1: Loading Data\n",
        "Read the Iowa data file into a Pandas DataFrame called `home_data`.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "iB4u7W7Irpt3",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Path of the file to read\n",
        "iowa_file_path = './input/home-data-for-ml-course/train.csv'\n",
        "\n",
        "# Fill in the line below to read the file into a variable home_data\n",
        "home_data = ____"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "XWx-bb4ErvCw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Step 2: Review The Data\n",
        "Use the command you learned to view summary statistics of the data. Then fill in variables to answer the following questions\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "OxsadG5Ir4It",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Print summary statistics in next line\n",
        "____\n",
        "\n",
        "\"\"\"\n",
        "For these questions, you need some basic python \n",
        "knowledge, but it's okay, we're going to code it \n",
        "together if needed\n",
        "\"\"\"\n",
        "\n",
        "# What is the average lot size (rounded to nearest integer)?\n",
        "avg_lot_size = ____\n",
        "\n",
        "# As of today, how old is the newest home (current year - the date in which it was built)\n",
        "newest_home_age = ____"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PhEllawAwdEO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Think About Your Data\n",
        "\n",
        "The newest house in your data isn't that new.  A few potential explanations for this:\n",
        "1. They haven't built new houses where this data was collected.\n",
        "1. The data was collected a long time ago. Houses built after the data publication wouldn't show up.\n",
        "\n",
        "If the reason is explanation #1 above, does that affect your trust in the model you build with this data? What about if it is reason #2?\n",
        "\n",
        "How could you dig into the data to see which explanation is more plausible?\n",
        "\n",
        "Check out this **[discussion thread](https://www.kaggle.com/learn-forum/60581)** to see what others think or to add your ideas.\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "60Q4zvbew9X1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Selecting Data for Modeling\n",
        "\n",
        "Your dataset had too many variables to wrap your head around, or even to print out nicely. How can you pare down this overwhelming amount of data to something you can understand?\n",
        "\n",
        "We'll start by picking a few variables using our intuition. Later courses will show you statistical techniques to automatically prioritize variables.\n",
        "\n",
        "To choose variables/columns, we'll need to see a list of all columns in the dataset. That is done with the columns property of the DataFrame (the bottom line of code below)."
      ]
    },
    {
      "metadata": {
        "id": "Iu1r_whGxFA2",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "\n",
        "melbourne_file_path = './input/melbourne-housing-snapshot/melb_data.csv'\n",
        "melbourne_data = pd.read_csv(melbourne_file_path) \n",
        "melbourne_data.columns"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "_-i66bCyxQRz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# The Melbourne data has some missing values (some houses for which some variables weren't recorded.)\n",
        "# We'll learn to handle missing values in a later tutorial.  \n",
        "# Your Iowa data doesn't have missing values in the columns you use. \n",
        "# So we will take the simplest option for now, and drop houses from our data. \n",
        "# Don't worry about this much for now, though the code is:\n",
        "\n",
        "# dropna drops missing values (think of na as \"not available\")\n",
        "melbourne_data = melbourne_data.dropna(axis=0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ODMU6PowxWkb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "There are many ways to select a subset of your data. The Pandas Course covers these in more depth, but we will focus on two approaches for now.\n",
        "\n",
        "Dot notation, which we use to select the \"prediction target\"\n",
        "Selecting with a column list, which we use to select the \"features\"\n",
        "\n",
        "# Selecting The Prediction Target\n",
        "You can pull out a variable with ** dot-notation**. This single column is stored in a **Series**, which is broadly like a DataFrame with only a single column of data.\n",
        "\n",
        "We'll use the dot notation to select the column we want to predict, which is called the **prediction target**. By convention, the prediction target is called y. So the code we need to save the house prices in the Melbourne data is"
      ]
    },
    {
      "metadata": {
        "id": "W2ziLZSKxbfa",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "y = melbourne_data.Price"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "dc60dMpQxnxt",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Choosing \"Features\"\n",
        "The columns that are inputted into our model (and later used to make predictions) are called \"features.\" In our case, those would be the columns used to determine the home price. Sometimes, you will use all columns except the target as features. Other times you'll be better off with fewer features.\n",
        "\n",
        "For now, we'll build a model with only a few features. Later on you'll see how to iterate and compare models built with different features.\n",
        "\n",
        "We select multiple features by providing a list of column names inside brackets. Each item in that list should be a string (with quotes).\n",
        "\n",
        "Here is an example:"
      ]
    },
    {
      "metadata": {
        "id": "D0JSPApVxpgg",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "melbourne_features = ['Rooms', 'Bathroom', 'Landsize', 'Lattitude', 'Longtitude']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "C6AkY6Pyxrne",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "By convention, this data is called **X**."
      ]
    },
    {
      "metadata": {
        "id": "UFE4_-SbxtYP",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "X = melbourne_data[melbourne_features]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "hxw9ABhyxuul",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Let's quickly review the data we'll be using to predict house prices using the describe method and the head method, which shows the top few rows.\n"
      ]
    },
    {
      "metadata": {
        "id": "GgA6GtXJxxKH",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "X.describe()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "t6A1FwUVx0oJ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "X.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "yyXd-hhBx1zD",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Visually checking your data with these commands is an important part of a data scientist's job. You'll frequently find surprises in the dataset that deserve further inspection."
      ]
    },
    {
      "metadata": {
        "id": "eliSUW__x6GU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Building Your Model\n",
        "\n",
        "You will use the scikit-learn library to create your models. When coding, this library is written as sklearn, as you will see in the sample code. Scikit-learn is easily the most popular library for modeling the types of data typically stored in DataFrames.\n",
        "\n",
        "The steps to building and using a model are:\n",
        "\n",
        "* **Define**: What type of model will it be? A decision tree? Some other type of model? Some other parameters of the model type are specified too.\n",
        "* **Fit**: Capture patterns from provided data. This is the heart of modeling.\n",
        "* **Predict**: Just what it sounds like\n",
        "* **Evaluate**: Determine how accurate the model's predictions are.\n",
        "\n",
        "Here is an example of defining a decision tree model with scikit-learn and fitting it with the features and target variable."
      ]
    },
    {
      "metadata": {
        "id": "-d2hPBuzyJUi",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from sklearn.tree import DecisionTreeRegressor\n",
        "\n",
        "# Define model. Specify a number for random_state to ensure same results each run\n",
        "melbourne_model = DecisionTreeRegressor(random_state=1)\n",
        "\n",
        "# Fit model\n",
        "melbourne_model.fit(X, y)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "nnzgeufjyUO1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Many machine learning models allow some randomness in model training. Specifying a number for `random_state` ensures you get the same results in each run. This is considered a good practice. You use any number, and model quality won't depend meaningfully on exactly what value you choose.\n",
        "\n",
        "We now have a fitted model that we can use to make predictions.\n",
        "\n",
        "In practice, you'll want to make predictions for new houses coming on the market rather than the houses we already have prices for. But we'll make predictions for the first few rows of the training data to see how the predict function works."
      ]
    },
    {
      "metadata": {
        "id": "nqqMCpa1yetd",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "print(\"Making predictions for the following 5 houses:\")\n",
        "print(X.head())\n",
        "print(\"The predictions are\")\n",
        "print(melbourne_model.predict(X.head()))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "faVG1aAwywcI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Exercise 2\n",
        "So far, you have loaded your data and reviewed it with the following code. Run this cell to set up your coding environment where the previous step left off.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "V8mjlIiwy-Fr",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Code you have previously used to load data\n",
        "import pandas as pd\n",
        "\n",
        "# Path of the file to read\n",
        "iowa_file_path = './input/home-data-for-ml-course/train.csv'\n",
        "\n",
        "home_data = pd.read_csv(iowa_file_path)\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "1B3tHfBVzn1r",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Step 1: Specify Prediction Target\n",
        "Select the target variable, which corresponds to the sales price. Save this to a new variable called `y`. You'll need to print a list of the columns to find the name of the column you need.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "Ff6ROGRXzpjk",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# print the list of columns in the dataset to find the name of the prediction target\n",
        "home_data._____\n",
        "\n",
        "# y = ?\n",
        "y = home_data[_____]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "RApE8YiT0IfH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Step 2: Create X\n",
        "Now you will create a DataFrame called `X` holding the predictive features.\n",
        "\n",
        "Since you want only some columns from the original data, you'll first create a list with the names of the columns you want in `X`.\n",
        "\n",
        "You'll use just the following columns in the list (you can copy and paste the whole list to save some typing, though you'll still need to add quotes):\n",
        "    * LotArea\n",
        "    * YearBuilt\n",
        "    * 1stFlrSF\n",
        "    * 2ndFlrSF\n",
        "    * FullBath\n",
        "    * BedroomAbvGr\n",
        "    * TotRmsAbvGrd\n",
        "\n",
        "After you've created that list of features, use it to create the DataFrame that you'll use to fit the model."
      ]
    },
    {
      "metadata": {
        "id": "k5QLCJlu0NGq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Create the list of features below\n",
        "feature_names = ___\n",
        "\n",
        "# select data corresponding to features in feature_names\n",
        "X = _____"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "-6ZN44Sa0qik",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Review Data\n",
        "Before building a model, take a quick look at **X** to verify it looks sensible"
      ]
    },
    {
      "metadata": {
        "id": "AKU7uYNY0rTy",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# print description or statistics from X\n",
        "print(_)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "YLphFKnv2b8L",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Step 3: Specify and Fit Model\n",
        "Create a `DecisionTreeRegressor` and save it iowa_model. Ensure you've done the relevant import from sklearn to run this command.\n",
        "\n",
        "Then fit the model you just created using the data in `X` and `y` that you saved above."
      ]
    },
    {
      "metadata": {
        "id": "u6ARSP6u2c-S",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from _ import _\n",
        "#specify the model. \n",
        "#For model reproducibility, set a numeric value for random_state when specifying the model\n",
        "iowa_model = _\n",
        "\n",
        "# Fit the model\n",
        "_"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "H05u2PjX2qeq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Step 4: Make Predictions\n",
        "Make predictions with the model's `predict` command using `X` as the data. Save the results to a variable called `predictions`."
      ]
    },
    {
      "metadata": {
        "id": "wg1hBvHm2rr6",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "predictions = _\n",
        "print(predictions)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ilIh41RU2wKG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Think About Your Results\n",
        "\n",
        "Use the `head` method to compare the top few predictions to the actual home values (in `y`) for those same homes. Anything surprising?\n",
        "\n",
        "You'll understand why this happened if you keep going."
      ]
    },
    {
      "metadata": {
        "id": "PdbqyFdL2xix",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "You've built a model. But how good is it?\n",
        "\n",
        "In this lesson, you will learn to use model validation to measure the quality of your model. Measuring model quality is the key to iteratively improving your models.\n",
        "\n",
        "# What is Model Validation\n",
        "\n",
        "You'll want to evaluate almost every model you ever build. In most (though not all) applications, the relevant measure of model quality is predictive accuracy. In other words, will the model's predictions be close to what actually happens.\n",
        "\n",
        "Many people make a huge mistake when measuring predictive accuracy. They make predictions with their training data and compare those predictions to the target values in the training data. You'll see the problem with this approach and how to solve it in a moment, but let's think about how we'd do this first.\n",
        "\n",
        "You'd first need to summarize the model quality into an understandable way. If you compare predicted and actual home values for 10,000 houses, you'll likely find mix of good and bad predictions. Looking through a list of 10,000 predicted and actual values would be pointless. We need to summarize this into a single metric.\n",
        "\n",
        "There are many metrics for summarizing model quality, but we'll start with one called Mean Absolute Error (also called MAE). Let's break down this metric starting with the last word, error.\n",
        "\n",
        "The prediction error for each house is: \n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "error=actual−predicted\n",
        "```\n",
        "\n",
        "So, if a house cost 150,000 and you predicted it would cost 100,000 the error is 50,000.\n",
        "\n",
        "With the MAE metric, we take the absolute value of each error. This converts each error to a positive number. We then take the average of those absolute errors. This is our measure of model quality. In plain English, it can be said as\n",
        "\n",
        "\n",
        "```\n",
        "On average, our predictions are off by about X.\n",
        "```\n",
        "To calculate MAE, we first need a model. \n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "HjMJl-ZV4NUM",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Data Loading Code Hidden Here\n",
        "import pandas as pd\n",
        "\n",
        "# Load data\n",
        "melbourne_file_path = './input/melbourne-housing-snapshot/melb_data.csv'\n",
        "melbourne_data = pd.read_csv(melbourne_file_path) \n",
        "# Filter rows with missing price values\n",
        "filtered_melbourne_data = melbourne_data.dropna(axis=0)\n",
        "# Choose target and features\n",
        "y = filtered_melbourne_data.Price\n",
        "melbourne_features = ['Rooms', 'Bathroom', 'Landsize', 'BuildingArea', \n",
        "                        'YearBuilt', 'Lattitude', 'Longtitude']\n",
        "X = filtered_melbourne_data[melbourne_features]\n",
        "\n",
        "from sklearn.tree import DecisionTreeRegressor\n",
        "# Define model\n",
        "melbourne_model = DecisionTreeRegressor()\n",
        "# Fit model\n",
        "melbourne_model.fit(X, y)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0dWB0Fpw7Fp5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Once we have a model, here is how we calculate the mean absolute error:"
      ]
    },
    {
      "metadata": {
        "id": "76Kd9WV77GQz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import mean_absolute_error\n",
        "\n",
        "predicted_home_prices = melbourne_model.predict(X)\n",
        "mean_absolute_error(y, predicted_home_prices)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "nQwOOJ3U7Jpu",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# The Problem with \"In-Sample\" Scores\n",
        "The measure we just computed can be called an \"in-sample\" score. We used a single \"sample\" of houses for both building the model and evaluating it. Here's why this is bad.\n",
        "\n",
        "Imagine that, in the large real estate market, door color is unrelated to home price.\n",
        "\n",
        "However, in the sample of data you used to build the model, all homes with green doors were very expensive. The model's job is to find patterns that predict home prices, so it will see this pattern, and it will always predict high prices for homes with green doors.\n",
        "\n",
        "Since this pattern was derived from the training data, the model will appear accurate in the training data.\n",
        "\n",
        "But if this pattern doesn't hold when the model sees new data, the model would be very inaccurate when used in practice.\n",
        "\n",
        "Since models' practical value come from making predictions on new data, we measure performance on data that wasn't used to build the model. The most straightforward way to do this is to exclude some data from the model-building process, and then use those to test the model's accuracy on data it hasn't seen before. This data is called **validation data**.\n",
        "\n",
        "# Coding It\n",
        "The scikit-learn library has a function train_test_split to break up the data into two pieces. We'll use some of that data as training data to fit the model, and we'll use the other data as validation data to calculate mean_absolute_error.\n",
        "\n",
        "Here is the code:\n"
      ]
    },
    {
      "metadata": {
        "id": "PFtjd-0C7SwM",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "# split data into training and validation data, for both features and target\n",
        "# The split is based on a random number generator. Supplying a numeric value to\n",
        "# the random_state argument guarantees we get the same split every time we\n",
        "# run this script.\n",
        "train_X, val_X, train_y, val_y = train_test_split(X, y, random_state = 0)\n",
        "# Define model\n",
        "melbourne_model = DecisionTreeRegressor()\n",
        "# Fit model\n",
        "melbourne_model.fit(train_X, train_y)\n",
        "\n",
        "# get predicted prices on validation data\n",
        "val_predictions = melbourne_model.predict(val_X)\n",
        "print(mean_absolute_error(val_y, val_predictions))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "E6cS_fsM7WRt",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Wow!\n",
        "Your mean absolute error for the in-sample data was about 500 dollars. Out-of-sample it is more than 250,000 dollars.\n",
        "\n",
        "This is the difference between a model that is almost exactly right, and one that is unusable for most practical purposes. As a point of reference, the average home value in the validation data is 1.1 million dollars. So the error in new data is about a quarter of the average home value.\n",
        "\n",
        "There are many ways to improve this model, such as experimenting to find better features or different model types."
      ]
    },
    {
      "metadata": {
        "id": "Afq_NyzA7ebZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Exercise 3\n",
        "\n",
        "You've built a model. In this exercise you will test how good your model is.\n",
        "\n",
        "Run the cell below to set up your coding environment where the previous exercise left off."
      ]
    },
    {
      "metadata": {
        "id": "URH34-e47poc",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Code you have previously used to load data\n",
        "import pandas as pd\n",
        "from sklearn.tree import DecisionTreeRegressor\n",
        "\n",
        "# Path of the file to read\n",
        "iowa_file_path = './input/home-data-for-ml-course/train.csv'\n",
        "\n",
        "home_data = pd.read_csv(iowa_file_path)\n",
        "y = home_data.SalePrice\n",
        "feature_columns = ['LotArea', 'YearBuilt', '1stFlrSF', '2ndFlrSF', 'FullBath', 'BedroomAbvGr', 'TotRmsAbvGrd']\n",
        "X = home_data[feature_columns]\n",
        "\n",
        "# Specify Model\n",
        "iowa_model = DecisionTreeRegressor()\n",
        "# Fit Model\n",
        "iowa_model.fit(X, y)\n",
        "\n",
        "print(\"First in-sample predictions:\", iowa_model.predict(X.head()))\n",
        "print(\"Actual target values for those homes:\", y.head().tolist())\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "drrP0psg7rxN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Step 1: Split Your Data\n",
        "Use the `train_test_split` function to split up your data.\n",
        "\n",
        "Give it the argument `random_state=1` so the `check` functions know what to expect when verifying your code.\n",
        "\n",
        "Recall, your features are loaded in the DataFrame **X** and your target is loaded in **y**.\n"
      ]
    },
    {
      "metadata": {
        "id": "iR19MJWT7xB9",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Import the train_test_split function \n",
        "from _ import _\n",
        "\n",
        "# fill in \n",
        "train_X, val_X, train_y, val_y = ____"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "tnfO6Df872Ft",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Step 2: Specify and Fit the Model\n",
        "\n",
        "Create a `DecisionTreeRegressor` model and fit it to the relevant data.\n",
        "Set `random_state` to 1 again when creating the model."
      ]
    },
    {
      "metadata": {
        "id": "QkYHFg8974mw",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# You imported DecisionTreeRegressor in your last exercise\n",
        "# and that code has been copied to the setup code above. So, no need to\n",
        "# import it again\n",
        "\n",
        "# Specify the model\n",
        "iowa_model = ____\n",
        "\n",
        "# Fit iowa_model with the training data.\n",
        "____"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "lTxjoqWF76uo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Step 3: Make Predictions with Validation data\n"
      ]
    },
    {
      "metadata": {
        "id": "ry-Jztxg78t8",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Predict with all validation observations\n",
        "val_predictions = ____"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Dk2va0dz7-b4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Inspect your predictions and actual values from validation data."
      ]
    },
    {
      "metadata": {
        "id": "4P2wbDLH7--l",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# print the top few validation predictions\n",
        "print(____)\n",
        "# print the top few actual prices from validation data\n",
        "print(____)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ligeodPH8B5K",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "What do you notice that is different from what you saw with in-sample predictions (which are printed after the top code cell in this page).\n",
        "\n",
        "Do you remember why validation predictions differ from in-sample (or training) predictions? This is an important idea from the last lesson.\n",
        "\n",
        "## Step 4: Calculate the Mean Absolute Error in Validation Data\n"
      ]
    },
    {
      "metadata": {
        "id": "PPZ15CRn8GAK",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import mean_absolute_error\n",
        "val_mae = ____\n",
        "print(val_mae)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "D9QoFZ9z8Jpn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Is that MAE good?  There isn't a general rule for what values are good that applies across applications. But you'll see how to use (and improve) this number in the next step.\n"
      ]
    },
    {
      "metadata": {
        "id": "nCRot6GX8MTw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "At the end of this step, you will understand the concepts of underfitting and overfitting, and you will be able to apply these ideas to make your models more accurate.\n",
        "\n",
        "# Experimenting With Different Models\n",
        "Now that you have a reliable way to measure model accuracy, you can experiment with alternative models and see which gives the best predictions. But what alternatives do you have for models?\n",
        "\n",
        "You can see in scikit-learn's [documentation](http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html) that the decision tree model has many options (more than you'll want or need for a long time). The most important options determine the tree's depth. Recall from the first lesson that a tree's depth is a measure of how many splits it makes before coming to a prediction. This is a relatively shallow tree\n",
        "![shallow tree](http://i.imgur.com/R3ywQsR.png)\n",
        "In practice, it's not uncommon for a tree to have 10 splits between the top level (all houses) and a leaf. As the tree gets deeper, the dataset gets sliced up into leaves with fewer houses. If a tree only had 1 split, it divides the data into 2 groups. If each group is split again, we would get 4 groups of houses. Splitting each of those again would create 8 groups. If we keep doubling the number of groups by adding more splits at each level, we'll have  2^10  groups of houses by the time we get to the 10th level. That's 1024 leaves.\n",
        "\n",
        "When we divide the houses amongst many leaves, we also have fewer houses in each leaf. Leaves with very few houses will make predictions that are quite close to those homes' actual values, but they may make very unreliable predictions for new data (because each prediction is based on only a few houses).\n",
        "\n",
        "This is a phenomenon called **overfitting**, where a model matches the training data almost perfectly, but does poorly in validation and other new data. On the flip side, if we make our tree very shallow, it doesn't divide up the houses into very distinct groups.\n",
        "\n",
        "At an extreme, if a tree divides houses into only 2 or 4, each group still has a wide variety of houses. Resulting predictions may be far off for most houses, even in the training data (and it will be bad in validation too for the same reason). When a model fails to capture important distinctions and patterns in the data, so it performs poorly even in training data, that is called **underfitting**.\n",
        "\n",
        "Since we care about accuracy on new data, which we estimate from our validation data, we want to find the sweet spot between underfitting and overfitting. Visually, we want the low point of the (red) validation curve in\n",
        "![validation curve](http://i.imgur.com/2q85n9s.png)\n",
        "\n",
        "## Example\n",
        "There are a few alternatives for controlling the tree depth, and many allow for some routes through the tree to have greater depth than other routes. But the max_leaf_nodes argument provides a very sensible way to control overfitting vs underfitting. The more leaves we allow the model to make, the more we move from the underfitting area in the above graph to the overfitting area.\n",
        "\n",
        "We can use a utility function to help compare MAE scores from different values for max_leaf_nodes:"
      ]
    },
    {
      "metadata": {
        "id": "GshHKhqe9ECg",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import mean_absolute_error\n",
        "from sklearn.tree import DecisionTreeRegressor\n",
        "\n",
        "def get_mae(max_leaf_nodes, train_X, val_X, train_y, val_y):\n",
        "    model = DecisionTreeRegressor(max_leaf_nodes=max_leaf_nodes, random_state=0)\n",
        "    model.fit(train_X, train_y)\n",
        "    preds_val = model.predict(val_X)\n",
        "    mae = mean_absolute_error(val_y, preds_val)\n",
        "    return(mae)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jPr-JwhW9F0G",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The data is loaded into train_X, val_X, train_y and val_y using the code you've already seen (and which you've already written)\n",
        "\n",
        "We can use a for-loop to compare the accuracy of models built with different values for max_leaf_nodes."
      ]
    },
    {
      "metadata": {
        "id": "CudAMI9T9KpB",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# compare MAE with differing values of max_leaf_nodes\n",
        "for max_leaf_nodes in [5, 50, 500, 5000]:\n",
        "    my_mae = get_mae(max_leaf_nodes, train_X, val_X, train_y, val_y)\n",
        "    print(\"Max leaf nodes: %d  \\t\\t Mean Absolute Error:  %d\" %(max_leaf_nodes, my_mae))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pB9qTFrC9M3Q",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Of the options listed, 500 is the optimal number of leaves.\n",
        "\n",
        "---\n",
        "\n",
        "# Conclusion\n",
        "Here's the takeaway: Models can suffer from either:\n",
        "\n",
        "* **Overfitting**: capturing spurious patterns that won't recur in the future, leading to less accurate predictions, or\n",
        "* **Underfitting**: failing to capture relevant patterns, again leading to less accurate predictions.\n",
        "\n",
        "We use validation data, which isn't used in model training, to measure a candidate model's accuracy. This lets us try many candidate models and keep the best one.\n",
        "\n",
        "Credits to [Kaggle's machine learning course](https://www.kaggle.com/learn/machine-learning)\n",
        "\n",
        "I hope you enjoyed it, feel free to contact me if needed at : ea_remache@esi.dz"
      ]
    }
  ]
}