Ex2.Reducing Traffic Mortality in the USA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/ML-Challenge/week4-unsupervised-learning/blob/master/Ex2.Reducing%20Traffic%20Mortality%20in%20the%20USA.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T08:26:33.163118Z",
     "start_time": "2020-02-15T08:26:33.150125Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Download lesson datasets\n",
    "# Required if you're using Google Colab\n",
    "!wget \"https://github.com/ML-Challenge/week4-unsupervised-learning/raw/master/datasets.zip\"\n",
    "!unzip -o datasets.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Project Description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the rate of fatal road accidents has been decreasing steadily since the 80s, the past ten years have seen a stagnation in this reduction. Coupled with the increase in number of miles driven in the nation, the total number of traffic related-fatalities has now reached a ten year high and is rapidly increasing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By looking at the demographics of traffic accident victims for each US state, we find that there is a lot of variation between states. Now we want to understand if there are patterns in this variation in order to derive suggestions for a policy action plan. In particular, instead of implementing a costly nation-wide plan we want to focus on groups of states with similar profiles. How can we find such groups in a statistically sound way and communicate the result effectively?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The raw data files and their format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "3"
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "<p><img src=\"assets/car-accident.jpg\" alt=\"\"></p>\n",
    "<p>While the rate of fatal road accidents has been decreasing steadily since the 80s, the past ten years have seen a stagnation in this reduction. Coupled with the increase in number of miles driven in the nation, the total number of traffic related-fatalities has now reached a ten year high and is rapidly increasing.</p>\n",
    "<p>Per request of the US Department of Transportation, we are currently investigating how to derive a strategy to reduce the incidence of road accidents across the nation. By looking at the demographics of traﬃc accident victims for each US state, we find that there is a lot of variation between states. Now we want to understand if there are patterns in this variation in order to derive suggestions for a policy action plan. In particular, instead of implementing a costly nation-wide plan we want to focus on groups of  states with similar profiles. How can we find such groups in a statistically sound way and communicate the result effectively?</p>\n",
    "<p>To accomplish these tasks, we will make use of data wrangling, plotting, dimensionality reduction, and unsupervised clustering.</p>\n",
    "<p>The data given to us was originally collected by the National Highway Traffic Safety Administration and the National Association of Insurance Commissioners. This particular dataset was compiled and released as a <a href=\"https://github.com/fivethirtyeight/data/tree/master/bad-drivers\">CSV-file</a> by FiveThirtyEight under the <a href=\"https://github.com/ﬁvethirtyeight/data\">CC-BY4.0 license</a>.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 1: Instructions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Explore your current folder and view the main dataset file:\n",
    "\n",
    "* Check the name of the current folder.\n",
    "* List all files in this folder.\n",
    "* List all files in the `data\\` folder.\n",
    "* View the first 20 lines of `road-accidents.csv` in the data\\ folder.\n",
    "\n",
    "Helpful links:\n",
    "* [How to run shell commands in a Jupyter Notebook](https://jakevdp.github.io/PythonDataScienceHandbook/01.05-ipython-and-shell-commands.html) (through the underlying IPython interpreter)\n",
    "* For viewing the first few lines of a file using shell commands, see [this](https://kb.iu.edu/d/acri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "3"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# Check the name of the current folder\n",
    "current_dir = ...\n",
    "print(current_dir)\n",
    "\n",
    "# List all files in this folder\n",
    "file_list = ...\n",
    "print(file_list)\n",
    "\n",
    "# List all files in the data directory\n",
    "dataset_list = ...\n",
    "print(dataset_list)\n",
    "\n",
    "# View the first 20 lines of data/road-accidents.csv\n",
    "accidents_head = ...\n",
    "accidents_head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read in and get an overview of the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "10"
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "<p>Next, we will orient ourselves to get to know the data with which we are dealing.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2: Instructions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read in the main dataset file and start exploring the data.\n",
    "\n",
    "* Read in `road-accidents.csv` (which is in the data/ folder) using read_csv() from pandas. Set the `comment` and `sep` parameters based on the output from task 1.\n",
    "* Save the number of rows columns as a tuple.\n",
    "* Generate an overview of the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "10"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# Import the `pandas` module as \"pd\"\n",
    "# ... YOUR CODE FOR TASK 2 ...\n",
    "\n",
    "# Read in `road-accidents.csv`\n",
    "car_acc = ...\n",
    "\n",
    "# Save the number of rows columns as a tuple\n",
    "rows_and_cols = ...\n",
    "print('There are {} rows and {} columns.\\n'.format(\n",
    "    rows_and_cols[0], rows_and_cols[1]))\n",
    "\n",
    "# Generate an overview of the DataFrame\n",
    "car_acc_information = ...\n",
    "print(car_acc_information)\n",
    "\n",
    "# Display the last five rows of the DataFrame\n",
    "car_acc.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a textual and a graphical summary of the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "17"
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "<p>We now have an idea of what the dataset looks like. To further familiarize ourselves with this data, we will calculate summary statistics and produce a graphical overview of the data. The graphical overview is good to get a sense for the distribution of variables within the data and could consist of one histogram per column. It is often a good idea to also explore the pairwise relationship between all columns in the data set by using a using pairwise scatter plots (sometimes referred to as a \"scatterplot matrix\").</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 3: Instructions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a textual and graphical overview of the data.\n",
    "\n",
    "* Compute the summary statistics of all columns in the `car_acc` DataFrame.\n",
    "* Create a pairwise scatter plot to explore the data, using `sns.pairplot()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "17"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# import seaborn and make plots appear inline\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "\n",
    "# Compute the summary statistics of all columns in the `car_acc` DataFrame\n",
    "sum_stat_car = ...\n",
    "print(sum_stat_car)\n",
    "\n",
    "# Create a pairwise scatter plot to explore the data\n",
    "# ... YOUR CODE FOR TASK 3 ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quantify the association of features and accidents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "24"
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "<p>We can already see some potentially interesting relationships between the target variable (the number of fatal accidents) and the feature variables (the remaining three columns).</p>\n",
    "<p>To quantify the pairwise relationships that we observed in the scatter plots, we can compute the Pearson correlation coefficient matrix. The Pearson correlation coefficient is one of the most common methods to quantify correlation between variables, and by convention, the following thresholds are usually used:</p>\n",
    "<ul>\n",
    "<li>0.2 = weak</li>\n",
    "<li>0.5 = medium</li>\n",
    "<li>0.8 = strong</li>\n",
    "<li>0.9 = very strong</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 4: Instructions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Explore the correlation between all column pairs in the DataFrame.\n",
    "\n",
    "* Compute the correlation coefficient for all column pairs in car_acc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "24"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# Compute the correlation coefficent for all column pairs\n",
    "corr_columns = ...\n",
    "corr_columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit a multivariate linear regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "31"
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "<p>From the correlation table, we see that the amount of fatal accidents is most strongly correlated with alcohol consumption (first row). But in addition, we also see that some of the features are correlated with each other, for instance, speeding and alcohol consumption are positively correlated. We, therefore, want to compute the association of the target with each feature while adjusting for the effect of the remaining features. This can be done using multivariate linear regression.</p>\n",
    "<p>Both the multivariate regression and the correlation measure how strongly the features are associated with the outcome (fatal accidents). When comparing the regression coefficients with the correlation coefficients, we will see that they are slightly different. The reason for this is that the multiple regression computes the association of a feature with an outcome, given the association with all other features, which is not accounted for when calculating the correlation coefficients.</p>\n",
    "<p>A particularly interesting case is when the correlation coefficient and the regression coefficient of the same feature have opposite signs. How can this be? For example, when a feature A is positively correlated with the outcome Y but also positively correlated with a different feature B that has a negative effect on Y, then the indirect correlation (A-&gt;B-&gt;Y) can overwhelm the direct correlation (A-&gt;Y). In such a case, the regression coefficient of feature A could be positive, while the correlation coefficient is negative. This is sometimes called a <em>masking</em> relationship. Let’s see if the multivariate regression can reveal such a phenomenon.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 5: Instructions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit a multivariate linear regression model using the fatal accident rate as the outcome.\n",
    "\n",
    "* Import the `linear_model` function from sklearn.\n",
    "* Create the features and target DataFrames, by subsetting the DataFrame `car_acc`.\n",
    "* Create a linear regression object.\n",
    "* Fit a multivariate linear regression model.\n",
    "* Retrieve the regression coefficients from the fitted regression object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "31"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# Import the linear model function from sklearn\n",
    "from sklearn import ...\n",
    "\n",
    "# Create the features and target DataFrames\n",
    "features = ...\n",
    "target = ...\n",
    "\n",
    "# Create a linear regression object\n",
    "reg = ...\n",
    "\n",
    "# Fit a multivariate linear regression model\n",
    "# ... YOUR CODE FOR TASK 5 ...\n",
    "\n",
    "# Retrieve the regression coefficients\n",
    "fit_coef = ...\n",
    "fit_coef"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perform PCA on standardized data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "38"
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "<p>We have learned that alcohol consumption is weakly associated with the number of fatal accidents across states. This could lead us to conclude that alcohol consumption should be a focus for further investigations and maybe strategies should divide states into high versus low alcohol consumption in accidents. But there are also associations between  alcohol consumptions and the other two features, so it might be worth trying to split the states in a way that accounts for all three features.</p>\n",
    "<p>One way of clustering the data is to use PCA to visualize data in reduced dimensional space where we can try to pick up patterns by eye. PCA uses the absolute variance to calculate the overall variance explained for each principal component, so it is important that the features are on a similar scale (unless we would have a particular reason that one feature should be weighted more).</p>\n",
    "<p>We'll use the appropriate scaling function to standardize the features to be centered with mean 0 and scaled with standard deviation 1.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 6: Instructions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perform a principal component analysis on the standardized data.\n",
    "\n",
    "* Standardize and center the feature columns, using the `StandardScaler`\n",
    "* Import the PCA class.\n",
    "* Fit the standardized data to the PCA class.\n",
    "* Compute the cumulative proportion of variance explained by the first two principal components, either by adding them together or by using the cumulative summation method (`cumsum`) of the explained variance array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "38"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# Standardize and center the feature columns\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "scaler = StandardScaler()\n",
    "features_scaled = ...\n",
    "\n",
    "# Import the PCA class function from sklearn\n",
    "# ... YOUR CODE FOR TASK 6 ...\n",
    "pca = PCA()\n",
    "\n",
    "# Fit the standardized data to the pca\n",
    "# ... YOUR CODE FOR TASK 6 ...\n",
    "\n",
    "# Plot the proportion of variance explained on the y-axis of the bar plot\n",
    "import matplotlib.pyplot as plt\n",
    "plt.bar(range(1, pca.n_components_ + 1),  pca.explained_variance_ratio_)\n",
    "plt.xlabel('Principal component #')\n",
    "plt.ylabel('Proportion of variance explained')\n",
    "plt.xticks([1, 2, 3])\n",
    "\n",
    "# Compute the cumulative proportion of variance explained by the first two principal components\n",
    "two_first_comp_var_exp = ...\n",
    "print(\"The cumulative variance of the first two principal components is {}\".format(\n",
    "    round(two_first_comp_var_exp, 5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize the first two principal components"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "45"
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "<p>The first two principal components enable visualization of the data in two dimensions while capturing a high proportion of the variation (79%) from all three features: speeding, alcohol influence, and first-time accidents. This enables us to use our eyes to try to discern patterns in the data with the goal to find groups of similar states. Although clustering algorithms are becoming increasingly efficient, human pattern recognition is an easily accessible and very efficient method of assessing patterns in data.</p>\n",
    "<p>We will create a scatter plot of the first principle components and explore how the states cluster together in this visualization.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 7: Instructions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transform the data and visualize the first two principal components in a scatter plot.\n",
    "\n",
    "* Create a PCA object with two components. Assign the result to the variable, pca.\n",
    "* Transform the scaled features using two principal components.\n",
    "* Extract the first and second component to use for the scatter plot. Assign the results to p_comp1 and p_comp2, respectively.\n",
    "* Plot the first two principal components in a scatter plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "45"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# Transform the scaled features using two principal components\n",
    "pca = ...\n",
    "p_comps = ...\n",
    "\n",
    "# Extract the first and second component to use for the scatter plot\n",
    "p_comp1 = p_comps[...]\n",
    "p_comp2 = p_comps[...]\n",
    "\n",
    "# Plot the first two principal components in a scatter plot\n",
    "# ... YOUR CODE FOR TASK 7 ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find clusters of similar states in the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "52"
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "<p>It was not entirely clear from the PCA scatter plot how many groups in which the states cluster. To assist with identifying a reasonable number of clusters, we can use KMeans clustering by creating a scree plot and finding the \"elbow\", which is an indication of when the addition of more clusters does not add much explanatory power.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 8: Instructions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cluster the states using the KMeans algorithm and visualize the explanatory power for different numbers of clusters.\n",
    "\n",
    "* Import `KMeans`.\n",
    "* Initialize the `KMeans` object using the current number of clusters (k).\n",
    "* Fit the scaled features to the `KMeans` object.\n",
    "* Append the inertia for `km` to the list of inertias.\n",
    "* Plot the results in a line plot. This type of plot is also called a scree plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "52"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# Import KMeans from sklearn\n",
    "# ... YOUR CODE FOR TASK 8 ...\n",
    "\n",
    "# A loop will be used to plot the explanatory power for up to 10 KMeans clusters\n",
    "ks = range(1, 10)\n",
    "inertias = []\n",
    "for k in ks:\n",
    "    # Initialize the KMeans object using the current number of clusters (k)\n",
    "    km = KMeans(n_clusters=..., random_state=8)\n",
    "    # Fit the scaled features to the KMeans object\n",
    "    km.fit(...)\n",
    "    # Append the inertia for `km` to the list of inertias\n",
    "    inertias.append(...)\n",
    "    \n",
    "# Plot the results in a line plot\n",
    "plt.plot(..., ..., marker='o')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## KMeans to visualize clusters in the PCA scatter plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "59"
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "<p>Since there wasn't a clear elbow in the scree plot, assigning the states to either two or three clusters is a reasonable choice, and we will resume our analysis using three clusters. Let's see how the PCA scatter plot looks if we color the states according to the cluster to which they are assigned.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Task 9: Instructions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Highlight the clusters of the K-means fit with three clusters in the PCA scatter plot.\n",
    "\n",
    "* Create a KMeans object with k (your choice) clusters, setting random_state to 8 as in the previous task.\n",
    "* Fit the data to the `km` object.\n",
    "* Create a scatter plot of the first two principal components and color it according to the KMeans cluster assignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "59"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# Create a KMeans object with 3 clusters, use random_state=8 \n",
    "km = ...\n",
    "\n",
    "# Fit the data to the `km` object\n",
    "#  ... YOUR CODE FOR TASK 9 ...\n",
    "\n",
    "# Create a scatter plot of the first two principal components\n",
    "# and color it according to the KMeans cluster assignment \n",
    "# ... YOUR CODE FOR TASK 9 ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize the feature differences between the clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "66"
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "<p>Thus far, we have used both our visual interpretation of the data and the KMeans clustering algorithm to reveal patterns in the data, but what do these patterns mean?</p>\n",
    "<p>Remember that the information we have used to cluster the states into three distinct groups are the percentage of drivers speeding, under alcohol influence and that has not previously been involved in an accident. We used these clusters to visualize how the states group together when considering the first two principal components. This is good for us to understand structure in the data, but not always easy to understand, especially not if the findings are to be communicated to a non-specialist audience.</p>\n",
    "<p>A reasonable next step in our analysis is to explore how the three clusters are different in terms of the three features that we used for clustering. Instead of using the scaled features, we return to using the unscaled features to help us interpret the differences.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 10: Instructions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize the distribution of speeding, alcohol influence and percentage of first-time accidents in a direct comparison of the clusters.\n",
    "\n",
    "* Create a new column with the labels from the KMeans clustering, using `km.labels_`.\n",
    "* Reshape the DataFrame to the long format, using `pd.melt()`. Use the features as the value variables and give them the name `'measurement'` in the new DataFrame. Name the value column `'percent'`.\n",
    "* Create a violin plot splitting and coloring the results according to the km-clusters using the `hue` parameter. Plot the measurements along the y-axis and the percent values along the x-axis.\n",
    "\n",
    "Helpful links:\n",
    "\n",
    "* [Pandas.melt()](https://www.geeksforgeeks.org/python-pandas-melt/)\n",
    "* [seaborn violin plots](https://seaborn.pydata.org/generated/seaborn.violinplot.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "66"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# Create a new column with the labels from the KMeans clustering\n",
    "car_acc['cluster'] = ...\n",
    "\n",
    "# Reshape the DataFrame to the long format\n",
    "melt_car = pd.melt(...)\n",
    "\n",
    "# Create a violin plot splitting and coloring the results according to the km-clusters\n",
    "sns.violinplot(...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute the number of accidents within each cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "73"
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "<p>Now it is clear that different groups of states may require different interventions. Since resources and time are limited, it is useful to start off with an intervention in one of the three groups first. Which group would this be? To determine this, we will include data on how many miles are driven in each state, because this will help us to compute the total number of fatal accidents in each state. Data on miles driven is available in another tab-delimited text file. We will assign this new information to a column in the DataFrame and create a violin plot for how many total fatal traffic accidents there are within each state cluster.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-20T16:04:10.120619Z",
     "start_time": "2020-02-20T16:04:10.108649Z"
    }
   },
   "source": [
    "### Task 11: Instructions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add data on the number of miles driven per state to compute total number of fatal accidents and total accidents for each cluster.\n",
    "\n",
    "* Merge the `car_acc` DataFrame with the `miles_driven` DataFrame. Merge on the common column state.\n",
    "* Create a new column for the number of drivers involved in fatal accidents. Use the columns `'drvr_fatl_col_bmiles'` and `'million_miles_annually'`, note that these are in billions and million respectively.\n",
    "* Calculate the number of states in each cluster, their average and total number of drivers involved in fatal accidents.\n",
    "* Using `sns.barplot`, create a bar plot of the total number of fatal accidents per cluster, setting the `estimator` parameter to `sum`.\n",
    "\n",
    "Helpful links:\n",
    "\n",
    "* [Pandas merging 101](https://stackoverflow.com/questions/53645882/pandas-merging-101)\n",
    "* [Summarising, Aggregating, and Grouping data in Python Pandas](https://www.shanelynn.ie/summarising-aggregation-and-grouping-data-in-python-pandas/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "73"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# Read in the new dataset\n",
    "miles_driven = pd.read_csv('data/miles-driven.csv', sep='|')\n",
    "\n",
    "# Merge the `car_acc` DataFrame with the `miles_driven` DataFrame\n",
    "car_acc_miles = carr_acc.merge(...)\n",
    "\n",
    "# Create a new column for the number of drivers involved in fatal accidents\n",
    "car_acc_miles['num_drvr_fatl_col'] = ...\n",
    "\n",
    "# Create a barplot of the total number of accidents per cluster\n",
    "sns.barplot(x=..., y=..., data=..., estimator=..., ci=None)\n",
    "\n",
    "# Calculate the number of states in each cluster and their 'num_drvr_fatl_col' mean and sum.\n",
    "count_mean_sum = ...\n",
    "count_mean_sum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make a decision when there is no clear right choice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "dc": {
     "key": "80"
    },
    "tags": [
     "context"
    ]
   },
   "source": [
    "<p>As we can see, there is no obvious correct choice regarding which cluster is the most important to focus on. Yet, we can still argue for a certain cluster and motivate this using our findings above. Which cluster do you think should be a focus for policy intervention and further investigation?</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Decide which cluster to focus your resources on.\n",
    "\n",
    "* Which cluster would you choose: 1, 2, or 3? Assign one of these integers to `cluster_num`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "80"
    },
    "tags": [
     "sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# Which cluster would you choose?\n",
    "cluster_num = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**[Week 4 - Unsupervised Learning](https://radu-enuca.gitbook.io/ml-challenge/unsupervised-learning)**\n",
    "\n",
    "*Have questions or comments? Visit the ML Challenge Mattermost Channel.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}