01 Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Introduction to Data Analysis with Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- [Getting the data into Python](#Getting-the-data-into-Python)\n",
    "    -  using `read_csv` and dealing with missing data\n",
    "- [Accessing columns](#Accessing-the-columns)\n",
    "    -  using dot notation and square brackets\n",
    "    -  setting the index\n",
    "    -  using `loc`\n",
    "- [Sorting and filtering](#Sorting-and-filtering)\n",
    "    -  the `sort_values` function\n",
    "    -  how to get documentation\n",
    "    -  default arguments\n",
    "    -  passing a Boolean to `loc[]`\n",
    "    -  compound filters\n",
    "- [Summary statistics](#Summary-statistics)\n",
    "    -  not so useful for this data set but good to know\n",
    "- [Investigating relationships](#Investigating-relationships)\n",
    "    -  drawing scatter plots in `pandas`\n",
    "    -  drawing better scatter plots in `seaborn`\n",
    "    -  getting the correlation coefficient\n",
    "- [Time series](#Time-Series)\n",
    "    -  plotting simple time series\n",
    "    -  applying a calculation and creating new columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# We tend to abbreviate the pandas library as pd\n",
    "import pandas as pd\n",
    "# Stop pandas from abbreviating tables to fit in the notebook\n",
    "pd.options.display.max_columns = 1000\n",
    "pd.options.display.max_rows = 1000\n",
    "# Display graphs in the notebook\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Getting the data into Python\n",
    "\n",
    "The `pandas` library stores data in what it calls a *dataframe*, which is really just a smart table.\n",
    "\n",
    "We use the `read_csv` function to read in data from a csv file. In this case it's data about London Boroughs.\n",
    "\n",
    "Don't forget to run each cell when you get to it with either `ctrl`+`enter` or `shift`+`enter`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# read in our csv file, and automatically change missing values (a dot in the csv) into NaN\n",
    "#boroughs = pd.read_csv('boroughs.csv', na_values = ['.',' '], thousands=',')\n",
    "boroughs = pd.read_csv('https://raw.githubusercontent.com/adaapp/dav-introductionToPandas/master/boroughs.csv', na_values = ['.',' '], thousands=',')\n",
    "# Use the head function to see the first few rows\n",
    "#boroughs.head(5)\n",
    "boroughs.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Q1\n",
    "\n",
    "> What do you think `NaN` stands for?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Accessing the columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "A single column of the data is accessible using Python dot notation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "boroughs.Anxiety"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Or we can use square brackets, a bit like with a Python list or dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "boroughs['Population']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Q2\n",
    "\n",
    "> Try out both ways of accessing columns.\n",
    ">\n",
    "> This isn't as helpful as it could be. Why not?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Square brackets are more flexible. We can give them a list of headings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# note the nested brackets\n",
    "boroughs[['Borough','Population','Happy']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "This is better. But it would be nice if we didn't have to keep including the `Borough` column. So let's make that our *index*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "boroughs = boroughs.set_index(boroughs.Borough)\n",
    "boroughs.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Q3 \n",
    "\n",
    "> What changed?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Now, when we ask for column, we'll get the borough for free"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "boroughs[['Age','WorkAge']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Now we can also use the `loc` function (which uses square brackets, too) to *filter* the data and *locate* the index Haringey."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "boroughs.loc['Haringey']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Q4\n",
    "\n",
    "> Pick another borough to retreive the data for. Compare it to Haringey."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "boroughs.loc[['Haringey','Hackney']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Sorting and filtering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Let's find out which boroughs have the highest population.\n",
    "\n",
    "`pandas` dataframes have a `sort_values` function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Q5\n",
    "\n",
    "Remember in a jupyter notebook, you can put the cursor in the function brackets and hit `shift`+`tab` to bring up documentation for that function.\n",
    "\n",
    "> Make the sort_values function below work, to put the boroughs in order of population\n",
    ">\n",
    "> Now put them in *descending* order\n",
    ">\n",
    "> Which borough has the largest population?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# *** broken ***\n",
    "boroughs.sort_values()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "What if we wanted to only include **innerLondon** boroughs?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "boroughs.loc[boroughs[\"InnerOuter\"]=='Inner London']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "So we can pass a Boolean into those square brackets to *filter* the data. `pandas` square brackets are clearly a bit more powerful than regular Python square brackets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Q6\n",
    "\n",
    "> Filter the data to show only Outer London boroughs\n",
    ">\n",
    "> Apply `sort_values` to give the Outer London boroughs in descending order of population"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "boroughs.loc[boroughs[\"InnerOuter\"]==\"Outer London\"].sort_values(\"Population\")[[\"Area\",\"Age\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "If you want to combine two Booleans into one filter you'll need to put both into parentheses *for reasons*. For example,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "boroughs.loc[(boroughs.InnerOuter==\"Inner London\") | (boroughs.InnerOuter==\"Outer London\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "It might be useful to come back to this table of *just* the individual boroughs, so let's assign that to a variable `justBoroughs`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "justBoroughs = boroughs.loc[(boroughs.InnerOuter==\"Inner London\") | (boroughs.InnerOuter==\"Outer London\")]\n",
    "justBoroughs.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "### Note\n",
    "\n",
    "There is a subtle catch here that is worth thinking about when you're trying to do more advanced stuff with `pandas`.\n",
    "\n",
    "`boroughs[]` and `boroughs.loc[]` can appear to do the same thing, but they don't. In general it is better to use `loc`.\n",
    "\n",
    "See [this article](https://www.dataquest.io/blog/settingwithcopywarning/) later if you want more details. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Summary statistics\n",
    "\n",
    "The dataframe has built in functions for statistical measures like `mean`, `std`, `quantile` but you need to be careful whether using them makes sense."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# you can give loc a row label and a column label\n",
    "boroughs.loc['London','Age']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "justBoroughs['Age'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# you can give loc a row label and a column label\n",
    "boroughs.loc['London','Age']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Q7\n",
    "\n",
    "> Why is the mean of the average ages not the same as the London average age?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "So use the Inner London, Outer London and London averages from the main table rather than applying `mean` to a column."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Investigating relationships"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "We would expect there to be an obvious relationship between unemployment rates and employment rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "justBoroughs.plot.scatter(\"Employ\", \"Unemploy\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let's quantify that by asking for the correlation coefficient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "justBoroughs.Employ.corr(justBoroughs.Unemploy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Q8\n",
    "\n",
    "> How would you interpret this?\n",
    ">\n",
    "> What *correlation coefficient* is it using?\n",
    ">\n",
    "> Why isn't it a perfect correlation?\n",
    ">\n",
    "> Look for correlation between some other pairs of variables. Use a scatter plot first, then get the correlation coefficient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The `seaborn` library has some nice options for scatter plots, so let's import that and then see an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# pyplot is the grandparent of all python plotting packages\n",
    "import matplotlib.pyplot as plt\n",
    "# seaborn is based on pyplot but makes it easier to use\n",
    "import seaborn as sns\n",
    "# I don't know why we abbreviate seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Now an example,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# by default seaborn plots come out a bit small, so make ours 8in by 8in\n",
    "plt.figure(figsize=(8,8))\n",
    "# sns.scatterplot has options for controlling colour and dot size so we can use four variables on one graph\n",
    "sns.scatterplot(data=justBoroughs.loc[justBoroughs.Borough != \"City of London\"],\n",
    "                x=\"Employ\",\n",
    "                y=\"Medianpay\",\n",
    "                hue=\"Conservative\",\n",
    "                palette=\"RdBu\")\n",
    "#plt.axvline(justBoroughs.Employ.mean(), linestyle=\"--\", alpha=0.6)\n",
    "#plt.axhline(justBoroughs.Unemploy.mean(), linestyle=\"--\", alpha=0.6)\n",
    "plt.title(\"My beautiful scatter plot\")\n",
    "# where to put the legend\n",
    "plt.legend(loc='upper right');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "boroughs[\"PopThousands\"] = boroughs[\"Population\"]/1000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "boroughs[\"AvgHouseholdSize\"] = boroughs[\"Population\"]/boroughs[\"Households\"]\n",
    "boroughs.sort_values(\"AvgHouseholdSize\", ascending=False)[\"AvgHouseholdSize\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "justBoroughs.corr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.lmplot(data=boroughs,\n",
    "          x=\"Pay\",\n",
    "          y=\"Happy\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Time Series\n",
    "\n",
    "The other `csv` files all contain time series. Let's look at how recycling has changed over recent years."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "recycling = pd.read_csv('recycling.csv')\n",
    "recycling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.to_datetime(recycling.Year,format=\"%Y\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "This time we'll make `Year` the index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "recycling = recycling.set_index(\"Year\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Now we can draw a time series graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "recycling.Barnet.plot(c=\"red\")\n",
    "recycling[\"Barking and Dagenham\"].plot(c=\"green\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "It would be helpful to be able to show that Barking and Dagenham has improved by more *as a proportion* of their starting point than Barnet has.\n",
    "\n",
    "We can make a new a column, call it BarnetIndexed say, and fill it with the percentages scaled to 1 at 2004. And the same for Barking and Dagenham."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "recycling[\"BarnetIndexed\"] = recycling.Barnet/recycling.Barnet[2004]\n",
    "recycling[\"Barking and DagenhamIndexed\"] =recycling[\"Barking and Dagenham\"]/recycling[\"Barking and Dagenham\"][2004]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Things to note about the above\n",
    "\n",
    "* you can make a new column just by saying `recycling[\"New column name\"]=`\n",
    "* you can divide every number in a column by the value in 2004 by just doing `recycling.Barnet/recycling.Barnet[2004]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "recycling.BarnetIndexed.plot(c=\"red\")\n",
    "recycling[\"Barking and DagenhamIndexed\"].plot(c=\"blue\");\n",
    "# note the `c` for colour"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In fact, let's go ahead and do that for all the boroughs. We can use a `for` loop over all the columns (remember that in this dataframe it's the boroughs that are columns and the years are rows.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "for column in recycling.columns:\n",
    "    recycling[\"{}Indexed\".format(column)] = recycling[column]/recycling[column][2004]\n",
    "recycling.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "recycling[\"Newham\"].plot(c=\"green\")\n",
    "recycling[\"NewhamIndexed\"].plot(c=\"blue\")\n",
    "recycling[\"Barnet\"].plot(c=\"orange\")\n",
    "recycling[\"BarnetIndexed\"].plot(c=\"red\")\n",
    "plt.title(\"Recycling in Newham and Barnet\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "There was a small fudge in here. If you check `recycling.dtypes` you'll see that `Year` was an `int64` (an integer) which worked okay for us this time, but in future we'll want to explicit turn it into a `datetime` object instead, so `pandas` knows we're dealing with time. We'll do that with [`to_datetime`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.to_datetime.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Documentation for [`pandas` is here](http://pandas.pydata.org/pandas-docs/stable/).\n",
    "\n",
    "We've installed several visualisation libraries that you might find useful\n",
    "\n",
    "* [`pyplot`](https://matplotlib.org/)\n",
    "* [`seaborn`](https://seaborn.pydata.org/)\n",
    "* [`bokeh`](https://bokeh.pydata.org/)\n",
    "* [`chartify`](https://labs.spotify.com/2018/11/15/introducing-chartify-easier-chart-creation-in-python-for-data-scientists/)\n",
    "* [`geopandas`](http://geopandas.org/)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}