1. Intro to Minimally Sufficient Pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Download material at [github.com/tdpetrou/minimally-sufficient-pandas][1]\n",
    "\n",
    "[1]: http://github.com/tdpetrou/minimally-sufficient-pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Minimally Sufficient Pandas by Ted Petrou\n",
    "\n",
    "## 3rd Annual Global Big Data Conference, Santa Clara, CA, January 23, 2019\n",
    "\n",
    "## About Me\n",
    "\n",
    "* Founder of [Dunder Data][0] - Professional data science training company\n",
    "![][1]\n",
    "* Author of Pandas Cookbook\n",
    "* Author of Master Data Analysis with Pandas\n",
    "    * 3 Volume Set, 1,000 pages, 500 exercises, 10+ projects\n",
    "* Author of Exercise Python\n",
    "    * Introduction to Python with over 100 exercises and several projects\n",
    "* Author of Dexplo and Dexplot\n",
    "    * Data exploration and visualization libraries\n",
    "* Twitter - [TedPetrou][2]\n",
    "    \n",
    "[0]: https://dunderdata.com/\n",
    "[1]: images/pc.png?a=4\n",
    "[2]: https://twitter.com/tedPetrou"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Target Audience\n",
    "\n",
    "This tutorial targets those that have used Pandas before and want to have a much more streamlined and efficient approach to using the library.\n",
    "\n",
    "## What does minimally sufficient Pandas mean?\n",
    "Pandas is the most popular Python library for doing data analysis. Unfortunately, it is also one of the most difficult libraries to use properly. There are several reasons for this:\n",
    "\n",
    "* There is often more than one way to get the same result\n",
    "* Many tutorials will show different ways to do the same thing\n",
    "* There are over 300 total attributes and methods\n",
    "* Some of these methods do the exact same thing (are aliases of each other)\n",
    "* Some of these methods are very similar and could be condensed into one\n",
    "* There are many tutorials (as well as the documentation) that show highly inefficient and non-idiomatic approaches to Pandas\n",
    "\n",
    "### My definition for minimally sufficient Pandas\n",
    "* A small subset of the Pandas library is sufficient to accomplish nearly everything that it has to offer. \n",
    "* Focus on doing data analysis and don't get bogged down with syntax\n",
    "* The whole point of being an analyst is to analyze data, not to learn every single possible method and trick that the library has to offer.\n",
    "\n",
    "### Minimize Complex Code\n",
    "\n",
    "With a minimally sufficient subset ...\n",
    "\n",
    "* Your code will be simple, explicit, straightforward, and boring\n",
    "* You will choose one obvious way to accomplish a task\n",
    "* Use this obvious way every single time\n",
    "* You won't have to retain as many commands in working memory\n",
    "* Your code will be easier to read by others and by you after a break\n",
    "\n",
    "### Do any of these apply to you?\n",
    "\n",
    "* Get anxiety because of the enormous number of methods and feel you might be missing out on some special part of the library\n",
    "* Wonder why there are so many methods that do the exact same thing\n",
    "* Work with team members that use Pandas code that is difficult to understand\n",
    "* Don't know the difference between `[], iloc, loc, ix, at, iat` and why there are so many different ways to select subsets of data\n",
    "* Have trouble dealing with the index\n",
    "* Have even more trouble dealing with multi-level indexes (MultiIndex)\n",
    "* Have no idea what to do with the `SettingWithCopy` warning\n",
    "* Use the `apply` method frequently\n",
    "* Use any for-loops at all\n",
    "* Get confused by all the different `groupby` syntaxes\n",
    "* Write custom `groupby` functions that are extremely slow\n",
    "* Find yourself wishing it was more like R\n",
    "\n",
    "\n",
    "\n",
    "### No Tricks\n",
    "Eliminating much of the library will come with some (good) limitations. \n",
    "\n",
    "* Knowing many obscure Pandas tricks might impress your friends, but it doesn't usually lead to good code. \n",
    "* Knowing more tricks can lead to very long lines of code\n",
    "* Longer line of code may be harder to debug\n",
    "* Ask yourself whether method B gives you more functionality than method A\n",
    "* Pandas is difficult to use in production - striving for consistency and simplicity can make a big difference\n",
    "* There are an incredible amount of issues/bugs and using a minimally sufficient subset of Pandas can help avoid landing on a bug"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tutorial Objectives\n",
    "\n",
    "### Core objectives\n",
    "\n",
    "* Know why having a single method for doing a particular task is good practice\n",
    "* Have guidance on how to approach very common data analysis tasks with a single suggested way to accomplish it\n",
    "* Have my complete list of attributes and methods that allow you to accomplish nearly all tasks\n",
    "\n",
    "### More specific objectives\n",
    "* Idiomatic subset selection\n",
    "* Handling the `SettingWithCopyWarning`\n",
    "* Avoiding aliases\n",
    "* Know how to avoid `apply` and for-loops\n",
    "* Know how to handle the MultiIndex after a `groupby`\n",
    "* Know the equivalence of `groupby`, `pivot_table`, and `pd.crosstab`\n",
    "* Know why having Pandas guidelines for productions is a very good thing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "c = pd.read_csv('data/college.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "243"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len([m for m in dir(c) if not m.startswith('_')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attendance and Skill Level"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import IFrame\n",
    "IFrame('https://directpoll.com/v?XDVhEtE5iKBCVu81bhI3a9iCToKziCmYz', 600, 400)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import IFrame\n",
    "IFrame('https://directpoll.com/r?XDbzPBd3ixYqg8gcNzkffHI6YeLC2OAj8TtSsViOxH3', 600, 400)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "pd.set_option('display.max_columns', 100)\n",
    "college = pd.read_csv('data/college.csv')\n",
    "college.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas Skills Test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1\n",
    "<span  style=\"color:green; font-size:16px\">What is the median SAT Math score (satmtmid) for University of Arkansas?</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2\n",
    "<span  style=\"color:green; font-size:16px\">What state (stabbr) has the 10th highest total undergraduate population (ugds) and what is that population?</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas Skills Test Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame('https://directpoll.com/v?XDVhEtjm0FjufI7yoKzYsetJkwpN8VAh', 600, 400)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame('https://directpoll.com/r?XDbzPBd3ixYqg8BJfuINpfH3lH2hKtLqPbOqEe5nDf', 600, 400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}