webscraping/webscraping_in_python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Web Scraping with Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copying and pasting is great. You should definitely use it if it's the simplest way. But if you don't want to copy and paste 10 webpages into Excel or if you have some time to kill, then web scraping is the answer or time sink you've been looking for."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by importing the python modules that we will need to do web scraping. We will be using `requests` to fetch html pages and `BeautifulSoup` to parse the html page. `pandas` will be used for data manipulation. The `pd.options.display` lines are for formatting purposes when printing out results in `pandas`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd\n",
    "\n",
    "pd.options.display.max_rows = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example 1: Wikipedia Page (HTML Table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by opening the <a href='https://en.wikipedia.org/wiki/List_of_Michigan_locations_by_per_capita_income'>website of interest</a> in a browser. We can see that it looks nicely formatted like a table. We start with passing the website of interest to the `requests.get` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://en.wikipedia.org/wiki/List_of_Michigan_locations_by_per_capita_income'\n",
    "R = requests.get(url)\n",
    "R.raise_for_status()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: Without the `R.raise_for_status()` line, bad urls will fail silently which is probably not what you want."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then use `BeautifulSoup` to parse the contents of the response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(R.content, 'html.parser')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I like to use the browser's developer tool to do the detective work of figuring out where the data resides (*right click -> Inspect*). In this example, the data of interest resides in a html table (makes life easier)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the data of interest resides in a table tag &lt;table&gt;. To grab everything between the table tags, we use the `find_all` method (one of many options but probably the only one you need and the one you will use most often)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "tables = soup.find_all('table')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`find_all` returns a list of matches. We can use the `len` function to see how many matches came back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check the contents of the table using the `text` method. It will be a formatting mess but that's okay."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'This article is part of a series onIncome in theUnited States of America\\nTopics\\nHousehold\\nPersonal\\nAffluence\\nSocial class\\nIncome inequality\\ngender pay gap\\nethnic wage gap\\n\\nLists by income\\nStates (by equality (Gini))\\nCounties (highest\\xa0/ lowest)\\nLocations (lowest)\\nMetropolitan statistical areas\\nUrban areas\\nZIP Code Tabulation Areas\\nEthnic groups\\n\\n United States portalvte'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tables[0].text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n\\nRank\\n\\nCounty\\n\\nPer capitaincome\\n\\nMedianhouseholdincome\\n\\nMedianfamilyincome\\n\\nPopulation\\n\\nNumber ofhouseholds\\n\\n\\n1\\n\\nOakland\\n\\n$56,138\\n\\n$85,991\\n\\n$94,783\\n\\n1,202,362\\n\\n483,698\\n\\n\\n2\\n\\nLeelanau\\n\\n$32,194\\n\\n$56,527\\n\\n$65,342\\n\\n21,708\\n\\n9,255\\n\\n\\n3\\n\\nLivingston\\n\\n$31,609\\n\\n$72,129\\n\\n$82,637\\n\\n180,967\\n\\n67,380\\n\\n\\n4\\n\\nWashtenaw\\n\\n$31,316\\n\\n$59,065\\n\\n$82,184\\n\\n344,791\\n\\n137,193\\n\\n\\n5\\n\\nCharlevoix\\n\\n$28,403\\n\\n$48,704\\n\\n$57,022\\n\\n25,949\\n\\n10,882\\n\\n\\n6\\n\\nMidland\\n\\n$28,363\\n\\n$51,103\\n\\n$63,299\\n\\n83,629\\n\\n33,437\\n\\n\\n7\\n\\nEmmet\\n\\n$28,308\\n\\n$49,235\\n\\n$61,600\\n\\n32,694\\n\\n13,601\\n\\n\\n\\n\\nUnited States\\n\\n$27,334\\n\\n$51,914\\n\\n$62,982\\n\\n308,745,538\\n\\n116,716,292\\n\\n\\n8\\n\\nClinton\\n\\n$27,223\\n\\n$58,016\\n\\n$69,611\\n\\n75,382\\n\\n28,766\\n\\n\\n9\\n\\nGrand Traverse\\n\\n$27,091\\n\\n$50,647\\n\\n$61,780\\n\\n86,986\\n\\n35,328\\n\\n\\n10\\n\\nMacomb\\n\\n$26,524\\n\\n$53,996\\n\\n$67,423\\n\\n840,978\\n\\n331,667\\n\\n\\n11\\n\\nEaton\\n\\n$25,963\\n\\n$54,885\\n\\n$66,788\\n\\n107,759\\n\\n43,494\\n\\n\\n12\\n\\nMonroe\\n\\n$25,520\\n\\n$55,366\\n\\n$66,549\\n\\n152,021\\n\\n58,230\\n\\n\\n13\\n\\nKalamazoo\\n\\n$25,138\\n\\n$44,794\\n\\n$61,622\\n\\n250,331\\n\\n100,610\\n\\n\\n\\n\\nMichigan\\n\\n$25,135\\n\\n$48,432\\n\\n$60,341\\n\\n9,883,640\\n\\n3,872,508\\n\\n\\n14\\n\\nLapeer\\n\\n$25,110\\n\\n$55,005\\n\\n$63,061\\n\\n88,319\\n\\n32,776\\n\\n\\n15\\n\\nOttawa\\n\\n$25,045\\n\\n$55,095\\n\\n$65,474\\n\\n263,801\\n\\n93,775\\n\\n\\n16\\n\\nKent\\n\\n$24,791\\n\\n$49,532\\n\\n$61,097\\n\\n602,622\\n\\n227,239\\n\\n\\n17\\n\\nBarry\\n\\n$24,493\\n\\n$51,869\\n\\n$61,202\\n\\n59,173\\n\\n22,551\\n\\n\\n18\\n\\nBerrien\\n\\n$24,025\\n\\n$42,625\\n\\n$54,751\\n\\n156,813\\n\\n63,054\\n\\n\\n19\\n\\nAntrim\\n\\n$23,912\\n\\n$43,123\\n\\n$50,424\\n\\n23,580\\n\\n9,890\\n\\n\\n20\\n\\nIngham\\n\\n$23,883\\n\\n$45,808\\n\\n$61,680\\n\\n280,895\\n\\n111,162\\n\\n\\n21\\n\\nDickinson\\n\\n$23,854\\n\\n$42,586\\n\\n$54,053\\n\\n26,168\\n\\n11,359\\n\\n\\n22\\n\\nSt. Clair\\n\\n$23,828\\n\\n$49,120\\n\\n$59,969\\n\\n163,040\\n\\n63,841\\n\\n\\n23\\n\\nBenzie\\n\\n$23,649\\n\\n$44,718\\n\\n$53,250\\n\\n17,525\\n\\n7,298\\n\\n\\n24\\n\\nMarquette\\n\\n$23,347\\n\\n$45,130\\n\\n$61,798\\n\\n67,077\\n\\n27,538\\n\\n\\n25\\n\\nAllegan\\n\\n$23,108\\n\\n$50,240\\n\\n$57,831\\n\\n111,408\\n\\n42,018\\n\\n\\n26\\n\\nBay\\n\\n$23,049\\n\\n$44,659\\n\\n$53,824\\n\\n107,771\\n\\n44,603\\n\\n\\n27\\n\\nCheboygan\\n\\n$23,038\\n\\n$37,903\\n\\n$45,769\\n\\n26,152\\n\\n11,133\\n\\n\\n28\\n\\nCass\\n\\n$22,698\\n\\n$45,177\\n\\n$54,813\\n\\n52,293\\n\\n20,604\\n\\n\\n29\\n\\nOtsego\\n\\n$22,568\\n\\n$45,531\\n\\n$54,110\\n\\n24,164\\n\\n9,756\\n\\n\\n30\\n\\nLenawee\\n\\n$22,529\\n\\n$48,618\\n\\n$60,028\\n\\n99,892\\n\\n37,514\\n\\n\\n31\\n\\nGenesee\\n\\n$22,458\\n\\n$43,483\\n\\n$54,072\\n\\n425,790\\n\\n169,202\\n\\n\\n32\\n\\nMackinac\\n\\n$22,170\\n\\n$39,339\\n\\n$51,376\\n\\n11,113\\n\\n5,024\\n\\n\\n33\\n\\nCalhoun\\n\\n$22,166\\n\\n$42,568\\n\\n$52,533\\n\\n136,146\\n\\n54,016\\n\\n\\n34\\n\\nWayne\\n\\n$22,125\\n\\n$42,241\\n\\n$52,946\\n\\n1,820,584\\n\\n702,749\\n\\n\\n35\\n\\nHuron\\n\\n$22,098\\n\\n$40,038\\n\\n$49,444\\n\\n33,118\\n\\n14,348\\n\\n\\n36\\n\\nDelta\\n\\n$22,064\\n\\n$41,951\\n\\n$51,442\\n\\n37,069\\n\\n15,992\\n\\n\\n37\\n\\nVan Buren\\n\\n$22,002\\n\\n$44,435\\n\\n$54,499\\n\\n76,258\\n\\n28,928\\n\\n\\n38\\n\\nJackson\\n\\n$21,947\\n\\n$46,117\\n\\n$56,314\\n\\n160,248\\n\\n60,771\\n\\n\\n39\\n\\nShiawassee\\n\\n$21,869\\n\\n$46,453\\n\\n$54,363\\n\\n70,648\\n\\n27,481\\n\\n\\n40\\n\\nMason\\n\\n$21,760\\n\\n$40,039\\n\\n$49,131\\n\\n28,705\\n\\n11,940\\n\\n\\n41\\n\\nSaginaw\\n\\n$21,662\\n\\n$42,954\\n\\n$53,171\\n\\n200,169\\n\\n79,011\\n\\n\\n42\\n\\nMenominee\\n\\n$21,624\\n\\n$41,332\\n\\n$49,394\\n\\n24,029\\n\\n10,474\\n\\n\\n43\\n\\nManistee\\n\\n$21,612\\n\\n$40,853\\n\\n$50,101\\n\\n24,733\\n\\n10,308\\n\\n\\n44\\n\\nOntonagon\\n\\n$21,448\\n\\n$35,269\\n\\n$47,330\\n\\n6,780\\n\\n3,258\\n\\n\\n45\\n\\nKeweenaw\\n\\n$21,307\\n\\n$38,872\\n\\n$46,414\\n\\n2,156\\n\\n1,013\\n\\n\\n46\\n\\nAlpena\\n\\n$21,140\\n\\n$36,695\\n\\n$47,256\\n\\n29,598\\n\\n12,791\\n\\n\\n47\\n\\nCrawford\\n\\n$21,002\\n\\n$39,665\\n\\n$45,362\\n\\n14,074\\n\\n6,016\\n\\n\\n48\\n\\nNewaygo\\n\\n$20,870\\n\\n$43,218\\n\\n$49,499\\n\\n48,460\\n\\n18,406\\n\\n\\n49\\n\\nPresque Isle\\n\\n$20,870\\n\\n$37,383\\n\\n$43,797\\n\\n13,376\\n\\n5,982\\n\\n\\n50\\n\\nGladwin\\n\\n$20,571\\n\\n$37,936\\n\\n$44,427\\n\\n25,692\\n\\n10,753\\n\\n\\n51\\n\\nIosco\\n\\n$20,513\\n\\n$36,861\\n\\n$44,175\\n\\n25,887\\n\\n11,757\\n\\n\\n52\\n\\nSchoolcraft\\n\\n$20,455\\n\\n$36,925\\n\\n$48,141\\n\\n8,485\\n\\n3,759\\n\\n\\n53\\n\\nChippewa\\n\\n$20,309\\n\\n$40,194\\n\\n$54,066\\n\\n38,520\\n\\n14,329\\n\\n\\n54\\n\\nRoscommon\\n\\n$20,194\\n\\n$33,542\\n\\n$40,015\\n\\n24,449\\n\\n11,433\\n\\n\\n55\\n\\nSt. Joseph\\n\\n$20,192\\n\\n$44,392\\n\\n$52,586\\n\\n61,295\\n\\n23,244\\n\\n\\n56\\n\\nHillsdale\\n\\n$20,006\\n\\n$42,989\\n\\n$50,546\\n\\n46,688\\n\\n17,792\\n\\n\\n57\\n\\nIron\\n\\n$19,986\\n\\n$33,734\\n\\n$44,560\\n\\n11,817\\n\\n5,577\\n\\n\\n58\\n\\nWexford\\n\\n$19,952\\n\\n$39,997\\n\\n$46,659\\n\\n32,735\\n\\n13,021\\n\\n\\n59\\n\\nTuscola\\n\\n$19,937\\n\\n$42,198\\n\\n$50,262\\n\\n55,729\\n\\n21,590\\n\\n\\n60\\n\\nGogebic\\n\\n$19,933\\n\\n$33,673\\n\\n$45,182\\n\\n16,427\\n\\n7,037\\n\\n\\n61\\n\\nAlcona\\n\\n$19,904\\n\\n$34,858\\n\\n$43,482\\n\\n10,942\\n\\n5,089\\n\\n\\n62\\n\\nAlger\\n\\n$19,858\\n\\n$38,262\\n\\n$47,548\\n\\n9,601\\n\\n3,898\\n\\n\\n63\\n\\nKalkaska\\n\\n$19,770\\n\\n$39,350\\n\\n$45,417\\n\\n17,153\\n\\n6,962\\n\\n\\n64\\n\\nMuskegon\\n\\n$19,719\\n\\n$40,670\\n\\n$50,101\\n\\n172,188\\n\\n65,616\\n\\n\\n65\\n\\nSanilac\\n\\n$19,645\\n\\n$40,818\\n\\n$49,005\\n\\n43,114\\n\\n17,132\\n\\n\\n66\\n\\nMissaukee\\n\\n$19,560\\n\\n$40,376\\n\\n$46,371\\n\\n14,849\\n\\n5,843\\n\\n\\n67\\n\\nIonia\\n\\n$19,386\\n\\n$46,454\\n\\n$54,595\\n\\n63,905\\n\\n22,144\\n\\n\\n68\\n\\nBaraga\\n\\n$19,107\\n\\n$40,541\\n\\n$50,549\\n\\n8,860\\n\\n3,444\\n\\n\\n69\\n\\nMontmorency\\n\\n$19,102\\n\\n$34,447\\n\\n$41,230\\n\\n9,765\\n\\n4,416\\n\\n\\n70\\n\\nArenac\\n\\n$19,073\\n\\n$36,689\\n\\n$45,376\\n\\n15,899\\n\\n6,701\\n\\n\\n71\\n\\nBranch\\n\\n$19,049\\n\\n$42,133\\n\\n$50,931\\n\\n45,248\\n\\n16,419\\n\\n\\n72\\n\\nMecosta\\n\\n$18,745\\n\\n$35,887\\n\\n$48,145\\n\\n42,798\\n\\n16,101\\n\\n\\n73\\n\\nMontcalm\\n\\n$18,569\\n\\n$39,775\\n\\n$46,673\\n\\n63,342\\n\\n23,432\\n\\n\\n74\\n\\nOscoda\\n\\n$18,524\\n\\n$32,346\\n\\n$39,335\\n\\n8,640\\n\\n3,772\\n\\n\\n75\\n\\nIsabella\\n\\n$18,510\\n\\n$36,880\\n\\n$55,183\\n\\n70,311\\n\\n25,586\\n\\n\\n76\\n\\nClare\\n\\n$18,491\\n\\n$34,399\\n\\n$42,519\\n\\n30,926\\n\\n12,966\\n\\n\\n77\\n\\nOceana\\n\\n$18,402\\n\\n$39,543\\n\\n$46,424\\n\\n26,570\\n\\n10,174\\n\\n\\n78\\n\\nGratiot\\n\\n$18,388\\n\\n$40,114\\n\\n$49,989\\n\\n42,476\\n\\n14,852\\n\\n\\n79\\n\\nOgemaw\\n\\n$18,321\\n\\n$35,968\\n\\n$41,810\\n\\n21,699\\n\\n9,283\\n\\n\\n80\\n\\nHoughton\\n\\n$18,267\\n\\n$34,174\\n\\n$46,819\\n\\n36,628\\n\\n14,232\\n\\n\\n81\\n\\nOsceola\\n\\n$17,861\\n\\n$38,341\\n\\n$44,613\\n\\n23,528\\n\\n9,222\\n\\n\\n82\\n\\nLuce\\n\\n$17,195\\n\\n$40,041\\n\\n$46,510\\n\\n6,631\\n\\n2,412\\n\\n\\n83\\n\\nLake\\n\\n$16,084\\n\\n$31,205\\n\\n$38,996\\n\\n11,539\\n\\n5,158\\n\\n'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tables[1].text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like this is the table we want. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A better way to find the table of interest is to pass in extra search terms to the `find_all` method after using the developer tool to find searchable attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tables = soup.find_all('table', class_=\"wikitable sortable\")\n",
    "len(tables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get one result now instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "table = tables[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data resides in a html table standard cell tag `<td>` within a table row tag `<tr>`. We use `find_all` to look for all the table row tags within the table tag `<table>`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "86"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table_rows = table.find_all('tr')\n",
    "len(table_rows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now see that there are a lot of matches for that tag, as expected. It's close to the total number of countries. Let's look at the first few entries of `table_rows`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tr valign=\"bottom\">\n",
       "<th>Rank\n",
       "</th>\n",
       "<th>County\n",
       "</th>\n",
       "<th>Per capita<br/>income\n",
       "</th>\n",
       "<th>Median<br/>household<br/>income\n",
       "</th>\n",
       "<th>Median<br/>family<br/>income\n",
       "</th>\n",
       "<th>Population\n",
       "</th>\n",
       "<th>Number of<br/>households\n",
       "</th></tr>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table_rows[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like the header row. The `<th>` tag also gives it away, fyi."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tr>\n",
       "<td>1\n",
       "</td>\n",
       "<td><a href=\"/wiki/Oakland_County,_Michigan\" title=\"Oakland County, Michigan\">Oakland</a>\n",
       "</td>\n",
       "<td>$56,138\n",
       "</td>\n",
       "<td>$85,991\n",
       "</td>\n",
       "<td>$94,783\n",
       "</td>\n",
       "<td>1,202,362\n",
       "</td>\n",
       "<td>483,698\n",
       "</td></tr>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table_rows[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Its the row data for the __first county__. \n",
    "\n",
    "We will use a nested `for` loop to go through the list of table rows. The inner `for` loop will go through each `<td>` tag appending the text to a list. We will grab all the data in the tags regardless of whether we want to keep them for now. \n",
    "\n",
    "We have two lists in the `for` loop. `row` will contain a list of the each `td` tag in a table row. Once the row is iterated through, we will convert it to a `pandas` dataframe. `list_df` will contain a list of those dataframes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "rows = []\n",
    "for row in table_rows[1:]:\n",
    "    table_cells = row.find_all('td')\n",
    "    cells = []\n",
    "    for cell in table_cells:\n",
    "        cells.append(cell.text)\n",
    "    rows.append(cells)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We take the list of dataframes and concatenate them together into one dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.DataFrame(rows) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the first 5 rows of the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1\\n</td>\n",
       "      <td>Oakland\\n</td>\n",
       "      <td>$56,138\\n</td>\n",
       "      <td>$85,991\\n</td>\n",
       "      <td>$94,783\\n</td>\n",
       "      <td>1,202,362\\n</td>\n",
       "      <td>483,698\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2\\n</td>\n",
       "      <td>Leelanau\\n</td>\n",
       "      <td>$32,194\\n</td>\n",
       "      <td>$56,527\\n</td>\n",
       "      <td>$65,342\\n</td>\n",
       "      <td>21,708\\n</td>\n",
       "      <td>9,255\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3\\n</td>\n",
       "      <td>Livingston\\n</td>\n",
       "      <td>$31,609\\n</td>\n",
       "      <td>$72,129\\n</td>\n",
       "      <td>$82,637\\n</td>\n",
       "      <td>180,967\\n</td>\n",
       "      <td>67,380\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4\\n</td>\n",
       "      <td>Washtenaw\\n</td>\n",
       "      <td>$31,316\\n</td>\n",
       "      <td>$59,065\\n</td>\n",
       "      <td>$82,184\\n</td>\n",
       "      <td>344,791\\n</td>\n",
       "      <td>137,193\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>5\\n</td>\n",
       "      <td>Charlevoix\\n</td>\n",
       "      <td>$28,403\\n</td>\n",
       "      <td>$48,704\\n</td>\n",
       "      <td>$57,022\\n</td>\n",
       "      <td>25,949\\n</td>\n",
       "      <td>10,882\\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0             1          2          3          4            5          6\n",
       "0  1\\n     Oakland\\n  $56,138\\n  $85,991\\n  $94,783\\n  1,202,362\\n  483,698\\n\n",
       "1  2\\n    Leelanau\\n  $32,194\\n  $56,527\\n  $65,342\\n     21,708\\n    9,255\\n\n",
       "2  3\\n  Livingston\\n  $31,609\\n  $72,129\\n  $82,637\\n    180,967\\n   67,380\\n\n",
       "3  4\\n   Washtenaw\\n  $31,316\\n  $59,065\\n  $82,184\\n    344,791\\n  137,193\\n\n",
       "4  5\\n  Charlevoix\\n  $28,403\\n  $48,704\\n  $57,022\\n     25,949\\n   10,882\\n"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We didn't get the header row because it was contained in `<th>` tags and not `<td>` tags. We use the same type of `for` loop to extract the text from the table headers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = table_rows[0].find_all('th')\n",
    "columns = []\n",
    "for header in headers:\n",
    "    columns.append(header.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the column headers of the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Rank\\n</th>\n",
       "      <th>County\\n</th>\n",
       "      <th>Per capitaincome\\n</th>\n",
       "      <th>Medianhouseholdincome\\n</th>\n",
       "      <th>Medianfamilyincome\\n</th>\n",
       "      <th>Population\\n</th>\n",
       "      <th>Number ofhouseholds\\n</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1\\n</td>\n",
       "      <td>Oakland\\n</td>\n",
       "      <td>$56,138\\n</td>\n",
       "      <td>$85,991\\n</td>\n",
       "      <td>$94,783\\n</td>\n",
       "      <td>1,202,362\\n</td>\n",
       "      <td>483,698\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2\\n</td>\n",
       "      <td>Leelanau\\n</td>\n",
       "      <td>$32,194\\n</td>\n",
       "      <td>$56,527\\n</td>\n",
       "      <td>$65,342\\n</td>\n",
       "      <td>21,708\\n</td>\n",
       "      <td>9,255\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3\\n</td>\n",
       "      <td>Livingston\\n</td>\n",
       "      <td>$31,609\\n</td>\n",
       "      <td>$72,129\\n</td>\n",
       "      <td>$82,637\\n</td>\n",
       "      <td>180,967\\n</td>\n",
       "      <td>67,380\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4\\n</td>\n",
       "      <td>Washtenaw\\n</td>\n",
       "      <td>$31,316\\n</td>\n",
       "      <td>$59,065\\n</td>\n",
       "      <td>$82,184\\n</td>\n",
       "      <td>344,791\\n</td>\n",
       "      <td>137,193\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>5\\n</td>\n",
       "      <td>Charlevoix\\n</td>\n",
       "      <td>$28,403\\n</td>\n",
       "      <td>$48,704\\n</td>\n",
       "      <td>$57,022\\n</td>\n",
       "      <td>25,949\\n</td>\n",
       "      <td>10,882\\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Rank\\n      County\\n Per capitaincome\\n Medianhouseholdincome\\n  \\\n",
       "0    1\\n     Oakland\\n          $56,138\\n               $85,991\\n   \n",
       "1    2\\n    Leelanau\\n          $32,194\\n               $56,527\\n   \n",
       "2    3\\n  Livingston\\n          $31,609\\n               $72,129\\n   \n",
       "3    4\\n   Washtenaw\\n          $31,316\\n               $59,065\\n   \n",
       "4    5\\n  Charlevoix\\n          $28,403\\n               $48,704\\n   \n",
       "\n",
       "  Medianfamilyincome\\n Population\\n Number ofhouseholds\\n  \n",
       "0            $94,783\\n  1,202,362\\n             483,698\\n  \n",
       "1            $65,342\\n     21,708\\n               9,255\\n  \n",
       "2            $82,637\\n    180,967\\n              67,380\\n  \n",
       "3            $82,184\\n    344,791\\n             137,193\\n  \n",
       "4            $57,022\\n     25,949\\n              10,882\\n  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.columns = columns\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have all the data of interest. __Webscraping DONE!__ If you want to do some data cleanup and management, this is where knowledge of `pandas` becomes useful (CSCAR has a panda workshop on how to do just that)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `pandas` Approach to HTML Tables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pandas `read_html` method can read in HTML Tables (and only HTML Tables). It uses `BeautifulSoup` under the hood."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_tables = pd.read_html(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: If you get an `ImportError: html5lib not found, please install it` or `ImportError: lxml not found, please install it` message, you will need to install it via `conda install html5lib` or `conda install lxml` and restart the kernel or Juypter Notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This returns a list of dataframes. One dataframe for each table in the html page. Recall how many tables there were. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(list_tables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check their contents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Rank</th>\n",
       "      <th>County</th>\n",
       "      <th>Per capitaincome</th>\n",
       "      <th>Medianhouseholdincome</th>\n",
       "      <th>Medianfamilyincome</th>\n",
       "      <th>Population</th>\n",
       "      <th>Number ofhouseholds</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Oakland</td>\n",
       "      <td>$56,138</td>\n",
       "      <td>$85,991</td>\n",
       "      <td>$94,783</td>\n",
       "      <td>1202362</td>\n",
       "      <td>483698</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Leelanau</td>\n",
       "      <td>$32,194</td>\n",
       "      <td>$56,527</td>\n",
       "      <td>$65,342</td>\n",
       "      <td>21708</td>\n",
       "      <td>9255</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Livingston</td>\n",
       "      <td>$31,609</td>\n",
       "      <td>$72,129</td>\n",
       "      <td>$82,637</td>\n",
       "      <td>180967</td>\n",
       "      <td>67380</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Washtenaw</td>\n",
       "      <td>$31,316</td>\n",
       "      <td>$59,065</td>\n",
       "      <td>$82,184</td>\n",
       "      <td>344791</td>\n",
       "      <td>137193</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Charlevoix</td>\n",
       "      <td>$28,403</td>\n",
       "      <td>$48,704</td>\n",
       "      <td>$57,022</td>\n",
       "      <td>25949</td>\n",
       "      <td>10882</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Rank      County Per capitaincome Medianhouseholdincome Medianfamilyincome  \\\n",
       "0   1.0     Oakland          $56,138               $85,991            $94,783   \n",
       "1   2.0    Leelanau          $32,194               $56,527            $65,342   \n",
       "2   3.0  Livingston          $31,609               $72,129            $82,637   \n",
       "3   4.0   Washtenaw          $31,316               $59,065            $82,184   \n",
       "4   5.0  Charlevoix          $28,403               $48,704            $57,022   \n",
       "\n",
       "   Population  Number ofhouseholds  \n",
       "0     1202362               483698  \n",
       "1       21708                 9255  \n",
       "2      180967                67380  \n",
       "3      344791               137193  \n",
       "4       25949                10882  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list_tables[1].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same table contains our data of interest (as expected)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also pass tag attributes to `pandas` like we did before. Use the keyword argument `match` to do a text search within the table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Rank</th>\n",
       "      <th>County</th>\n",
       "      <th>Per capitaincome</th>\n",
       "      <th>Medianhouseholdincome</th>\n",
       "      <th>Medianfamilyincome</th>\n",
       "      <th>Population</th>\n",
       "      <th>Number ofhouseholds</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Oakland</td>\n",
       "      <td>$56,138</td>\n",
       "      <td>$85,991</td>\n",
       "      <td>$94,783</td>\n",
       "      <td>1202362</td>\n",
       "      <td>483698</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Leelanau</td>\n",
       "      <td>$32,194</td>\n",
       "      <td>$56,527</td>\n",
       "      <td>$65,342</td>\n",
       "      <td>21708</td>\n",
       "      <td>9255</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Livingston</td>\n",
       "      <td>$31,609</td>\n",
       "      <td>$72,129</td>\n",
       "      <td>$82,637</td>\n",
       "      <td>180967</td>\n",
       "      <td>67380</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Washtenaw</td>\n",
       "      <td>$31,316</td>\n",
       "      <td>$59,065</td>\n",
       "      <td>$82,184</td>\n",
       "      <td>344791</td>\n",
       "      <td>137193</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Charlevoix</td>\n",
       "      <td>$28,403</td>\n",
       "      <td>$48,704</td>\n",
       "      <td>$57,022</td>\n",
       "      <td>25949</td>\n",
       "      <td>10882</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Rank      County Per capitaincome Medianhouseholdincome Medianfamilyincome  \\\n",
       "0   1.0     Oakland          $56,138               $85,991            $94,783   \n",
       "1   2.0    Leelanau          $32,194               $56,527            $65,342   \n",
       "2   3.0  Livingston          $31,609               $72,129            $82,637   \n",
       "3   4.0   Washtenaw          $31,316               $59,065            $82,184   \n",
       "4   5.0  Charlevoix          $28,403               $48,704            $57,022   \n",
       "\n",
       "   Population  Number ofhouseholds  \n",
       "0     1202362               483698  \n",
       "1       21708                 9255  \n",
       "2      180967                67380  \n",
       "3      344791               137193  \n",
       "4       25949                10882  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list_tables = pd.read_html(url, attrs={'class':'wikitable sortable'})\n",
    "list_tables[0].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This leaves us at a similiar point that we encountered earlier in the example. `pandas` has basically done the web scraping for us and left us with the data cleanup and wrangling. You should always expect to do some data manipulation if you use the `pd.read_html()` method or any webscraping for that matter.\n",
    "\n",
    "Q: So why did we learn a more complicated way of doing things when `pd.read_html()` can do it for you?  \n",
    "A: Because not everything resides in an HTML table and you'll need to use the same techniques to get at the data. If the data resides in a HTML table, consider yourself fortunate, use `pandas` and move on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scrape Serena Williams' Wikipedia page https://en.wikipedia.org/wiki/Serena_Williams for the data in the html table **Grand Slam tournament finals**  for *Singles* in the *Career Statistics* section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example 2: USA TODAY Best Selling Books (not in HTML Table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, the website of interest is https://www.usatoday.com/life/books/best-selling/. Suppose we are interested in getting some basic information about the book list. For this example, the data of interest does not exist in a html table."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start with the boilerplate template of passing the website of interest to the `requests.get` method. We then use `BeautifulSoup` to parse the contents of the response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://www.usatoday.com/life/books/best-selling/'\n",
    "R = requests.get(url)\n",
    "R.raise_for_status()\n",
    "soup = BeautifulSoup(R.content, 'html.parser')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Navigate to the website and use the browser's developer tool to inspect items of interest."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data seems to be residing in a `div` tag. Let's search for that and the class info and see how many matches we get"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "booklist = soup.find_all('div', class_=\"front-booklist-info-container\")\n",
    "len(booklist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the text associated within each `<div>` book tag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 1Bloody GeniusbyJohn SandfordVirgil Flowers investigates the murder of a scholar at a local university; 12th in seriesGenre:General fictionDebuted:October 10 2019\n",
      "                            Buy Now\n",
      "                        \n",
      "                                    Buy Now\n",
      "                                AmazonBarnes & NobleGoogle PlayiBooksIndieboundKoboZola1Weeks Listed#1Best Week\n",
      "1 2BlowoutbyRachel MaddowSubtitle: \"Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth\"Genre:Current affairsDebuted:October 10 2019\n",
      "                            Buy Now\n",
      "                        \n",
      "                                    Buy Now\n",
      "                                AmazonBarnes & NobleGoogle PlayiBooksIndieboundKoboZola1Weeks Listed#2Best Week\n",
      "2 3Where the Crawdads SingbyDelia OwensThe reclusive Kya Clark is suspected in the death of Chase AndrewsGenre:General fictionDebuted:September 13 2018\n",
      "                            Buy Now\n",
      "                        \n",
      "                                    Buy Now\n",
      "                                AmazonBarnes & NobleGoogle PlayiBooksIndieboundKoboZola#8Last Week57Weeks Listed#1Best Week\n",
      "3 4The InstitutebyStephen KingLuke Ellis is whisked away to the Institute, where psychic kids are being held captive to save the worldGenre:General fictionDebuted:September 19 2019\n",
      "                            Buy Now\n",
      "                        \n",
      "                                    Buy Now\n",
      "                                AmazonBarnes & NobleGoogle PlayiBooksIndieboundKoboZola#6Last Week4Weeks Listed#1Best Week\n",
      "4 5The Water DancerbyTa-Nehisi CoatesHiram Walker is born with the gift, and a near drowning floods him with images of his ancestors – and a yearning to escape slaveryGenre:Fantasy/Sci-fiDebuted:October 03 2019\n",
      "                            Buy Now\n",
      "                        \n",
      "                                    Buy Now\n",
      "                                AmazonBarnes & NobleGoogle PlayiBooksIndieboundKoboZola#1Last Week2Weeks Listed#1Best Week\n",
      "5 6The Dutch HousebyAnn PatchettA doomed house, distant father and wicked stepmother forge an unbreakable bond between two siblingsGenre:General fictionDebuted:October 03 2019\n",
      "                            Buy Now\n",
      "                        \n",
      "                                    Buy Now\n",
      "                                AmazonBarnes & NobleGoogle PlayiBooksIndieboundKoboZola#9Last Week2Weeks Listed#6Best Week\n",
      "6 7Dog Man: For Whom the Ball RollsbyDav PilkeyYouth: Dog Man discovers he's the target of a new villain; seventh in seriesGenre:YouthDebuted:August 22 2019\n",
      "                            Buy Now\n",
      "                        \n",
      "                                    Buy Now\n",
      "                                AmazonBarnes & NobleGoogle PlayiBooksIndieboundKoboZola#11Last Week8Weeks Listed#1Best Week\n",
      "7 8The Book of Gutsy WomenbyHillary Rodham Clinton, Chelsea ClintonSubtitle: \"Favorite Stories of Courage and Resilience\"Genre:BiographyDebuted:October 10 2019\n",
      "                            Buy Now\n",
      "                        \n",
      "                                    Buy Now\n",
      "                                AmazonBarnes & NobleGoogle PlayiBooksIndieboundKoboZola1Weeks Listed#8Best Week\n",
      "8 9Lethal AgentbyVince Flynn, Kyle MillsMitch Rapp and Irene Kennedy investigate an ISIS terrorist threat; 16th in seriesGenre:General fictionDebuted:October 03 2019\n",
      "                            Buy Now\n",
      "                        \n",
      "                                    Buy Now\n",
      "                                AmazonBarnes & NobleGoogle PlayiBooksIndieboundKoboZola#2Last Week2Weeks Listed#2Best Week\n",
      "9 10GutsbyRaina TelgemeierYouth: Author explores how worries about her friends and school affect her physical healthGenre:YouthDebuted:September 26 2019\n",
      "                            Buy Now\n",
      "                        \n",
      "                                    Buy Now\n",
      "                                AmazonBarnes & NobleGoogle PlayiBooksIndieboundKoboZola#7Last Week3Weeks Listed#1Best Week\n"
     ]
    }
   ],
   "source": [
    "for i, book in enumerate(booklist):\n",
    "    print(i, book.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we want the title, author and genre. Let's use the browser to inspect which tags the data lies in. Then we'll use the `find_all` method to search for it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Bloody Genius, John Sandford, Genre:General fiction\n",
      "2. Blowout, Rachel Maddow, Genre:Current affairs\n",
      "3. Where the Crawdads Sing, Delia Owens, Genre:General fiction\n",
      "4. The Institute, Stephen King, Genre:General fiction\n",
      "5. The Water Dancer, Ta-Nehisi Coates, Genre:Fantasy/Sci-fi\n",
      "6. The Dutch House, Ann Patchett, Genre:General fiction\n",
      "7. Dog Man: For Whom the Ball Rolls, Dav Pilkey, Genre:Youth\n",
      "8. The Book of Gutsy Women, Hillary Rodham Clinton, Chelsea Clinton, Genre:Biography\n",
      "9. Lethal Agent, Vince Flynn, Kyle Mills, Genre:General fiction\n",
      "10. Guts, Raina Telgemeier, Genre:Youth\n"
     ]
    }
   ],
   "source": [
    "for rank, book in enumerate(booklist, start=1):\n",
    "    title = book.find_all('h3', class_='books-front-meta-title')[0].text\n",
    "    author = book.find_all('span', class_='books-front-meta-authorInfo')[0].text\n",
    "    genre = book.find_all('div', class_='books-front-meta-genre')[0].text\n",
    "    print(f'{rank}. {title}, {author}, {genre}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Webscraping DONE! At least for page one. To scrape the rest of the pages, let's navigate to the other pages. Notice anything different in the url?\n",
    "\n",
    "Before: https://www.usatoday.com/life/books/best-selling/  \n",
    "After: https://www.usatoday.com/life/books/best-selling/week/2019/41/page/2/\n",
    "\n",
    "The **before url** is the webpage for the current bestseller list, page 1. The **after url** is for a specific week and page. We can use this format to scrape any page for any given week. Much more useful than the generic url."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Bloody Genius, John Sandford, Genre:General fiction\n",
      "2. Blowout, Rachel Maddow, Genre:Current affairs\n",
      "3. Where the Crawdads Sing, Delia Owens, Genre:General fiction\n",
      "4. The Institute, Stephen King, Genre:General fiction\n",
      "5. The Water Dancer, Ta-Nehisi Coates, Genre:Fantasy/Sci-fi\n",
      "6. The Dutch House, Ann Patchett, Genre:General fiction\n",
      "7. Dog Man: For Whom the Ball Rolls, Dav Pilkey, Genre:Youth\n",
      "8. The Book of Gutsy Women, Hillary Rodham Clinton, Chelsea Clinton, Genre:Biography\n",
      "9. Lethal Agent, Vince Flynn, Kyle Mills, Genre:General fiction\n",
      "10. Guts, Raina Telgemeier, Genre:Youth\n",
      "11. Talking to Strangers, Malcolm Gladwell, Genre:Current affairs\n",
      "12. The United States of Trump, Bill O'Reilly, Genre:Current affairs\n",
      "13. Room on the Broom, Julia Donaldson, Axel Scheffler, Genre:Children\n",
      "14. The Testaments, Margaret Atwood, Genre:General fiction\n",
      "15. Stillness Is the Key, Ryan Holiday, Genre:Psychology/Self-help\n",
      "16. Inside Out, Demi Moore, Genre:Memoir\n",
      "17. The Trials of Apollo: The Tyrant's Tomb, Rick Riordan, Genre:Youth\n",
      "18. The Book of Dust: The Secret Commonwealth, Philip Pullman, Genre:Youth\n",
      "19. Educated, Tara Westover, Genre:Memoir\n",
      "20. Quantum, Patricia Cornwell, Genre:Mystery\n",
      "21. A Mrs. Miracle Christmas, Debbie Macomber, Genre:Romance\n",
      "22. The Great Alone, Kristin Hannah, Genre:General fiction\n",
      "23. Cilka's Journey, Heather Morris, Genre:General fiction\n",
      "24. Everything You Need, David Jeremiah, Genre:Religion/Inspiration\n",
      "25. White Knight, Meghan March, Genre:Romance\n",
      "26. On Tyranny, Timothy Snyder, Genre:Current affairs\n",
      "27. Pete the Cat: Trick or Pete, James Dean, Genre:Children\n",
      "28. Rhythms of Renewal, Rebekah Lyons, Genre:Religion/Inspiration\n",
      "29. Where Do I Begin, Elvis Duran, Genre:Memoir\n",
      "30. The Tattooist of Auschwitz, Heather Morris, Genre:General fiction\n"
     ]
    }
   ],
   "source": [
    "rank = 0\n",
    "data = []\n",
    "for page in range(1,4):    \n",
    "    url = f'http://www.usatoday.com/life/books/best-selling/week/2019/41/page/{page}/'\n",
    "    R = requests.get(url)\n",
    "    R.raise_for_status()\n",
    "    soup = BeautifulSoup(R.content, 'html.parser')\n",
    "    # find the element of interest matching our criteria\n",
    "    booklist = soup.find_all('div', class_='front-booklist-info-container')\n",
    "    for book in booklist:\n",
    "        title = book.find_all('h3', class_='books-front-meta-title')[0].text\n",
    "        author = book.find_all('span', class_='books-front-meta-authorInfo')[0].text\n",
    "        genre = book.find_all('div', class_='books-front-meta-genre')[0].text\n",
    "        rank += 1\n",
    "        print(f'{rank}. {title}, {author}, {genre}')\n",
    "        data.append((rank,title,author,genre))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's convert the data into a `pandas` dataframe with column headers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>rank</th>\n",
       "      <th>title</th>\n",
       "      <th>author</th>\n",
       "      <th>genre</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Bloody Genius</td>\n",
       "      <td>John Sandford</td>\n",
       "      <td>Genre:General fiction</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>Blowout</td>\n",
       "      <td>Rachel Maddow</td>\n",
       "      <td>Genre:Current affairs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>Where the Crawdads Sing</td>\n",
       "      <td>Delia Owens</td>\n",
       "      <td>Genre:General fiction</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>The Institute</td>\n",
       "      <td>Stephen King</td>\n",
       "      <td>Genre:General fiction</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>The Water Dancer</td>\n",
       "      <td>Ta-Nehisi Coates</td>\n",
       "      <td>Genre:Fantasy/Sci-fi</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>25</td>\n",
       "      <td>26</td>\n",
       "      <td>On Tyranny</td>\n",
       "      <td>Timothy Snyder</td>\n",
       "      <td>Genre:Current affairs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>26</td>\n",
       "      <td>27</td>\n",
       "      <td>Pete the Cat: Trick or Pete</td>\n",
       "      <td>James Dean</td>\n",
       "      <td>Genre:Children</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>27</td>\n",
       "      <td>28</td>\n",
       "      <td>Rhythms of Renewal</td>\n",
       "      <td>Rebekah Lyons</td>\n",
       "      <td>Genre:Religion/Inspiration</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>28</td>\n",
       "      <td>29</td>\n",
       "      <td>Where Do I Begin</td>\n",
       "      <td>Elvis Duran</td>\n",
       "      <td>Genre:Memoir</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>29</td>\n",
       "      <td>30</td>\n",
       "      <td>The Tattooist of Auschwitz</td>\n",
       "      <td>Heather Morris</td>\n",
       "      <td>Genre:General fiction</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>30 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    rank                        title            author  \\\n",
       "0      1                Bloody Genius     John Sandford   \n",
       "1      2                      Blowout     Rachel Maddow   \n",
       "2      3      Where the Crawdads Sing       Delia Owens   \n",
       "3      4                The Institute      Stephen King   \n",
       "4      5             The Water Dancer  Ta-Nehisi Coates   \n",
       "..   ...                          ...               ...   \n",
       "25    26                   On Tyranny    Timothy Snyder   \n",
       "26    27  Pete the Cat: Trick or Pete        James Dean   \n",
       "27    28           Rhythms of Renewal     Rebekah Lyons   \n",
       "28    29             Where Do I Begin       Elvis Duran   \n",
       "29    30   The Tattooist of Auschwitz    Heather Morris   \n",
       "\n",
       "                         genre  \n",
       "0        Genre:General fiction  \n",
       "1        Genre:Current affairs  \n",
       "2        Genre:General fiction  \n",
       "3        Genre:General fiction  \n",
       "4         Genre:Fantasy/Sci-fi  \n",
       "..                         ...  \n",
       "25       Genre:Current affairs  \n",
       "26              Genre:Children  \n",
       "27  Genre:Religion/Inspiration  \n",
       "28                Genre:Memoir  \n",
       "29       Genre:General fiction  \n",
       "\n",
       "[30 rows x 4 columns]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topsellers = pd.DataFrame(data, columns=['rank','title','author','genre'])\n",
    "topsellers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Modify the code below to scrape the entire list and add columns such as when it debuted and how long its been on the charts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Bloody Genius, John Sandford, Genre:General fiction\n",
      "2. Blowout, Rachel Maddow, Genre:Current affairs\n",
      "3. Where the Crawdads Sing, Delia Owens, Genre:General fiction\n",
      "4. The Institute, Stephen King, Genre:General fiction\n",
      "5. The Water Dancer, Ta-Nehisi Coates, Genre:Fantasy/Sci-fi\n",
      "6. The Dutch House, Ann Patchett, Genre:General fiction\n",
      "7. Dog Man: For Whom the Ball Rolls, Dav Pilkey, Genre:Youth\n",
      "8. The Book of Gutsy Women, Hillary Rodham Clinton, Chelsea Clinton, Genre:Biography\n",
      "9. Lethal Agent, Vince Flynn, Kyle Mills, Genre:General fiction\n",
      "10. Guts, Raina Telgemeier, Genre:Youth\n",
      "11. Talking to Strangers, Malcolm Gladwell, Genre:Current affairs\n",
      "12. The United States of Trump, Bill O'Reilly, Genre:Current affairs\n",
      "13. Room on the Broom, Julia Donaldson, Axel Scheffler, Genre:Children\n",
      "14. The Testaments, Margaret Atwood, Genre:General fiction\n",
      "15. Stillness Is the Key, Ryan Holiday, Genre:Psychology/Self-help\n",
      "16. Inside Out, Demi Moore, Genre:Memoir\n",
      "17. The Trials of Apollo: The Tyrant's Tomb, Rick Riordan, Genre:Youth\n",
      "18. The Book of Dust: The Secret Commonwealth, Philip Pullman, Genre:Youth\n",
      "19. Educated, Tara Westover, Genre:Memoir\n",
      "20. Quantum, Patricia Cornwell, Genre:Mystery\n",
      "21. A Mrs. Miracle Christmas, Debbie Macomber, Genre:Romance\n",
      "22. The Great Alone, Kristin Hannah, Genre:General fiction\n",
      "23. Cilka's Journey, Heather Morris, Genre:General fiction\n",
      "24. Everything You Need, David Jeremiah, Genre:Religion/Inspiration\n",
      "25. White Knight, Meghan March, Genre:Romance\n",
      "26. On Tyranny, Timothy Snyder, Genre:Current affairs\n",
      "27. Pete the Cat: Trick or Pete, James Dean, Genre:Children\n",
      "28. Rhythms of Renewal, Rebekah Lyons, Genre:Religion/Inspiration\n",
      "29. Where Do I Begin, Elvis Duran, Genre:Memoir\n",
      "30. The Tattooist of Auschwitz, Heather Morris, Genre:General fiction\n"
     ]
    }
   ],
   "source": [
    "rank = 0\n",
    "data = []\n",
    "for page in range(1,4):    \n",
    "    url = r'http://www.usatoday.com/life/books/best-selling/week/2019/41/page/{}/'.format(page)    \n",
    "    R = requests.get(url)\n",
    "    R.raise_for_status()\n",
    "    soup = BeautifulSoup(R.content, 'html.parser')\n",
    "    # find the element of interest matching our criteria\n",
    "    booklist = soup.find_all('div', class_='front-booklist-info-container')\n",
    "    for book in booklist:\n",
    "        title = book.find_all('h3', class_='books-front-meta-title')[0].text\n",
    "        author = book.find_all('span', class_='books-front-meta-authorInfo')[0].text\n",
    "        genre = book.find_all('div', class_='books-front-meta-genre')[0].text\n",
    "        rank += 1\n",
    "        print(f'{rank}. {title}, {author}, {genre}')\n",
    "        data.append((rank,title,author,genre))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example 3: Geocoding (using an API) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example shows how to use an API to gather data of interest. Sometimes, you will need to get an API key or access token to access the website. The website should have a developers or API section to let you know how to query the API with the appropriate parameters. This section will also detail the terms of usage and any usage limits on using the API. APIs usually follow a freemium (free for a little stuff, pay for more stuff) business model. \n",
    "\n",
    "We'll show you how to geocode addresses using the Google Maps Geocoding API. Let's look at the documentation quickly.    \n",
    "https://developers.google.com/maps/documentation/geocoding/intro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The documentation tells us the format looks something like this\n",
    "\n",
    "`https://maps.googleapis.com/maps/api/geocode/outputFormat?parameters`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In summary, the documentation tells us about:\n",
    "1. Needing an API key.\n",
    "2. Format of the request needed\n",
    "2. Choosing an output format\n",
    "3. Required parameters\n",
    "4. Optional parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: Starting June 2018, Google requires you to enable billing to use its Google Maps APIs. It gives you $200/month of free usage before billing starts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Query String Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A url that contains a query string will contain three parts:\n",
    "1. Resource (base) URL\n",
    "2. Question Mark (?)\n",
    "3. Parameters (key=value pairs) separated by an ampersand (&)\n",
    "\n",
    "Let's look at the Google geocoding example with our own API key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://maps.googleapis.com/maps/api/geocode/json?address=915+E+Washington%2C+Ann+Arbor&key=AIzaSyCmXigJXZovZrP16Yjzn8i7XbozDoibO9A'"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "apikey = 'AIzaSyBaneriwLXc9UQy5pNGF0sThcQ9sfMUibQ'\n",
    "url = f'https://maps.googleapis.com/maps/api/geocode/json?address=915+E+Washington%2C+Ann+Arbor&key={apikey}'\n",
    "url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's break down the different pieces for this example.\n",
    "1. The resource url is `https://maps.googleapis.com/maps/api/geocode/json`\n",
    "2. The output format chosen was json\n",
    "3. `?`\n",
    "4. First required parameter is `address=1600+Amphitheatre+Parkway,+Mountain+View,+CA`\n",
    "5. `&`\n",
    "6. Second required parameter is `key=<YOUR API KEY>`\n",
    "7. No optional key/value pair"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that some characters can not be part of the URL like spaces in the above example. Spaces are encoded as `+` or `%20`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now can make a `GET` request. Same as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'results': [{'address_components': [{'long_name': '915',\n",
       "     'short_name': '915',\n",
       "     'types': ['street_number']},\n",
       "    {'long_name': 'East Washington Street',\n",
       "     'short_name': 'E Washington St',\n",
       "     'types': ['route']},\n",
       "    {'long_name': 'Burns Park',\n",
       "     'short_name': 'Burns Park',\n",
       "     'types': ['neighborhood', 'political']},\n",
       "    {'long_name': 'Ann Arbor',\n",
       "     'short_name': 'Ann Arbor',\n",
       "     'types': ['locality', 'political']},\n",
       "    {'long_name': 'Washtenaw County',\n",
       "     'short_name': 'Washtenaw County',\n",
       "     'types': ['administrative_area_level_2', 'political']},\n",
       "    {'long_name': 'Michigan',\n",
       "     'short_name': 'MI',\n",
       "     'types': ['administrative_area_level_1', 'political']},\n",
       "    {'long_name': 'United States',\n",
       "     'short_name': 'US',\n",
       "     'types': ['country', 'political']},\n",
       "    {'long_name': '48109', 'short_name': '48109', 'types': ['postal_code']},\n",
       "    {'long_name': '1070',\n",
       "     'short_name': '1070',\n",
       "     'types': ['postal_code_suffix']}],\n",
       "   'formatted_address': '915 E Washington St, Ann Arbor, MI 48109, USA',\n",
       "   'geometry': {'location': {'lat': 42.2808083, 'lng': -83.7382583},\n",
       "    'location_type': 'ROOFTOP',\n",
       "    'viewport': {'northeast': {'lat': 42.2821572802915,\n",
       "      'lng': -83.73690931970849},\n",
       "     'southwest': {'lat': 42.2794593197085, 'lng': -83.7396072802915}}},\n",
       "   'place_id': 'ChIJG0N-sUGuPIgRhWJ5XDKXPt4',\n",
       "   'plus_code': {'compound_code': '77J6+8M Ann Arbor, Michigan, United States',\n",
       "    'global_code': '86JR77J6+8M'},\n",
       "   'types': ['street_address']}],\n",
       " 'status': 'OK'}"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "R = requests.get(url)\n",
    "response = R.json()\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Webscraping done! API conquered!\n",
    "\n",
    "Manually encoding strings for URLs can be a pain. Thankfully, the `requests` library takes care of all this for us if we pass a dictionary to the `params` keyword argument.\n",
    "\n",
    "First we need to construct the dictionary for the parameters and the specify the resource url."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'address': '915 E Washington, Ann Arbor',\n",
    "          'key':apikey\n",
    "         }\n",
    "baseurl = 'https://maps.googleapis.com/maps/api/geocode/json'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make a `GET` request with the params keyword argument"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'results': [{'address_components': [{'long_name': '915',\n",
       "     'short_name': '915',\n",
       "     'types': ['street_number']},\n",
       "    {'long_name': 'East Washington Street',\n",
       "     'short_name': 'E Washington St',\n",
       "     'types': ['route']},\n",
       "    {'long_name': 'Burns Park',\n",
       "     'short_name': 'Burns Park',\n",
       "     'types': ['neighborhood', 'political']},\n",
       "    {'long_name': 'Ann Arbor',\n",
       "     'short_name': 'Ann Arbor',\n",
       "     'types': ['locality', 'political']},\n",
       "    {'long_name': 'Washtenaw County',\n",
       "     'short_name': 'Washtenaw County',\n",
       "     'types': ['administrative_area_level_2', 'political']},\n",
       "    {'long_name': 'Michigan',\n",
       "     'short_name': 'MI',\n",
       "     'types': ['administrative_area_level_1', 'political']},\n",
       "    {'long_name': 'United States',\n",
       "     'short_name': 'US',\n",
       "     'types': ['country', 'political']},\n",
       "    {'long_name': '48109', 'short_name': '48109', 'types': ['postal_code']},\n",
       "    {'long_name': '1070',\n",
       "     'short_name': '1070',\n",
       "     'types': ['postal_code_suffix']}],\n",
       "   'formatted_address': '915 E Washington St, Ann Arbor, MI 48109, USA',\n",
       "   'geometry': {'location': {'lat': 42.2808083, 'lng': -83.7382583},\n",
       "    'location_type': 'ROOFTOP',\n",
       "    'viewport': {'northeast': {'lat': 42.2821572802915,\n",
       "      'lng': -83.73690931970849},\n",
       "     'southwest': {'lat': 42.2794593197085, 'lng': -83.7396072802915}}},\n",
       "   'place_id': 'ChIJG0N-sUGuPIgRhWJ5XDKXPt4',\n",
       "   'plus_code': {'compound_code': '77J6+8M Ann Arbor, Michigan, United States',\n",
       "    'global_code': '86JR77J6+8M'},\n",
       "   'types': ['street_address']}],\n",
       " 'status': 'OK'}"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "R = requests.get(baseurl, params=params)\n",
    "R.raise_for_status()\n",
    "response = R.json()\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same result. And we can also spy the actual url that was sent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://maps.googleapis.com/maps/api/geocode/json?address=915+E+Washington%2C+Ann+Arbor&key=AIzaSyCmXigJXZovZrP16Yjzn8i7XbozDoibO9A'"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "R.url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it! Pretty simple, eh! A lot of APIs work just like this. Some, of course, are a bit more complicated. \n",
    "\n",
    "**Note:** CSCAR occasionally runs workshops on using social media APIs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Traversing a JSON object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, that we have a JSON response from the API, we need to know how to parse it for the information we are looking for. A JSON object behaves like a Python dictionary in that it consists of key-value pairs. JSON objects consist of dictionaries and lists."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To look at the variables in a JSON object hierarchically, use the `keys` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['results', 'status'])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Access the key like you would a Python dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'address_components': [{'long_name': '915',\n",
       "    'short_name': '915',\n",
       "    'types': ['street_number']},\n",
       "   {'long_name': 'East Washington Street',\n",
       "    'short_name': 'E Washington St',\n",
       "    'types': ['route']},\n",
       "   {'long_name': 'Burns Park',\n",
       "    'short_name': 'Burns Park',\n",
       "    'types': ['neighborhood', 'political']},\n",
       "   {'long_name': 'Ann Arbor',\n",
       "    'short_name': 'Ann Arbor',\n",
       "    'types': ['locality', 'political']},\n",
       "   {'long_name': 'Washtenaw County',\n",
       "    'short_name': 'Washtenaw County',\n",
       "    'types': ['administrative_area_level_2', 'political']},\n",
       "   {'long_name': 'Michigan',\n",
       "    'short_name': 'MI',\n",
       "    'types': ['administrative_area_level_1', 'political']},\n",
       "   {'long_name': 'United States',\n",
       "    'short_name': 'US',\n",
       "    'types': ['country', 'political']},\n",
       "   {'long_name': '48109', 'short_name': '48109', 'types': ['postal_code']},\n",
       "   {'long_name': '1070',\n",
       "    'short_name': '1070',\n",
       "    'types': ['postal_code_suffix']}],\n",
       "  'formatted_address': '915 E Washington St, Ann Arbor, MI 48109, USA',\n",
       "  'geometry': {'location': {'lat': 42.2808083, 'lng': -83.7382583},\n",
       "   'location_type': 'ROOFTOP',\n",
       "   'viewport': {'northeast': {'lat': 42.2821572802915,\n",
       "     'lng': -83.73690931970849},\n",
       "    'southwest': {'lat': 42.2794593197085, 'lng': -83.7396072802915}}},\n",
       "  'place_id': 'ChIJG0N-sUGuPIgRhWJ5XDKXPt4',\n",
       "  'plus_code': {'compound_code': '77J6+8M Ann Arbor, Michigan, United States',\n",
       "   'global_code': '86JR77J6+8M'},\n",
       "  'types': ['street_address']}]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response['results']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Occasionally, you will encounter a `list` of key-value pairs in the hierarchy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You would access the `list` the same way as a Python list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'address_components': [{'long_name': '915',\n",
       "   'short_name': '915',\n",
       "   'types': ['street_number']},\n",
       "  {'long_name': 'East Washington Street',\n",
       "   'short_name': 'E Washington St',\n",
       "   'types': ['route']},\n",
       "  {'long_name': 'Burns Park',\n",
       "   'short_name': 'Burns Park',\n",
       "   'types': ['neighborhood', 'political']},\n",
       "  {'long_name': 'Ann Arbor',\n",
       "   'short_name': 'Ann Arbor',\n",
       "   'types': ['locality', 'political']},\n",
       "  {'long_name': 'Washtenaw County',\n",
       "   'short_name': 'Washtenaw County',\n",
       "   'types': ['administrative_area_level_2', 'political']},\n",
       "  {'long_name': 'Michigan',\n",
       "   'short_name': 'MI',\n",
       "   'types': ['administrative_area_level_1', 'political']},\n",
       "  {'long_name': 'United States',\n",
       "   'short_name': 'US',\n",
       "   'types': ['country', 'political']},\n",
       "  {'long_name': '48109', 'short_name': '48109', 'types': ['postal_code']},\n",
       "  {'long_name': '1070',\n",
       "   'short_name': '1070',\n",
       "   'types': ['postal_code_suffix']}],\n",
       " 'formatted_address': '915 E Washington St, Ann Arbor, MI 48109, USA',\n",
       " 'geometry': {'location': {'lat': 42.2808083, 'lng': -83.7382583},\n",
       "  'location_type': 'ROOFTOP',\n",
       "  'viewport': {'northeast': {'lat': 42.2821572802915,\n",
       "    'lng': -83.73690931970849},\n",
       "   'southwest': {'lat': 42.2794593197085, 'lng': -83.7396072802915}}},\n",
       " 'place_id': 'ChIJG0N-sUGuPIgRhWJ5XDKXPt4',\n",
       " 'plus_code': {'compound_code': '77J6+8M Ann Arbor, Michigan, United States',\n",
       "  'global_code': '86JR77J6+8M'},\n",
       " 'types': ['street_address']}"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response['results'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, there is only one element in this list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to drill deeper into the `results` key, you would repeat the process to look at available keys and so forth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['address_components', 'formatted_address', 'geometry', 'place_id', 'plus_code', 'types'])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response['results'][0].keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To arrive at the latitude coordinate, you would need the following syntax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42.2808083"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response['results'][0]['geometry']['location']['lat']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: If you have a keen eye, you can just eyeball the hierarchy without needing to use the `keys` method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example 4: Weather Data (using an API)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is an example of an API that doesn't follow the query string parameter format.\n",
    "\n",
    "We will be using <a href='https://darksky.net/app/'>https://darksky.net/app/</a> to gather weather data.  The developer section is at https://darksky.net/dev/. The API documentation is at https://darksky.net/dev/docs. You will need to create an account to get an API key (or you can borrow mine).\n",
    "\n",
    "**Note**: Dark Sky is a relatively simple API (part of the reason why I'm using it as an example). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Weather Forecast"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "APIs usually have different endpoints depending on the data you are interested in. The documentation page shows that you can make two types of API requests. \n",
    "1. The current weather forecast for the next week (forecast request)\n",
    "2. An observed or forecast weather conditions for a date in the past or future (time machine request)\n",
    "\n",
    "**Note**: BTW, two is a relatively small number. The twitter API has over 100.\n",
    "\n",
    "Let's look at endpoint #2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The documentation shows that a forecast request takes the form of:\n",
    "\n",
    "https://api.darksky.net/forecast/[key]/[latitude],[longitude],[time]\n",
    "\n",
    "A historical weather request returns the observed weather at a given time (for many places, up to 60 years in the past)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start by specifying the API key and then the GPS coordinates and timestamp."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "apikey = 'b80fad6bb807f251eff7b0e5cca2ca21'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "latitude = 42.28\n",
    "longitude = -83.74\n",
    "time = '2017-01-31T16:00:00'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "url = f'https://api.darksky.net/forecast/{apikey}/{latitude},{longitude},{time}'\n",
    "R = requests.get(url)\n",
    "R.raise_for_status()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "API responses consist of a JSON-formatted object (UTF-8) according to the documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "pastforecast = R.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Webscraping DONE!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hourly summary and temperature for that day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Overcast 23.11\n",
      "1 Foggy 23.74\n",
      "2 Foggy 24.37\n",
      "3 Foggy 24.85\n",
      "4 Foggy 25.39\n",
      "5 Foggy 25.96\n",
      "6 Foggy 26.29\n",
      "7 Overcast 26.23\n",
      "8 Overcast 26.45\n",
      "9 Overcast 27.3\n",
      "10 Overcast 28.36\n",
      "11 Overcast 29.75\n",
      "12 Overcast 30.92\n",
      "13 Overcast 32.4\n",
      "14 Overcast 32.64\n",
      "15 Overcast 32.67\n",
      "16 Overcast 32.79\n",
      "17 Possible Flurries 32.22\n",
      "18 Overcast 31.31\n",
      "19 Overcast 31.11\n",
      "20 Overcast 31.21\n",
      "21 Overcast 31.31\n",
      "22 Mostly Cloudy 31.33\n",
      "23 Mostly Cloudy 31.41\n"
     ]
    }
   ],
   "source": [
    "for i, hr in enumerate(pastforecast[\"hourly\"][\"data\"]):\n",
    "    print(i, hr['summary'], hr['temperature'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: API responses come in a variety of flavours. Some common ones are JSON, XML, and CSV format. I recommend choosing JSON over XML whenever possible. XML is a pain to work with IMO. If you do choose XML, you can use `BeautifulSoup` or `ElementTree` to parse the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## API Wrappers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the more popular APIs (like Twitter and Google), the open source community has probably written a Python wrapper for the API to abstract away some of the details for you. The Dark Sky API is pretty simple as you can see but someone has written a wrapper for it because of its popularity. The Python module is called `python-forecastio`. You can install it using `pip install python-forecastio`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "import forecastio\n",
    "\n",
    "lon = -83.7\n",
    "lat = 42.3\n",
    "forecast = forecastio.load_forecast(apikey, lat, lon)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accessing the data depends on the structure the developer set up which is not necessarily the same as the API. You will need to read the documentation of the module to understand the structure or figure it out by trial and error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 2019-10-15 19:00:00 Partly Cloudy 61.38\n",
      "1 2019-10-15 20:00:00 Possible Drizzle 62.41\n",
      "2 2019-10-15 21:00:00 Partly Cloudy 63.43\n",
      "3 2019-10-15 22:00:00 Partly Cloudy 63.64\n",
      "4 2019-10-15 23:00:00 Mostly Cloudy 60.42\n",
      "5 2019-10-16 00:00:00 Mostly Cloudy 57.06\n",
      "6 2019-10-16 01:00:00 Overcast 57.09\n",
      "7 2019-10-16 02:00:00 Mostly Cloudy 57.98\n",
      "8 2019-10-16 03:00:00 Mostly Cloudy 57.87\n",
      "9 2019-10-16 04:00:00 Mostly Cloudy 57.64\n",
      "10 2019-10-16 05:00:00 Possible Light Rain 57.49\n",
      "11 2019-10-16 06:00:00 Possible Light Rain 56.27\n",
      "12 2019-10-16 07:00:00 Possible Light Rain 53.41\n",
      "13 2019-10-16 08:00:00 Overcast 50.4\n",
      "14 2019-10-16 09:00:00 Mostly Cloudy 47.65\n",
      "15 2019-10-16 10:00:00 Overcast 45.1\n",
      "16 2019-10-16 11:00:00 Overcast 43.11\n",
      "17 2019-10-16 12:00:00 Overcast 41.87\n",
      "18 2019-10-16 13:00:00 Mostly Cloudy 42.67\n",
      "19 2019-10-16 14:00:00 Mostly Cloudy 44.72\n",
      "20 2019-10-16 15:00:00 Mostly Cloudy 46.8\n",
      "21 2019-10-16 16:00:00 Mostly Cloudy 48.84\n",
      "22 2019-10-16 17:00:00 Mostly Cloudy 50.28\n",
      "23 2019-10-16 18:00:00 Mostly Cloudy 50.83\n",
      "24 2019-10-16 19:00:00 Mostly Cloudy 50.29\n",
      "25 2019-10-16 20:00:00 Mostly Cloudy 49.13\n",
      "26 2019-10-16 21:00:00 Mostly Cloudy 47.68\n",
      "27 2019-10-16 22:00:00 Mostly Cloudy 47.39\n",
      "28 2019-10-16 23:00:00 Overcast 46.74\n",
      "29 2019-10-17 00:00:00 Overcast 46.3\n",
      "30 2019-10-17 01:00:00 Overcast 45.79\n",
      "31 2019-10-17 02:00:00 Overcast 45.37\n",
      "32 2019-10-17 03:00:00 Overcast 45.01\n",
      "33 2019-10-17 04:00:00 Overcast 44.61\n",
      "34 2019-10-17 05:00:00 Overcast 44.09\n",
      "35 2019-10-17 06:00:00 Overcast 43.67\n",
      "36 2019-10-17 07:00:00 Overcast 43.17\n",
      "37 2019-10-17 08:00:00 Mostly Cloudy 42.41\n",
      "38 2019-10-17 09:00:00 Mostly Cloudy 41.73\n",
      "39 2019-10-17 10:00:00 Mostly Cloudy 40.93\n",
      "40 2019-10-17 11:00:00 Mostly Cloudy 39.94\n",
      "41 2019-10-17 12:00:00 Mostly Cloudy 39.5\n",
      "42 2019-10-17 13:00:00 Mostly Cloudy 40.21\n",
      "43 2019-10-17 14:00:00 Mostly Cloudy 41.45\n",
      "44 2019-10-17 15:00:00 Mostly Cloudy 43.21\n",
      "45 2019-10-17 16:00:00 Mostly Cloudy 45.33\n",
      "46 2019-10-17 17:00:00 Mostly Cloudy 47\n",
      "47 2019-10-17 18:00:00 Mostly Cloudy 48.81\n",
      "48 2019-10-17 19:00:00 Mostly Cloudy 50.49\n"
     ]
    }
   ],
   "source": [
    "for i, hr in enumerate(forecast.hourly().data):\n",
    "    print(i, hr.time, hr.summary, hr.temperature)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the Google Maps Distance Matrix API, get the the distance from Ann Arbor, MI to San Diego, CA and Anchorage, AK.\n",
    "\n",
    "The documentation is located at https://developers.google.com/maps/documentation/distance-matrix/start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 5: POST Requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example was taken from the book **Web Scraping with Python** by Ryan Mitchell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This website http://pythonscraping.com/pages/files/form.html shows a basic web form. Let's look at the page source to see info related to the `post` request."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The page source contains the variable names of the two input fields which need to be submitted in the `<form>` tag. We create a dictionary to represent these fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {'firstname':'Mister',\n",
    "        'lastname' :'Cao'}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is an `action` attribute associated with the `post` request. This is the url where the `post` is being sent. This is a **relative** path to the current url. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hello there, Mister Cao!'"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "R = requests.post(\"http://pythonscraping.com/pages/files/processing.php\", data=data)\n",
    "R.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll see in the next section a way to get at the same information through the browser's developer tool."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 6: Farm Equipment Crashes in Ann Arbor (data that is not visible in the page source)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example is meant to illustrate how to grab data that is visible on the webpage but not in the page source. The website of interest is https://www.michigantrafficcrashfacts.org/querytool. We are interested in grabbing the gps coordinates of the crashes on the map. Here is our query to generate the map https://www.michigantrafficcrashfacts.org/querytool#q1;0;2017,2016,2015,2014,2013;c8189;0,42:1. The crashes also has some information in a popup tooltip when you click on it.\n",
    "\n",
    "The goal is to find the URL where the GET/POST request is being sent. How do we do that? The answer lies in the reference link at the bottom of this example. Basically, you need your browser's developer tool and some detective work.  \n",
    "[SHORT OVERLUDE TO THE WEBPAGE]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congrats! You can consider yourself a developer now that you can use the toolbox :)\n",
    "\n",
    "Now that we have found the URL of interest, we go back to our regularly scheduled programming (pun intended). Everything should be easy peasy moving forward. We know the url where the `post` request is being sent. There is also some data being sent with it. We will create a dictionary for that bit of data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "mtcf_url = 'https://www.michigantrafficcrashfacts.org/qjson'\n",
    "query = {'q': '1;0;2017,2016,2015,2014,2013;c8189;0,42:1',\n",
    "        'v': 'map',\n",
    "        'p': '37.341651,-98.030605,50.964815,-73.289395,6',\n",
    "}\n",
    "R = requests.post(mtcf_url, data=query)\n",
    "R.raise_for_status()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grab json response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'warnings': [],\n",
       " 'errors': [],\n",
       " 'meta': {'count': 2,\n",
       "  'bounds': {'west': -83.78405039994,\n",
       "   'east': -83.762084436151,\n",
       "   'south': 42.2444715503,\n",
       "   'north': 42.281289282301},\n",
       "  'mode': 'points'},\n",
       " 'headers': {'crash.id': 'Crash ID',\n",
       "  'crash_day': 'Crash Day',\n",
       "  'crash_month': 'Crash Month',\n",
       "  'crash_year': 'Crash Year',\n",
       "  'crash_time_of_day': 'Crash Hour',\n",
       "  'crash_worst_injury': 'Crash Worst Injury',\n",
       "  'lng': 'Crash Lat',\n",
       "  'lat': 'Crash Long',\n",
       "  'seg_crnt': 'Road Segment',\n",
       "  'seg_orig': 'Orig Segment',\n",
       "  'vehicle_id': 'Unit Count',\n",
       "  'person_id': 'Person Count'},\n",
       " '0': {'crash.id': 2015113037,\n",
       "  'crash_day': 26,\n",
       "  'crash_month': 'February',\n",
       "  'crash_year': 2015,\n",
       "  'crash_time_of_day': '8:00 AM - 9:00 AM',\n",
       "  'crash_worst_injury': 'No injury (O)',\n",
       "  'lng': -83.78405039994,\n",
       "  'lat': 42.281289282301,\n",
       "  'seg_crnt': 4818100,\n",
       "  'seg_orig': 0,\n",
       "  'vehicle_id': 2,\n",
       "  'person_id': 2},\n",
       " '1': {'crash.id': 2015156119,\n",
       "  'crash_day': 14,\n",
       "  'crash_month': 'July',\n",
       "  'crash_year': 2015,\n",
       "  'crash_time_of_day': '5:00 PM - 6:00 PM',\n",
       "  'crash_worst_injury': 'Possible injury (C)',\n",
       "  'lng': -83.762084436151,\n",
       "  'lat': 42.2444715503,\n",
       "  'seg_crnt': 10003762,\n",
       "  'seg_orig': 0,\n",
       "  'vehicle_id': 3,\n",
       "  'person_id': 8},\n",
       " 'uid': '651a28bfd52b9d4eec4de4874e46f3af8813c8f9'}"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "crash_data = R.json()\n",
    "crash_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Webscraping DONE!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reference\n",
    "\n",
    "http://www.gregreda.com/2015/02/15/web-scraping-finding-the-api/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This website generates random VINs upon demand http://randomvin.com. This is probably one of the most simplest html page you will ever see (and one of the reasons I picked it)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is some example code to grab 3 VINs from the website. Supply the appropriate url to finish the code. What file format is the response?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for i in range(3):\n",
    "    R = requests.get('')\n",
    "    R.raise_for_status()\n",
    "    print(R.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Retrying Module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This module is aptly name. Even though you have written valid code, sometimes it will still crash for unforseen reasons (e.g. bad network connection). This is where the `retry` behaviour becomes handy. You will need to install the module first via `pip install retrying`. Documentation is at https://pypi.python.org/pypi/retrying."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "from retrying import retry"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Retry"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a simple case of how to use `retry`. First step is to put your code of interest in a function. This non-sensical function has a 90% chance of failing when run because of the `assert` statement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n"
     ]
    },
    {
     "ename": "AssertionError",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mAssertionError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-74-b7b5add69536>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      4\u001b[0m     \u001b[1;32massert\u001b[0m \u001b[0mB\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;36m7\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 6\u001b[1;33m \u001b[0mis_B_equal_to_lucky7\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      7\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'this has a 10% chance of printing'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-74-b7b5add69536>\u001b[0m in \u001b[0;36mis_B_equal_to_lucky7\u001b[1;34m()\u001b[0m\n\u001b[0;32m      2\u001b[0m     \u001b[0mB\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mrandom\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrandint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m10\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m     \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mB\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 4\u001b[1;33m     \u001b[1;32massert\u001b[0m \u001b[0mB\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;36m7\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      5\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m \u001b[0mis_B_equal_to_lucky7\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mAssertionError\u001b[0m: "
     ]
    }
   ],
   "source": [
    "def is_B_equal_to_lucky7():\n",
    "    B = random.randint(1, 10)\n",
    "    print(B)\n",
    "    assert B == 7\n",
    "\n",
    "is_B_equal_to_lucky7()\n",
    "print('this has a 10% chance of printing')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's add the `retry` behavior by adding an @ decorator at the beginning of the function. That's it! Pretty simple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "7\n",
      "this will ALWAYS print\n"
     ]
    }
   ],
   "source": [
    "@retry()\n",
    "def is_B_equal_to_lucky7():\n",
    "    B = random.randint(1, 10)\n",
    "    print(B)\n",
    "    assert B == 7\n",
    "    \n",
    "is_B_equal_to_lucky7()\n",
    "print('this will ALWAYS print')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also add an argument `wait_fixed` in milliseconds to specify how long to wait between retries. Good practice so you don't have to bombard the server with constant requests during a failed connection during webscraping. Gmail does an exponential version of this when it loses the network connection."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retry on specific or general exceptions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We saw the function fail earlier because of an `AssertionError`. We can tell `retry` to only retry when certain exceptions occur.  This requires using the argument `retry_on_exception` and passing it the name of a function. The function will return either `True` or `False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n",
      "7\n",
      "Lucky 7\n"
     ]
    }
   ],
   "source": [
    "def checkForSpecificError(exception):\n",
    "    return isinstance(exception, AssertionError)\n",
    "\n",
    "@retry(retry_on_exception=checkForSpecificError, wait_fixed=500)\n",
    "def is_it_lucky7():\n",
    "    B = random.randint(1, 10)\n",
    "    print(B)\n",
    "    assert B == 7\n",
    "\n",
    "is_it_lucky7()\n",
    "print(\"Lucky 7\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we had changed the last line in the function to `assert C == 7`, then the retry behaviour will not kick in because the function returns a `NameError`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retry on return value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You don't necessarily need to have a program error to invoke the `retry` behaviour. You can use the `return` value to decide. This requires using the argument `retry_on_result` and passing it the name of a function. The function will return either `True` or `False`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n",
      "3\n",
      "10\n",
      "6\n",
      "2\n",
      "2\n",
      "7\n",
      "Did I find a 7?\n"
     ]
    }
   ],
   "source": [
    "def checkReturnValue(value):\n",
    "    return value is True\n",
    "\n",
    "# This function will never crash but we can still use retry\n",
    "@retry(retry_on_result=checkReturnValue, wait_fixed=300)\n",
    "def main():\n",
    "    B = random.randint(1, 10)\n",
    "    print(B)\n",
    "    try:\n",
    "        assert B == 7\n",
    "        return None\n",
    "    except AssertionError:\n",
    "        return True\n",
    "\n",
    "main()\n",
    "print('Did I find a 7?')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other arguments of interest to `retry` which you can use are:  \n",
    "- stop_max_attempt_number\n",
    "- stop_max_delay\n",
    "- wait_random_min\n",
    "- wait_random_max\n",
    "- wait_exponential_multiplier\n",
    "- wait_exponential_max`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: You could implement the retry behavior without this module. You can use a `while` loop in some combination with `try` and `except` too. I don't recommend it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply the `retry` module to this flaky code for the Dark Sky API to try 5 times waiting one second in between attempts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_weather_from_Eastern_hemisphere(apikey):\n",
    "    lon = random.randint(0,400)\n",
    "    lat = random.randint(0,200)\n",
    "    print(f'{lon} deg E, {lat} deg N')\n",
    "    R = requests.get(f'https://api.darksky.net/forecast/{apikey}/{lat},{lon}')\n",
    "    R.raise_for_status()\n",
    "    return (lon,lat,R.json())\n",
    "    \n",
    "apikey = 'b80fad6bb807f251eff7b0e5cca2ca21'\n",
    "lon, lat, forecast = get_weather_from_Eastern_hemisphere(apikey)\n",
    "print(forecast[\"daily\"][\"data\"][0]['summary'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quiz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://goo.gl/forms/Ym3hbKu45nzQ2puv2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# cURL to Python Requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "cURL is a command line tool for getting or sending files using URL syntax. You can always get the cURL command from the developer tool. This usually also includes *cookies and headers*. While Python can't use it directly, you can convert it to a Python requests syntax. Googling *curl to python requests* will bring you to this page https://curl.trillworks.com/. You can paste the curl code and it will return the equivalent Python code for you. Python does have a module that is suppose to do this for you but I haven't gotten it to work yet. Some modules are `uncurl, runcurl, curl_to_requests` but they support Python2 only.\n",
    "\n",
    "For example, here is what is returned for the curl cmd from http://random.vin.com."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "cookies = {\n",
    "    '_ga': 'GA1.2.690271399.1483585728',\n",
    "    '_gat': '1',\n",
    "}\n",
    "\n",
    "headers = {\n",
    "    'Accept-Encoding': 'gzip, deflate, sdch',\n",
    "    'Accept-Language': 'en-US,en;q=0.8',\n",
    "    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36',\n",
    "    'Accept': '*/*',\n",
    "    'Referer': 'http://randomvin.com/',\n",
    "    'Connection': 'keep-alive',\n",
    "}\n",
    "\n",
    "requests.get('http://randomvin.com/getvin.php?type=real', headers=headers, cookies=cookies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Robots.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also known as the robot exclusion standard, *robots.txt* is a standard used by website to communicate to web crawlers, scrapers etc. The robots.txt is a file that is used to communicate which parts of the website is allowed or disallowed to be scraped. You will find the robots.txt file in the root directory of the website. For the english version of wikipedia, it is located at https://en.wikipedia.org/robots.txt.  \n",
    "\n",
    "The complement to *robots.txt* is the *sitemap* which is an XML file that lists the URLs for a site. "
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}