1. Simple scraper example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Scraping basics\n",
    "\n",
    "If you're reading this in advance of the scraping class, consider attending Melissa's session on How the Internet Works. It'll be a useful introduction to the processes underlying the code you write.\n",
    "\n",
    "Code-free tools for data harvesting are handy in a pinch, but scripts written in Python or another language are more flexible and adaptable. They can also run automatically in the background on a schedule. Also, you don't have to worry about a service or a tool ever disappearing, making all your hard work for naught.\n",
    "\n",
    "Using code -- especially if you host it on a site like GitHub -- also makes your work reproducible, saving you work the next time you need to complete a similar task, and making your methods more transparent.\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This exercise uses a simple HTML table as an example before trying a live site."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import modules to facilitate the scrape\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the file\n",
    "```python\n",
    "open('some file', 'read/write/append?')\n",
    "# open a file and tell Python how to treat it```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fetch the HTML file with open() and stick in a variable\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parse HTML with `BeautifulSoup`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let BeautifulSoup parse the HTML\n",
    "\n",
    "# close the initial file we opened\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Target the data\n",
    "\n",
    "Two key ways (among many) to isolate specific sections of the web page in question with `BeautifulSoup`:\n",
    "```python\n",
    "soup.find('some HTML tag')\n",
    "# returns the first tag that matches\n",
    "\n",
    "soup.find_all('some HTML tag')\n",
    "# returns a list of all tags that match\n",
    "```\n",
    "\n",
    "(`BeautifulSoup` also has [detailed documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) for the various ways in which it can parse HTML and XML.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# snip out the table and pass it to a new variable\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use .find_all to create a list of rows in the table\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# isolate the second row and print it\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use .find_all again to generate a list of the row's cells and return it\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`BeautifulSoup` lets you fetch text from within HTML tags:\n",
    "```python\n",
    "soup.text\n",
    "# returns the text in a tag as a string\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the first item in cells, then try printing what .text can extract from the same thing\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract the data\n",
    "\n",
    "OK, now for the tricky part. We need to create the list of rows in the table and then extract the text contents of each cell. We'll set up an empty list beforehand and append each row of extracted data to it as its own list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make an empty list to hold extracted data\n",
    "\n",
    "# loop through rows, and then each cell in each row, returning a list of extracted text\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write the data to CSV\n",
    "\n",
    "```python\n",
    "writer_obj = csv.writer('some file we opened')\n",
    "# make a writer object that can move information from your script to a file in CSV form\n",
    "\n",
    "writer_obj.writerow('some list of strings')\n",
    "# write a single row\n",
    "\n",
    "writer_obj.writerows('some list of lists of strings')\n",
    "# write a bunch of rows\n",
    "```\n",
    "\n",
    "Check out [the documentation](https://docs.python.org/3.6/library/csv.html) for more examples of how it all works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# open a file and write our data to it\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\"Manual\" example of the above, opening/writing to/closing file without \"with\":  \n",
    "\n",
    "```python\n",
    "outfile = open('simple.csv', 'w')\n",
    "writer = csv.writer(outfile)\n",
    "writer.writerows(table_data)\n",
    "outfile.close()```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}