nltk-session-3-updated-Python3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting the most out of what we've learned"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, now you know Python and NLTK! The main things we still have to do are:\n",
    "\n",
    "1. Address some specific questions\n",
    "2. Manage resources and results\n",
    "3. Brainstorm some other uses for NLTK\n",
    "4. Integrate IPython into your existing workflow\n",
    "5. Have an open discussion about what we've done\n",
    "6. Summarise and say goodbye!\n",
    "\n",
    "This lesson is pretty light on content and structure. Please do jump in at any point, and tell us about your research, and whether or not what you've learned here will be of much use.\n",
    "\n",
    "Or, ask us if Python can do a certain thing. Maybe we have some tips!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function, division\n",
    "import nltk\n",
    "import os\n",
    "from urllib import urlopen\n",
    "% matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting text\n",
    "The most important skill for using NLTK in your life as a researchers is going to be working with your own texts. First, let's look at reading in text files directly from the web."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "﻿The Project Gutenberg EBook of Cocoa and Chocolate, by Arthur W. Knapp\r\n",
      "\r\n",
      "This eBook is for the u\n"
     ]
    }
   ],
   "source": [
    "cocoa = urlopen('http://gutenberg.digitalfabulists.org/pg19073.txt').read()\n",
    "print(cocoa[:100])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Title: Cocoa and Chocolate\n",
      "﻿The Project Gutenberg EBook of Cocoa and Chocolate, by Arthur W. Knapp\r\n",
      "\r\n",
      "This eBook is for the use of anyone anywhere at no cost and with\r\n",
      "almost no restrictions whatsoever.  You may copy it, give i\n"
     ]
    }
   ],
   "source": [
    "raw = unicode(cocoa, 'utf-8')\n",
    "title = next(line for line in raw.splitlines() if line.startswith('Title'))\n",
    "print(title)\n",
    "print(raw[:200])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/fionatweedie/NLTK/books\r\n"
     ]
    }
   ],
   "source": [
    "!pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/fionatweedie/NLTK/books/books\n",
      "created file Title: Cocoa and Chocolate\n"
     ]
    }
   ],
   "source": [
    "# save text to file\n",
    "! mkdir books\n",
    "%cd books\n",
    "filetitle = title.replace(' ', '-')\n",
    "bookfile = open(filetitle, 'w')\n",
    "bookfile.write(raw.encode('utf-8'))\n",
    "bookfile.close()\n",
    "print('created file', title)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Beautiful Soup to read text from the web\n",
    "Of course, a lot of the text you're going to want to work with won't be in handy text files already. That's where a Python library called Beautiful Soup comes in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import urllib\n",
    "from urllib import urlopen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "url = \"http://en.wikipedia.org/wiki/Smog\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<type 'str'>\n",
      "e>Smog - Wikipedia, the free encyclopedia</title>\n",
      "<script>document.documentElement.className = docum\n"
     ]
    }
   ],
   "source": [
    "raw = urlopen(url).read()\n",
    "print(type(raw))\n",
    "print(raw[100:200])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful Soup breaks the single long string into its constituent parts, creating an object 'Beautiful Soup'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'bs4.BeautifulSoup'>\n"
     ]
    }
   ],
   "source": [
    "soup = BeautifulSoup(raw, 'html.parser')\n",
    "print(type(soup))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find all the paragraphs, and put them into a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[u'Smog is a type of air pollutant. The word \"smog\" was coined in the early 20th century as a portmanteau of the words smoke and fog to refer to smoky fog.[1] The word was then intended to refer to what was sometimes known as green soup fog, a familiar and serious problem in Mexico from the 19th century to the mid 20th century. This kind of smog is caused by the burning of large amounts of coal within a city; this smog contains soot particulates from smoke, sulphur dioxide and other components.', u'Modern smog, as found for example in Los Angeles, is a type of air pollution derived from vehicular emission from internal combustion engines and industrial fumes that react in the atmosphere with sunlight to form secondary pollutants that also combine with the primary emissions to form photochemical smog. In certain other cities, such as Delhi, smog severity is often aggravated by stubble burning in neighboring agricultural areas. The atmospheric pollution levels of Los Angeles, Beijing, Delhi, Mexico City and other cities are increased by inversion that traps pollution close to the ground. It is usually highly toxic to humans and can cause severe sickness, shortened life or death.', u'', u'', u'Coinage of the term \"smog\" is generally attributed to Dr. Henry Antoine Des Voeux in his 1905 paper, \"Fog and Smoke\" for a meeting of the Public Health Congress. The July 26, 1905 edition of the London newspaper Daily Graphic quoted Des Voeux, \"He said it required no science to see that there was something produced in great cities which was not found in the country, and that was smoky fog, or what was known as \\'smog.\\'\"[2] The following day the newspaper stated that \"Dr. Des Voeux did a public service in coining a new word for the London fog.\" However, this is predated by a Los Angeles Times article of January 19, 1893, in which the word is attributed to \"a witty English writer.\"', u\"Coal fires, used to heat individual buildings or in a power-producing plant, can emit significant clouds of smoke that contributes to smog. Air pollution from this source has been reported in England since the Middle Ages.[3] London, in particular, was notorious up through the mid-20th century for its coal-caused smogs, which were nicknamed 'pea-soupers.' Air pollution of this type is still a problem in areas that generate significant smoke from burning coal, as witnessed by the 2013 autumnal smog in Harbin, China, which closed roads, schools, and the airport.\", u'Traffic emissions \\u2013 such as from trucks, buses, and automobiles \\u2013 also contribute.[4] Airborne by-products from vehicle exhaust systems cause air pollution and are a major ingredient in the creation of smog in some large cities.[5][6][7][8]', u'The major culprits from transportation sources are carbon monoxide (CO),[9][10] nitrogen oxides (NO and NOx),[11][12][13] volatile organic compounds,[10][11] sulfur dioxide,[10] and hydrocarbons.[10] (Hydrocarbons are the main components of petroleum fuels such as gasoline and diesel fuel.) These molecules react with sunlight, heat, ammonia, moisture, and other compounds to form the noxious vapors, ground level ozone, and particles that comprise smog.[10][11]', u'Photochemical smog is the chemical reaction of sunlight, nitrogen oxides and volatile organic compounds in the atmosphere, which leaves airborne particles and ground-level ozone.[14] This noxious mixture of air pollutants may include the following:', u'A primary pollutant is an air pollutant emitted directly from a source. A secondary pollutant is not directly emitted as such, but forms when other pollutants (primary pollutants) react in the atmosphere. Examples of a secondary pollutant include ozone, which is formed when hydrocarbons (HC) and nitrogen oxides (NOx) combine in the presence of sunlight; nitrogen dioxide (NO2), which is formed as nitric oxide (NO) combines with oxygen in the air; and acid rain, which is formed when sulfur dioxide or nitrogen oxides react with water.[15] All of these harsh chemicals are usually highly reactive and oxidizing. Photochemical smog is therefore considered to be a problem of modern industrialization. It is present in all modern cities, but it is more common in cities with sunny, warm, dry climates and a large number of motor vehicles.[16] Because it travels with the wind, it can affect sparsely populated areas as well.']\n"
     ]
    }
   ],
   "source": [
    "texts = []\n",
    "for para in soup.find_all('p'):\n",
    "    text = para.text\n",
    "    texts.append(text)\n",
    "print(texts[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<type 'unicode'>\n",
      "Smog is a type of air pollutant. The word \"smog\" was coined in the early 20th century as a portmante\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "regex = re.compile('\\[[0-9]*\\]')\n",
    "joined_texts = '\\n'.join(texts)\n",
    "joined_texts = re.sub(regex, '', joined_texts)\n",
    "print(type(joined_texts))\n",
    "print(joined_texts[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to work on the text, the first step is to tokenise it into words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'Smog', u'is', u'a', u'type', u'of', u'air', u'pollutant', u'.']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "wordlist = nltk.word_tokenize(joined_texts)\n",
    "wordlist[:8]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For some other types of analysis, we'll need to create an NLTK text object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Displaying 25 of 40 matches:\n",
      "                                     Smog is a type of air pollutant . The wor\n",
      "                                     smog '' was coined in the early 20th cent\n",
      " the mid 20th century . This kind of smog is caused by the burning of large am\n",
      "amounts of coal within a city ; this smog contains soot particulates from smok\n",
      "ioxide and other components . Modern smog , as found for example in Los Angele\n",
      "mary emissions to form photochemical smog . In certain other cities , such as \n",
      "rtain other cities , such as Delhi , smog severity is often aggravated by stub\n",
      "fe or death . Coinage of the term `` smog '' is generally attributed to Dr. He\n",
      " clouds of smoke that contributes to smog . Air pollution from this source has\n",
      " , as witnessed by the 2013 autumnal smog in Harbin , China , which closed roa\n",
      " major ingredient in the creation of smog in some large cities . The major cul\n",
      " ozone , and particles that comprise smog . Photochemical smog is the chemical\n",
      "s that comprise smog . Photochemical smog is the chemical reaction of sunlight\n",
      "active and oxidizing . Photochemical smog is therefore considered to be a prob\n",
      "automotive exhaust and photochemical smog was discovered in the 1950s by Arie \n",
      "wo key components to the creation of smog . However , the smog created as a re\n",
      "the creation of smog . However , the smog created as a result of a volcanic er\n",
      "s been linked to the distribution of smog in some areas . For example , the cr\n",
      " has been shown to have an effect on smog distribution that is more than fossi\n",
      " than fossil fuel combustion alone . Smog is a serious problem in many cities \n",
      "o Medical Association announced that smog is responsible for an estimated 9,50\n",
      " who had healthy babies , found that smog in the San Joaquin Valley area of Ca\n",
      "w the current accepted safe levels . Smog can form in almost any climate where\n",
      "hi 's children and women . The dense smog in Delhi during winter season result\n",
      "results in severe intensification of smog over Delhi . The state government of\n"
     ]
    }
   ],
   "source": [
    "good_text = nltk.Text(wordlist)\n",
    "good_text.concordance('smog')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And once we've done all that work creating clean text, it's a good idea to save it for later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/fionatweedie\n",
      "/Users/fionatweedie/smog\n"
     ]
    }
   ],
   "source": [
    "%cd\n",
    "! mkdir smog\n",
    "%cd smog"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "NLTK_file = open(\"NLTK-Smog.txt\", \"w\")\n",
    "NLTK_file.write(str(wordlist))\n",
    "NLTK_file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "UnicodeEncodeError",
     "evalue": "'ascii' codec can't encode character u'\\u2013' in position 2455: ordinal not in range(128)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mUnicodeEncodeError\u001b[0m                        Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-22-553551661c90>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mtext_file\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Smog-text.txt\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"w\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mtext_file\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwrite\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mjoined_texts\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mtext_file\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mUnicodeEncodeError\u001b[0m: 'ascii' codec can't encode character u'\\u2013' in position 2455: ordinal not in range(128)"
     ]
    }
   ],
   "source": [
    "text_file = open(\"Smog-text.txt\", \"w\")\n",
    "text_file.write(joined_texts)\n",
    "text_file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'ions \\u2013 such as from '"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joined_texts[2450:2470]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#joined_texts[2450:2470]\n",
    "text_file = open(\"Smog-text.txt\", \"w\")\n",
    "text_file.write(joined_texts.encode(\"UTF-8\"))\n",
    "text_file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenge!\n",
    "* Find a webpage of interest to your studies and use Beautiful Soup to extract the text\n",
    "* Tokenise the text\n",
    "* Find the most common words in your text (Extension: remove the stop words)\n",
    "* Find trigrams in your text \n",
    "* Save your text to a text file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### PDF\n",
    "\n",
    "This part requires the `slate` library. Using DIT4C we found an error with the standard Slate, instead do:\n",
    "\n",
    "`sudo pip install --upgrade --ignore-installed slate==0.3 pdfminer==20110515`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!wget \"http://www.planetebook.com/ebooks/1984.pdf\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import slate\n",
    "with open('1984.pdf') as f:\n",
    "    doc = slate.PDF(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "doc.metadata"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}