quick-experiments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Project Gutenberg Poetry Corpus: Quick Experiments\n",
    "\n",
    "By [Allison Parrish](https://www.decontextualize.com/)\n",
    "\n",
    "I made [a corpus of around three million lines of poetry from Project Gutenberg](https://github.com/aparrish/gutenberg-poetry-corpus), which anyone can download and use. This notebook shows a couple of quick examples of using the corpus in Python, just to get you started.\n",
    "\n",
    "First, [download the corpus via this link](http://static.decontextualize.com/gutenberg-poetry-v001.ndjson.gz), or if you're following along in your own copy of Jupyter Notebook and you have `curl` installed, run the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100 52.2M  100 52.2M    0     0  2014k      0  0:00:26  0:00:26 --:--:-- 1089k\n"
     ]
    }
   ],
   "source": [
    "!curl -O http://static.decontextualize.com/gutenberg-poetry-v001.ndjson.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Three million lines of poetry in just over 52 megabytes! Not bad.\n",
    "\n",
    "The file is in gzipped [newline delimited JSON format](http://ndjson.org/): there's a JSON object on each line. You don't need to decompress the file to work with it, since Python has a handy library for working with gzipped files right in the code. The following cell will read in the file and create a list `all_lines` that contains all of these JSON objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import gzip, json\n",
    "all_lines = []\n",
    "for line in gzip.open(\"gutenberg-poetry-v001.ndjson.gz\"):\n",
    "    all_lines.append(json.loads(line.strip()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just to see what those lines look like, let's pick a handful at random:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'gid': '3305', 's': 'When shall we find the spring come in,'},\n",
       " {'gid': '33156', 's': \"How great, in the wild whirl of Time's pursuits,\"},\n",
       " {'gid': '40344', 's': 'She sighs in desert lands:'},\n",
       " {'gid': '34870', 's': '\"We came within the fosses deep, that moat'},\n",
       " {'gid': '37752', 's': 'The dense black-coated throng, and all a-strain'},\n",
       " {'gid': '1365', 's': 'One only lives.  Behold them where they lie'},\n",
       " {'gid': '32153', 's': 'With the rapturous adoration'},\n",
       " {'gid': '38877', 's': 'About them; and the horse of faery screamed'}]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random.sample(all_lines, 8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each object has a key `s` that contains the text of the line of poetry, and a key `gid` that contains the Project Gutenberg ID of the file in question. You can use this ID to look up the title and author of the book of poetry that the line came from (either using the [Project Gutenberg website](https://www.gutenberg.org/) or using pre-built metadata from, e.g., [Gutenberg, dammit](https://github.com/aparrish/gutenberg-dammit/))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concordances and counts\n",
    "\n",
    "The corpus could be useful for collecting, counting and comparing lines of poetry with certain characteristics. Here's our first experiment: find every line of poetry in the corpus with the word \"flower.\" I do this using a regular expression that finds the string `flower` between two word boundaries, without respect to case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "flower_lines = [line['s'] for line in all_lines if re.search(r'\\bflower\\b', line['s'], re.I)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, just to see what we have, we'll take a random sample:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Blooms for you some happy flower.',\n",
       " \"Low to his heart he said; 'the flower\",\n",
       " 'The blush is on the flower, and the bloom is on the tree,',\n",
       " 'Woo and win the Sahri-flower,',\n",
       " \"The very flower of Issland; 'twas a fair yet fearful scene.\",\n",
       " \"There's not a dew drop on the flower,\",\n",
       " \"Of fame, the world's alluring, phantom flower.\",\n",
       " 'Be it not mine to steal the cultured flower']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random.sample(flower_lines, 8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a cut-up method poem, that's not bad all on its own! But let's do a little bit of Digital Humanities and make an aligned concordance of these lines, with the lines sorted alphabetically by the word following \"flower,\" using a bit of regular expression trickery:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                        Or why sae sweet a flower as love\n",
      "                                                So sweet a flower as she.\"\n",
      "                                                         A flower as yet unblossomed. Warmth and light\n",
      "                                           Is only half in flower as yet. But why--\n",
      "                                        \"To gain so fair a flower as you,\n",
      "                                               Cast like a flower aside?\n",
      "                              (Yon scarlet fruit-bell is a flower asleep;)\n",
      "                                                 As doth a flower at Apollo's touch.\n",
      "                                             'Twas a pigmy flower at best,\n",
      "                                               But he, the flower at head and soil at root,\n",
      "                                               But he, the flower at head and soil at root,\n",
      "                                        Blooms the perfect flower at last.\n",
      "                                            Each plant and flower at length being view'd,\n",
      "                                              Shaking that flower at me with soft invitation\n",
      "                                           My sweet lovely flower at my ain fireside.\n",
      "                                       _Who can behold the flower at noon, nor seek_\n",
      "                                              Love's not a flower at sunset droops,\n",
      "                                               From the Is flower at the fence;\n",
      "                                          From the passion-flower at the gate,\n",
      "                                          From the passion-flower at the gate.\n",
      "                                          From the passion-flower at the gate.\n",
      "                                          From the passion-flower at the gate.\n",
      "                                              And those to flower at the prime (and yet\n",
      "                                                  I am the flower at your feet,\n",
      "                                           The plum-tree's flower awakens\n",
      "                                        And drink the very flower away.\n",
      "                                                 If such a flower be cast to the bleak winds,\n",
      "                                              Yes, let the flower be gathered in its bloom!\n",
      "                                    By thee shall herb and flower be kissed,\n",
      "                                    By thee shall herb and flower be kissed;\n",
      "                              The withered leaf, the faded flower be mine,\n",
      "                                                   Never a flower be near me set,\n",
      "                                                Before the flower be on the bramble spray?\n",
      "                                             \"If this fair flower be plucked, oh, misery!  oh,\n",
      "                                                    If the flower be scorched by the summer sun,\n",
      "                                                 And every flower be springing.\n",
      "                                                 And every flower be springing;\n",
      "                                                      Pale flower beaten by the rain!\n",
      "                        With careless scythe too near some flower bed,\n",
      "                        With careless scythe too near some flower bed,\n",
      "                        Humble little cottage, but a royal flower bed.\n",
      "                                 Dry wildness of the weedy flower bed;\n",
      "                                                       Nor flower bedropt with diamond dew;\n",
      "                                           Fruit trees and flower beds eaten bare,\n",
      "                                                 Among the flower beds her dear form sees,\n",
      "                                              So pluck the flower before it fades--\n",
      "                                        The colours of the flower before its leaves unclose;\n",
      "                                            pass silently, flower before our eyes, it is\n",
      "                                   Both beast and bird and flower before the Queen;\n",
      "                                        And like a fragile flower before the storm,\n"
     ]
    }
   ],
   "source": [
    "longest = max([len(x) for x in flower_lines]) # find the length of the longest line\n",
    "center = longest - len(\"flower\") # and use it to create a \"center\" offset that will work for all lines\n",
    "\n",
    "sorted_flower_lines = sorted(\n",
    "    [line for line in flower_lines if re.search(r\"\\bflower\\b\\s\\w\", line)], # only lines with word following\n",
    "    key=lambda line: line[re.search(r\"\\bflower\\b\\s\", line).end():]) # sort on the substring following the match\n",
    "\n",
    "for line in sorted_flower_lines[350:400]: # change these numbers to see a different slice\n",
    "    offset = center - re.search(r'\\bflower\\b', line, re.I).start()\n",
    "    print((\" \"*offset)+line) # left-pad the string with spaces to align on \"flower\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As another experiment, let's find all of the words that occur between either \"the\" or \"a\" and the word \"flower.\" English being the way it is, these words are pretty much guaranteed to be adjectives, so this is an ersatz but effective way of getting a (non-exhaustive) list of adjectives that are used to describe a flower in the corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "found_adj = []\n",
    "for line in flower_lines:\n",
    "    matches = re.findall(r\"(the|a)\\s(\\b\\w+\\b)\\s(\\bflower\\b)\", line, re.I)\n",
    "    for match in matches: \n",
    "        found_adj.append(match[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some adjectives at random:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['milky',\n",
       " 'sweetest',\n",
       " 'wild',\n",
       " 'fairer',\n",
       " 'moon',\n",
       " 'fairest',\n",
       " 'blue',\n",
       " 'flaming',\n",
       " 'splendid',\n",
       " 'golden',\n",
       " 'meanest',\n",
       " 'coveted']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random.sample(found_adj, 12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `Counter` object, we can easily count these up and find the twelve most common adjectives (used in the type of noun phrase we've identified) used to describe a flower:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('little', 26),\n",
       " ('white', 23),\n",
       " ('sweetest', 22),\n",
       " ('wild', 19),\n",
       " ('fairest', 15),\n",
       " ('tender', 13),\n",
       " ('sweet', 11),\n",
       " ('purple', 11),\n",
       " ('meanest', 11),\n",
       " ('lovely', 10),\n",
       " ('bonnie', 10),\n",
       " ('faded', 9)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Counter(found_adj).most_common(12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The little white sweetest wild fairest tender sweet purple meanest lovely bonnie faded flower..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rhymes\n",
    "\n",
    "Stretches of language identified as poetry characteristically exhibit some variety of rhyming, and the lines of poetry in the Gutenberg Poetry corpus are no different. Let's set ourselves a task of finding random rhyming lines in the corpus. To do this, we need to know how words are pronounced. The way that words are spelled in English doesn't really tell us anything helpful about how the word is pronounced, so we need some alternate method to get that information. The [CMU Pronouncing Dictionary](http://www.speech.cs.cmu.edu/cgi-bin/cmudict) is one such method: it's a big database of phonetic transcriptions for many thousands of English words.\n",
    "\n",
    "I made a Python library called [pronouncing](https://pypi.org/project/pronouncing/) to make it very easy to work with the CMU Pronouncing Dictionary in Python. You can install it like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pronouncing in /Users/allison/anaconda/lib/python3.6/site-packages\n",
      "Requirement already satisfied: cmudict>=0.4.0 in /Users/allison/anaconda/lib/python3.6/site-packages (from pronouncing)\n",
      "\u001b[33mYou are using pip version 9.0.3, however version 18.0 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install pronouncing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then import it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pronouncing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll consider two lines to rhyme with each other if the last words in the lines rhyme. To test this out, we'll pick a source word, say, \"flowering,\" and find all of the words that rhyme with it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "source_word = \"flowering\"\n",
    "source_word_rhymes = pronouncing.rhymes(source_word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['cowering',\n",
       " 'devouring',\n",
       " 'empowering',\n",
       " 'glowering',\n",
       " 'powering',\n",
       " 'scouring',\n",
       " 'showering',\n",
       " 'souring',\n",
       " 'towering']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "source_word_rhymes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then look through the lines of poetry in the corpus for lines that end with any of these words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In the Winter you are cowering\n",
      "\"Oh, yes!\" exclaimed John, with a towering\n",
      "In the Winter you are cowering\n",
      "winged things may never pass, nay, not even the cowering\n",
      "Ithaca, these are wooing me against my will, and devouring\n",
      "\"Of Coleridge, I can not speak but with reverence. His towering\n",
      "upbraid him. \"Son of Tydeus,\" he said, \"why stand you cowering\n",
      "the heaviness of his heart, \"why are the Achaeans again scouring\n",
      "Maidens with towering\n",
      "Are its waters, aye showering\n",
      "In the Winter you are cowering\n",
      "In the Winter you are cowering\n",
      "So hunted, yet defiant, cowering\n",
      "The moonlit crests of foaming waves gleam towering\n"
     ]
    }
   ],
   "source": [
    "for line in all_lines:\n",
    "    text = line['s']\n",
    "    match = re.search(r'(\\b\\w+\\b)\\W*$', text)\n",
    "    if match:\n",
    "        last_word = match.group()\n",
    "        if last_word in source_word_rhymes:\n",
    "            print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking through all three million lines of poetry to find rhyming lines one-by-one will be pretty slow. Another approach is to use the `phones_for_word()` and `rhyming_part()` functions in the `pronouncing` library to pre-build a data structure with all of the lines in the corpus grouped with their rhymes. The `phones_for_word()` function gives you the \"phones\" (sounds) of how a word is pronounced; the `rhyming_part()` function gives you just the portion of a string of phones that another word must share in order for them to be considered \"rhyming\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'F L AW1 ER0 IH0 NG'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "phones = pronouncing.phones_for_word(source_word)[0] # words may have multiple pronunciations, so this returns a list\n",
    "phones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'AW1 ER0 IH0 NG'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pronouncing.rhyming_part(phones)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell builds the data structure proposed above: a dictionary that maps rhyming parts to a dictionary that maps words with that rhyming part to the lines of poetry that they're found at the end of."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "by_rhyming_part = defaultdict(lambda: defaultdict(list))\n",
    "for line in all_lines:\n",
    "    text = line['s']\n",
    "    if not(32 < len(text) < 48): # only use lines of uniform lengths\n",
    "        continue\n",
    "    match = re.search(r'(\\b\\w+\\b)\\W*$', text)\n",
    "    if match:\n",
    "        last_word = match.group()\n",
    "        pronunciations = pronouncing.phones_for_word(last_word)\n",
    "        if len(pronunciations) > 0:\n",
    "            rhyming_part = pronouncing.rhyming_part(pronunciations[0])\n",
    "            # group by rhyming phones (for rhymes) and words (to avoid duplicate words)\n",
    "            by_rhyming_part[rhyming_part][last_word.lower()].append(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A random key/value pair from this dictionary, so you can see its structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('EH1 N S AH0 Z',\n",
       " defaultdict(list,\n",
       "             {'commences': ['Ancient history of Portugal commences',\n",
       "               'Each day some scene of woe commences'],\n",
       "              'expenses': ['Will pay for all the school expenses',\n",
       "               'Will pay for all the school expenses',\n",
       "               'Which brought great bothers and expenses'],\n",
       "              'fences': [\"We've been climbing trees an' fences\",\n",
       "               'And men too; and why there are fences']}))"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_rhyming_part = random.choice(list(by_rhyming_part.keys()))\n",
    "random_rhyming_part, by_rhyming_part[random_rhyming_part]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many rhyming parts are found in multiple lines, but only with one unique word. While it's true that identical words \"rhyme,\" it's a little disingenuous to claim that we've made a computer program that finds rhyming lines of poetry if it's mostly just finding lines that end in the same word. So we'll just find the groups from the `by_rhyming_part` dictionary that have at least two different line-ending words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "rhyme_groups = [group for group in by_rhyming_part.values() if len(group) >= 2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, find seven rhyming couplets by selecting a random rhyming group, sampling two keys (words) from that group, and printing a random line from both groups:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "For Brighton's size compared to Nairn\n",
      "The wind blaws clean about the cairn\n",
      "Or vermin, or, at best, of cock purloined\n",
      "There with the Romans in the camp were joined\n",
      "Nor wine nor wassail could raise a vassal\n",
      "You saw the day when Henry Schnetzen's castle\n",
      "The Legislative Bodies to assemble\n",
      "In vain would formal art dissemble\n",
      "Venus's Advice to Adonis on Hunting\n",
      "Growling, as was his wont, and grunting\n",
      "Of our successors should in part be seated\n",
      "Of ancient prudent words too much repeated\n",
      "Reared by a spring to stately height, amidst\n",
      "For here I read of Eden, and that in the midst\n"
     ]
    }
   ],
   "source": [
    "for i in range(7):\n",
    "    group = random.choice(rhyme_groups)\n",
    "    words = random.sample(list(group.keys()), 2)\n",
    "    print(random.choice(group[words[0]]))\n",
    "    print(random.choice(group[words[1]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Markov chain text generation\n",
    "\n",
    "Markov chain text generation uses statistical information about word co-occurrence to build a model that allows you to generate text that looks similar to your source text. [Markovify](https://github.com/jsvine/markovify) is a great library for Python that makes it easy to build and generate from Markov chain models. Install it like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: markovify in /Users/allison/anaconda/lib/python3.6/site-packages\n",
      "Requirement already satisfied: unidecode in /Users/allison/anaconda/lib/python3.6/site-packages (from markovify)\n",
      "\u001b[33mYou are using pip version 9.0.3, however version 18.0 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install markovify"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And import it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import markovify"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our goal is to use a Markov chain to generate new lines of poetry from the Gutenberg Poetry corpus. Markovify requires you to pass in your source text as a string, so first off we'll create a big string with a sample of the corpus, separated by newlines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "big_poem = \"\\n\".join([line['s'] for line in random.sample(all_lines, 250000)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(You can change the number as needed; I kept it low so that the model will build fast and not consume too much RAM.)\n",
    "\n",
    "Build the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = markovify.NewlineText(big_poem)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then generate some lines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Were emerald: snow new-fallen seem'd the white wisteria\n",
      "And the stars have hid their white faces\n",
      "Bot wel he sih hire wepe,\n",
      "Red like a moon-shaft silver and the flow!\n",
      "In love Heaven gave him last his country house, as if in Nature's scorn,\n",
      "Of cloud grew violet; how thy fame has felt joy and uproar, can ne'er be effaced--\n",
      "Thus Ráma spoke: the Vánar found,\n",
      "I feel him warm, but how it steams in your arms and hands forespent with toil,\n",
      "Or that starred Ethiop queen that we die in a pleasant dream.\n",
      "Leave the dead anew.\n",
      "Through the streets he passed,\n",
      "Home through the mire;\n",
      "Since, stranger! thou hast every gentle wight I pray,\n",
      "Soon made the clouds, as morning walks the sea,\n"
     ]
    }
   ],
   "source": [
    "for i in range(14):\n",
    "    print(model.make_sentence())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is okay but the lines don't make a lot of sense, and are sometimes too long. You can constrain the length using Markovify's `.make_short_sentence()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The record sound in the wood, or the glory moving on,'"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.make_short_sentence(60)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I find that Markov-generated text is best when you keep it short and force juxtapositions—otherwise the reader's attention will wander. The following cell generates a series of short, haiku-esque poems of two to five Markov-generated lines, and ensures that the last line of each poem ends with a period:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "There and here he died,\n",
      "Must be the king Theucer.\n",
      "\n",
      "～ ❀ ～\n",
      "\n",
      "They seemed the most beautiful;\n",
      "Better the rule maintain?\n",
      "Or kings be worn,\n",
      "From curl-crowned forehead to my good.\n",
      "\n",
      "～ ❀ ～\n",
      "\n",
      "And then the words upon our sphere,\n",
      "And so it runs away.\n",
      "Four-and-twenty years he spake\n",
      "They buried him at your length,\n",
      "I do not go from her flying.\n",
      "\n",
      "～ ❀ ～\n",
      "\n",
      "Bot of verray covenant\n",
      "amiable lady, by whom alone is giv'n.\n",
      "\n",
      "～ ❀ ～\n",
      "\n",
      "And there in the little earthen vessels,\n",
      "And owns no softer charm\n",
      "Is each to Heaven commends.\n",
      "\n",
      "～ ❀ ～\n",
      "\n",
      "I thought it very large.\n",
      "And wish'd confusion to the lute\n",
      "And, as he was kind,\n",
      "As the black stars, merrily.\n",
      "\n",
      "～ ❀ ～\n"
     ]
    }
   ],
   "source": [
    "for i in range(6):\n",
    "    print()\n",
    "    for i in range(random.randrange(1, 5)):\n",
    "        print(model.make_short_sentence(40))\n",
    "    # ensure last line has a period at the end, for closure\n",
    "    print(re.sub(r\"(\\w)[^\\w.]?$\", r\"\\1.\", model.make_short_sentence(40)))\n",
    "    print()\n",
    "    print(\"～ ❀ ～\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further reading\n",
    "\n",
    "The [README in the code repository](README.md) has a few more examples of (earlier iterations of) this corpus at work.\n",
    "\n",
    "If you're just getting started with Python and creative language generation, check out the notes for [Reading and Writing Electronic Text](http://rwet.decontextualize.com/), a class I teach at ITP."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}