Entry: Pirate Novel

[ikarth]

As usual, I'll probably try a bunch of experiments at the start of the month to see what sticks, but my initial plan is to work on a scene expansion system to write coherent interactions between characters, with the scene parameters dictated by a plot generator. (Either tree-expansion one I came up with last year, or a new one.)

Other promising avenues include word2vec, neural networks, and wave function collapse, which has had some interesting experiments with text already by @mewo2 .

[ikarth]

Progress so far: I can extract relevant sentences from a Gutenberg-sourced corpus, based on the vector similarity of their verbs. Here's the top 15 sentences for "attack" from an Arthurian-themed subset:

In: [p.sentence for p in get_sentences("attack")[:15]]
Out: [While these twenty hold the gate, the remaining ten should attack the tower and prevent the Count from barricading himself inside.,
 If he could get the chance the duke's nephew would gladly attack them and do them harm.,
 [Sidenote: Two giants attack Sir Launcelot],
 The legions would attack on the morrow, and so that night must be a night of sacrifice--the greatest sacrifice of all.,
 With his sword, which cuts so clean, he attacks the wicked serpent, first cleaving him through to the earth and cutting him in two, then continuing his blows until he reduces him to tiny bits.,
 The Britons who, suspecting no hostilities, were unprovided with the means of defence, were unanimously and incessantly attacked, both by the Scots from the west, and by the Picts from the north.,
 They were attacked, the legend says, first by a whale, then by a griffin, and then by a race of cyclops, or one-eyed giants.,
 Then the second horseman attacked him furiously, being wroth at the death of his companion.,
 Animated by his words, they attacked the Black Knight anew, whose best refuge was now to place his back against an oak, and defend himself with his sword.,
 CHAPTER VIII
 LAUNCELOT OF THE LAKE
 King Ban, of Brittany, the faithful ally of Arthur was attacked by his enemy Claudas, and after a long war saw himself reduced to the possession of a single fortress, where he was besieged by his enemy.,
 The earls rode to the wood; the Romanish men rode after; the Britons attacked them on their rested steeds, and smote in front, and felled an hundred anon.,
 He at once attacked her, and, thanks to the magic sword, slew her and all her brood after a hard struggle.,
 The magician's followers, eager to avenge their master, attacked the heroes and their men, but their efforts were vain.,
 After which both warriors attacked the young hero.,
 On one such occasion he and his companions were attacked and hard pressed by a party of Viennese under the leadership of Olivier.]

It's still going to take a lot of clean-up to be directly useable, obviously, but it does introduce a huge amount of variation. Depending on how much time I want to devote to this part of the project, I may just end up using it to extract some source sentences to turn into hand-templated actions.

[ikarth]

I also have pirates:

[And the wind whistles to bury the cries of drowning men that plague the memory.,
 But this part I bury in silence for the present.,
 They bury their dead, and in the graves deposit a large portion of the property of the deceased, often to a considerable value in gold ornaments, brass guns, jars, and arms.,
 But did Blackbeard really bury treasures, as tradition says, along the sandy shores he haunted?,
 [Illustration _Captain Kidd burying his Bible._]]

[superMDguy]

Nice!

[ikarth]

Status report: I am currently processing all of Project Gutenberg's English-language fiction books into a Textacy corpus. This may prove to be unworkable: thousands and thousands of fiction books have a very large memory footprint. I may have to reduce my focus to a subset of the total.

Meanwhile, I've put some work in on the scene expansion and plot generation parts of the system.

The part of scene expansion that I've finished is the Conflict system: two characters with opposing goals engage in a contest or combat in a scene. Right now, the only Conflict the system supports is sword fights:

['BEGIN',
 'So, without more ado, they came together, and thereupon began a fierce and mighty battle.',
 'The sword {ACTOR_WEAPON_VERB} and then met with a clank that sounded far and near.',
 'Thus they fought for nigh a half an hour, until the ground was all plowed up with the digging of their heels, and their breathing grew labored like the ox in the furrow.',
 "This way and that they fought, and back and forth, Sir Galahad's skill against the Robin Hood's strength.",
 'Yet in all this time neither had harmed the other nor caused his blood to flow.',
 'The dust of the highway rose up around them like a cloud, so that at times the onlookers could see nothing, but only hear the clank of sword against sword.',
 'This was no playful bout, but a grim and serious fight of real earnest.',
 'Right and left, and up and down and back and forth they fought.',
 'Then up and down and back and forth they trod, the blows falling so thick and fast that, at a distance, one would have thought that half a score were fighting.',
 "{PAR}At last Sir Galahad saw Sir Galahad's chance, and, throwing all the strength Sir Galahad felt going from her into one blow that might have felled an ox, Sir Galahad struck at Robin Hood with might and main.",
 "Thrice Robin Hood struck Sir Galahad; once upon the arm and twice upon the ribs, and yet Robin Hood warded all the other's blows, only one of which, had it met its mark, would have laid Robin Hood lower in the dust than Robin Hood had ever gone before.",
 'Now and then they stopped to rest, and each thought that they never had seen in all their life before such a hand at sword or sword.',
 'Thus they strove for an hour or more, pausing every now and then to rest, at which times each looked at the other with wonder, and thought that never had they seen so stout a foe; then once again they would go at it more fiercely than ever.',
 'So they stood, each in their place, neither moving a finger\'s-breadth back, for one good hour, and many blows were given and received by each in that time, till here and there were sore bones and bumps, yet neither thought of crying "Enough," nor seemed likely to fall.',
 '{PAR}Robin Hood made a feint, and then delivered a blow at Sir Galahad that, had it met its mark, would have tumbled Sir Galahad speedily.',
 "Three blows Robin Hood struck, yet never one touched so much as a hair of Sir Galahad's head. ",
 'The dust of the highway rose up around them like a cloud, so that at times the onlookers could see nothing, but only hear the clank of sword against sword.',
 'Thus they fought for nigh a half an hour, until the ground was all plowed up with the digging of their heels, and their breathing grew labored like the ox in the furrow.',
 "Well did Robin Hood hold Robin Hood's own that day.",
 'Right and left, and up and down and back and forth they fought.',
 "Well did Robin Hood hold Robin Hood's own that day.",
 'Yet in all this time neither had harmed the other nor caused his blood to flow.',
 'The sword {ACTOR_WEAPON_VERB} and then met with a clank that sounded far and near.',
 "{PAR}At last Sir Galahad saw Sir Galahad's chance, and, throwing all the strength Sir Galahad felt going from her into one blow that might have felled an ox, Sir Galahad struck at Robin Hood with might and main.",
 "Then Robin Hood saw Robin Hood's chance, and, ere you could count three, Sir Galahad's sword was flying away, and Sir Galahad herself lay upon the {GROUND_DESCRIPTION} with no more motion than you could find in an empty pudding bag.",
 "And so Sir Galahad was defeated by Robin Hood's sword"]

There's some pronoun cleanup that it could be doing but has the underlying data for, and some missing descriptions, but it's a fairly credible output. It repeats a bit, but that's mostly a factor of not enough raw input and the probability decay on repeating a statement being too rapid.

A basic action looks like this:

{Cmd.prereq: [incoming_attack, is_offbalance_actor], Cmd.effects: [efx_knockdown_target, efx_recover_balance_actor, efx_drop_weapon_target], Cmd.action: "But {ACTOR} regained {ACTOR_SELF} quickly and, at arm's length, struck back a blow at {ANTAGONIST}, and this time the stroke reached its mark, and down went {ANTAGONIST} at full length, {ANTAGONIST'S} {ANTAGONIST_WEAPON} flying from {ANTAGONIST'S} hand as {ANTAGONIST} fell."},

the prereq list has functions that query the Conflict state, while the effects list has functions that alter the world state. (Mostly modeled by adding and removing strings from a list of tags.) There's probably a way to build a context free grammar to handle this, but the system has to also process the changes to the world state, so I built this thing.

The system uses the prereq queries to find the list of valid actions, and then randomly picks from that list, with the chance of selecting an action being based on the number_of_prereqs * (1 - decay_factor), with the decay factor being reset to 1 every time that action gets used. There may be a better way to calculate that, since, as I said, the decay is decaying too fast.

The Conflict state isn't stored as a static list. Instead, it's dynamically calculated from the stack of actions that have been executed. This lets me do some tricky things with the effects while keeping the conflict state representation as a super simple Python collections.Counter. Note that the conflict state isn't intended to cover the entire novel: once the scene and conflict are done, the results are recorded in the transcript and we can toss the Conflict itself.

[ikarth]

Pirate name generator:

Jacquotte "Corsair" Every
Shih "Black Caesar" ibn Jabir al-Jalahimah
Edward Gibbs
Wang "Diamonds" Awilda
Samuel Bonnet
Gan Jeanne
Louis-Michel "Diabolito" Skytte
Grace "Bloody" Gambi
Shap "Irish" Roche
"Corsair" Rachel
Wang Christina
"Redbeard" Ching
Mizzenmast
"One-Eye" Roberts
Hayreddin "Captain Blood" Killigrew
Zheng Bully
Elane "Reis" ibn Jabir al-Jalahimah
Redbeard
Jeanne "One-Eye" Hill
Gan "Redbeard" William
"Honorable" Maffitt
"Gunpowder" Killigrew
Moses Cohen "The Lost" Mason
Shap Basilica
Fuma
Rags and Bones
"Blind"
Cheung "Deepwater" Lars
Lim John Newland

Mostly Tracery, with a bit of trickery to give different name orderings.

[ikarth]

Generating pirate ships and their crews:

Crossbones, a 3-masted barque
Higgs Aury, captain, who carries a sabre
"Qian Lord" Qian, first mate, who carries a kukri
"Elane Wall" Elane, quartermaster, who carries a cutlass
Rahmah "Bilge" Boyah, second_mate, who carries a sword
"Calico Jacquotte" Jacquotte, boatswain, who carries a kukri
Flora "Weevil Flora" Hornigold, cook, who carries a cutlass
Vincenzo Crickett, carpenter, who carries a cutlass
Liang "Mary Liang" Mary, sailmaker, who carries a claymore
Gunpowder Gan, steward, who carries a sword

[ikarth]

The book compilation pipeline is complete. I can now press a button and go from generating a book from scratch to outputting a PDF. Advice for the future: getting this set up earlier in the month will probably pay off. Going from iterating on parts of the system to iterating on the book as a whole is a good motivator.

*************
*   Story   *
*************

   The Story of Bartholomew Cofresí  
   also known as  
   "Bartholomew Cofresí"  
   being a true account  
   of the  
   adventures  
   of the  
   _Black Flag_    


 *TEMP: RANDOM DESTINATION*  *VOYAGE: BEGIN_VOYAGE*  *VOYAGE: WEIGH ANCHOR*  *BEGIN: WEIGH ANCHOR* 

The order was given, and soon the messenger was run out and the capstan manned. There came 
soon the familiar racket of making sail and trimming yards and the clank of the capstan pawls. The
 tars strained at the bars, the pawl clicking as they drove the capstan round. 

"That's your sort, my hearties," exclaimed "Barbarossa" encouragingly, as "Barbarossa" applied her
tremendous strength to the outer extremity of one of the bars, "heave with a will! heave, and she 
_must_ come! _heave_, all of us!! now--one--_two_--three!!!" The sailors pressed their broad chests
against the powerful levers, planted their feet firmly upon the deck, straightened out their backs, 
and slowly pawl after pawl was gained. 

As the capstan turned, the cable could be seen cutting through the surf. As the sailors turned the
capstan, "Barbarossa" took up the verse and the crew repeated the chorus: 
  
   I thought I heard the old man say  
   Good-bye, fare ye well,  
   Good-bye, fare ye well.  
   I thought I heard the old man say  
   Hooray my boys we're homeward bound.  
   
   
 The chorus of the shanty kept time with the clicks of the pawl. 

The anchor cable was hauled aboard, with another heave on the capstan. 

"That's your sort, my hearties," exclaimed "Barbarossa" encouragingly, as "Barbarossa" applied 
her tremendous strength to the outer extremity of one of the bars, "heave with a will! heave, 
and she _must_ come! _heave_, all of us!! now--one--_two_--three!!!" The tars pressed their 
broad chests against the powerful levers, planted their feet firmly upon the deck, straightened 
out their backs, and slowly pawl after pawl was gained. 

Below the waves, the anchor began to shift, the top lifting off the seafloor, and the crew heaved 
again. As the tars turned the capstan, "Barbarossa" took up the verse and the crew repeated 
the chorus: 
  
   Oh poor old Stormy's dead and gone.  
   Storm along boys,  
   Storm along.  
   Oh poor old Stormy's dead and gone.  
   Ah-ha, come along, get along,  
   Stormy along John.  
     
   I dug his grave with a silver spade.  
   Storm along boys,  
   Storm along.  
   I dug his grave with a silver spade.  
   Ah-ha, come along, get along,  
   Stormy along John.  
     
   Oh poor old Stormy's dead and gone.  
   Storm along boys,  
   Storm along.  
   Oh poor old Stormy's dead and gone.  
   Ah-ha, come along, get along,  
   Stormy along John.  
     
   I lower'd him down with a golden chain.  
   Storm along boys,  
   Storm along.  
   I lower'd him down with a golden chain.  
   Ah-ha, come along, get along,  
   Stormy along John.  
     
   I carried him away to Mobile Bay.  
   Storm along boys,  
   Storm along.  
   I carried him away to Mobile Bay.  
   Ah-ha, come along, get along,  
   Stormy along John.  
   
 

"Anchor aweigh," came the cry. 

 The anchor was soon secured to the cathead. The ((ship_type)) was alive and in motion.
 *END: WEIGH ANCHOR*  
*VOYAGE: LEAVE HARBOR*  
*VOYAGE: PERILS AT SEA*  
*VOYAGE: SAILING*  
*VOYAGE: ENTER HARBOR*  
*VOYAGE: DROP ANCHOR*  
*VOYAGE: END* 

[ikarth]

A recent bug in the transcript system that resulted in the pirate ships sailing in circles has been fixed, and leads me to conclude that my transcript system is too clever by half. A frequent NaNoGenMo hazard for me.

Island generation is in:

They had arrived at The White Isle of Oceanus.

Pines grow there, but they were more abundant, and seemingly larger, upon  
some other of the islands, particularly on the one to the westward.  

The tars told stories of a pantellerite from the island, which they remembered  
from past voyages. The tars were eager to catch the rhinoceros, which they  
had heard could be found there. A flower grows on the island, a buttercup  
with darkblue petals. That place is known for a rare kind of lighter purple panda. 

The inhabitants use what you might think of as a nutritional yeast in their cooking.

Like most of the systems in the generator, it could use a lot more content, but the interesting thing is that it's guaranteed to only output self-consistent island descriptions. The sentences of the descriptions have tags, such as "<+feature landscape flora>Pines</+> grow there, that get picked up by the island generator and used to ensure that only compatible tags are used, so that an island is never described as being both inhabited and uninhabited, or in having both three harbors and only one harbor.

In theory, these tags can also be picked up an reincorporated in the other generators in the system. The islands are generated at the start of the novel, and it stores the information so that, for example, it knows which animal is found on which island, and what sauce the inhabitants use to cook it. I have yet to implement much of anything that makes use of this information, but the data is there, and easy to access.

The island generator works completely differently from the conflict generator. Ideally, I'd be able to incorporate a number of generators that use wildly different principles, since I think that results in a much stronger output than spending the same amount of time on one complex generator. I doubt I'm going to have enough time this month to incorporate very many more, though.

[ikarth]

Turns out I did have time to add character descriptions, though I still need to add the hooks that will insert them into the transcript.

'Mizzenmast wore a red sash tied around her waist, and, as she pushed back 
her coat, you might glimpse the glitter of a pistol butt. She had once kept a 
pet gopher. She wore no hat, though she frequently wore orchid in her hair.'

[ikarth]

Release version!

The Story of Mary Youx
Git repository: https://github.com/ikarth/excalibur
Markdown version: https://github.com/ikarth/excalibur/blob/master/excalibur/output.markdown
PDF version: https://github.com/ikarth/excalibur/blob/master/excalibur/output.pdf

There's a lot more that could be done with this--it doesn't even show off the full potential of the underlying system, and the sword fights aren't in the main narrative--but it's the end of the month.

Written in Python 3.5. Contains a lot of text from various Project Gutenberg books, especially from works by Herman Melville, Joseph Conrad, and Richard Henry Dana, Jr. Though there is surprisingly very little from Moby Dick.

[spenteco]

Very nice, Isaac. I'm going to give it some close study, especially the tracery bits.

BTW, I got it to run on Ubuntu w/o too much trouble: set up a virtualenv, install tracery, numpy (probably not necessary, since it's almost certainly in ipython), pycorpora and ipython. I have some local installation issues to resolve to get the pdf, all of which are my problem. But I'm getting the output markdown, which is really all I need to study the key parts of your process..

Again, I find this very cool. Thanks for posting it!

[ikarth]

I'm kind of impressed that you got it to run, since I did no deployment testing. (Also, the code is a mess.) The PDF output requires Pandoc and a LaTeX environment plus some custom fonts; you may be able to delete a couple of lines from the latex template and have it work.

I should really do a writeup of what's going on under the hood here, because, while I didn't quite reach the literary quality I was aiming for, it did demonstrate that some of the approaches I was trying are viable. (And that some can be redesigned to make it simpler, because I learned a lot in the process.)

In particular, the Conflict system in action_commands.py is a bit too clever and I think can be be improved and (most importantly) made more robust. (Also, the swordfight, which can end in several different ways, shows it off much better than the weighing anchor sequences, though Melville would be proud of the detail.)

The island generator and the character description generator, on the other hand, turned out better than I was hoping. Tagging the content takes some effort (and they could use some more content) but reversing the assumptions of the usual description generator turned out to be very efficient. Instead of generating the numbers and writing sentences to describe them, pick sentences from the bag and use the characteristics in the sentences to define the island's properties. It means that your probabilities are dictated by the actual content you've written, which is, at least, a more efficient way to get use out of all of your writing.

The very-powerful-but-not-connected word2vec system in action_catalog.py and sourcing.py ended up as the great white whale of the project; I used it a little to manually find sentences that would be interesting to include (though I used grep way, way more). But I ran out of time before I could integrate it in an automated way, and I need to fix some bugs and implement some more features before it can be actually useful.