A5 Reinforcement Learning Solution to Towers of Hanoi.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right\">Maxwell You</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reinforcement Learning Solution to the Towers of Hanoi Puzzle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this assignment, I will use reinforcement learning to solve the [Towers of Hanoi](https://en.wikipedia.org/wiki/Tower_of_Hanoi) puzzle with three disks.\n",
    "\n",
    "To accomplish this, I have written a Temporal Difference learning algorithm which makes use of a Q table to store temporal differences. This code is adapted from Prof. Chuck Anderson's lecture notes on [Reinforcement Learning for Tic Tac Toe](http://nbviewer.jupyter.org/url/www.cs.colostate.edu/~anderson/cs440/notebooks/15%20Reinforcement%20Learning%20for%20Two-Player%20Games.ipynb).\n",
    "\n",
    "I have implemented the following functions:\n",
    "\n",
    "  - `printState(state)`: prints the state of the puzzle\n",
    "  - `validMoves(state)`: returns a list of valid moves from the current state\n",
    "  - `stateMoveTuple(state, move)`: returns the state and move as a tuple for keying into the Q dictionary\n",
    "  - `makeMove(state, move)`: returns the state resulting from executing the specified move\n",
    "  - `epsilonGreedy(epsilon, Q, state, validMovesF)`: returns a move that either explores or exploits\n",
    "  - `finished(state)`: returns True if all disks are on the third peg\n",
    "  - `trainQ(nRepetitions, learningRate, epsilonDecayFactor, validMovesF, makeMoveF)`: returns Q table along with a list of steps to reach the goal for each trial\n",
    "  - `testQ(Q, maxSteps, validMovesF, makeMoveF)`: uses Q to find the greedy move each step until goal is found. Returns path of states to goal\n",
    "  - `plotSteps(numTrials, stepsToGoal)`: plots number of steps to reach the goal for each trial\n",
    "\n",
    "At the end, I have made a plot to show how effectively the program learned to play Towers of Hanoi."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Representing Towers of Hanoi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### State"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To represent a state in Towers of Hanoi, we can use a list of lists. For example, `[[1, 2, 3], [], []]` represents the start state. We can then print out the starting state as follows:\n",
    "\n",
    "    1\n",
    "    2\n",
    "    3\n",
    "    -----\n",
    "    \n",
    "where the first, third, and fifth dashes represent the three pegs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def printState(state):\n",
    "    '''\n",
    "    prints the state of the puzzle\n",
    "    :param state:\n",
    "    :return:\n",
    "    '''\n",
    "    peg1 = state[0]\n",
    "    peg2 = state[1]\n",
    "    peg3 = state[2]\n",
    "    if len(peg1) == 3:\n",
    "        for disk in peg1:\n",
    "            print(disk)\n",
    "    elif len(peg1) == 2:\n",
    "        if len(peg2) == 1:\n",
    "            print('{}\\n{} {}'.format(peg1[0], peg1[1], peg2[0]))\n",
    "        else: # last disk on peg 3\n",
    "            print('{}\\n{}   {}'.format(peg1[0], peg1[1], peg3[0]))\n",
    "    elif len(peg1) == 1:\n",
    "        if (len(peg2) == 0):\n",
    "            print('    {}'.format(peg3[0]))\n",
    "            print('{}   {}'.format(peg1[0], peg3[1]))\n",
    "        elif (len(peg2) == 1):\n",
    "            print('{} {} {}'.format(peg1[0], peg2[0], peg3[0]))\n",
    "        else: # 2 disks on peg 2\n",
    "            print('  {}'.format(peg2[0]))\n",
    "            print('{} {}'.format(peg1[0], peg2[1]))\n",
    "    else: # peg 1 has no disks\n",
    "        if len(peg2) == 3:\n",
    "            print('  {}\\n  {}\\n  {}'.format(peg2[0], peg2[1], peg2[2]))\n",
    "        elif len(peg2) == 2: # no disks on peg 1, 2 disks on peg 2, then 1 disk on peg 3\n",
    "            print('  {}'.format(peg2[0]))\n",
    "            print('  {} {}'.format(peg2[1], peg3[0]))\n",
    "        elif len(peg2) == 1: # other 2 disks on peg 3\n",
    "            print('    {}'.format(peg3[0]))\n",
    "            print('  {} {}'.format(peg2[0], peg3[1]))\n",
    "        else: # all 3 disks on peg 3\n",
    "            print('    {}\\n    {}\\n    {}'.format(peg3[0], peg3[1], peg3[2]))\n",
    "    print('-----')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "-----\n"
     ]
    }
   ],
   "source": [
    "state = [[1, 2, 3], [], []]\n",
    "printState(state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Determining valid moves"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Towers of Hanoi, we can only move disks to empty pegs or on top of larger disks. Moves will be represented using a list (e.g. [1, 2] indicates moving the disk at the top of peg 1 to peg 2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def validMoves(state):\n",
    "    '''\n",
    "    Returns list of valid moves from current state.\n",
    "    Can't move larger disks on top of smaller disks. Can move disks to empty pegs.\n",
    "    :param state:\n",
    "    :return:\n",
    "    '''\n",
    "    peg1 = state[0]\n",
    "    peg2 = state[1]\n",
    "    peg3 = state[2]\n",
    "    validMoves = []\n",
    "\n",
    "    if peg1: # peg 1 is not empty\n",
    "        # all pegs not empty\n",
    "        if peg2 and peg3:\n",
    "            if peg1[0] < peg2[0]:\n",
    "                validMoves.append([1, 2])\n",
    "            if peg1[0] < peg3[0]:\n",
    "                validMoves.append([1, 3])\n",
    "            if peg2[0] < peg1[0]:\n",
    "                validMoves.append([2, 1])\n",
    "            if peg2[0] < peg3[0]:\n",
    "                validMoves.append([2, 3])\n",
    "            if peg3[0] < peg1[0]:\n",
    "                validMoves.append([3, 1])\n",
    "            if peg3[0] < peg2[0]:\n",
    "                validMoves.append([3, 2])\n",
    "        # disks on peg 1 & 2\n",
    "        elif peg2:\n",
    "            if peg1[0] < peg2[0]:\n",
    "                validMoves.append([1, 2])\n",
    "            if peg2[0] < peg1[0]:\n",
    "                validMoves.append([2, 1])\n",
    "            # add moves to empty pegs as valid moves\n",
    "            validMoves.append([1, 3])\n",
    "            validMoves.append([2, 3])\n",
    "        # disks on peg 1 & 3\n",
    "        elif peg3:\n",
    "            if peg1[0] < peg3[0]:\n",
    "                validMoves.append([1, 3])\n",
    "            if peg3[0] < peg1[0]:\n",
    "                validMoves.append([3, 1])\n",
    "            # add moves to empty pegs as valid moves\n",
    "            validMoves.append([1, 2])\n",
    "            validMoves.append([3, 2])\n",
    "        # all disks on peg 1\n",
    "        else:\n",
    "            # add moves to empty pegs as valid moves\n",
    "            validMoves.append([1, 2])\n",
    "            validMoves.append([1, 3])\n",
    "    # peg 1 is empty\n",
    "    else:\n",
    "        # disks on peg 2 & 3\n",
    "        if peg2 and peg3:\n",
    "            if peg2[0] < peg3[0]:\n",
    "                validMoves.append([2, 3])\n",
    "            if peg3[0] < peg2[0]:\n",
    "                validMoves.append([3, 2])\n",
    "            # add moves to empty pegs as valid moves\n",
    "            validMoves.append([2, 1])\n",
    "            validMoves.append([3, 1])\n",
    "        # all disks on peg 2\n",
    "        elif peg2:\n",
    "            # add moves to empty pegs as valid moves\n",
    "            validMoves.append([2, 1])\n",
    "            validMoves.append([2, 3])\n",
    "        # all disks on peg 3\n",
    "        elif peg3:\n",
    "            # add moves to empty pegs as valid moves\n",
    "            validMoves.append([3, 1])\n",
    "            validMoves.append([3, 2])\n",
    "\n",
    "    return validMoves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "-----\n",
      "valid moves are: [[1, 2], [1, 3]]\n"
     ]
    }
   ],
   "source": [
    "printState(state)\n",
    "print('valid moves are: {}'.format(validMoves(state)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making moves"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We don't want to modify the original state when making moves, so we will deep copy the state, do moves on the new state, and return that new state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from copy import deepcopy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def makeMove(state, move):\n",
    "    '''\n",
    "    Returns a new state where move was applied.\n",
    "    :param state:\n",
    "    :param move:\n",
    "    :return:\n",
    "    '''\n",
    "    # don't modify original state\n",
    "    newState = deepcopy(state)\n",
    "\n",
    "    # get pegs to move from and to, subtract 1 to get correct indices\n",
    "    moveFrom = move[0] - 1\n",
    "    moveTo = move[1] - 1\n",
    "\n",
    "    disk = newState[moveFrom].pop(0)\n",
    "    newState[moveTo].insert(0, disk)\n",
    "\n",
    "    return newState"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "-----\n",
      "making move [1, 2]\n",
      "2\n",
      "3 1\n",
      "-----\n"
     ]
    }
   ],
   "source": [
    "printState(state)\n",
    "print('making move [1, 2]')\n",
    "printState(makeMove(state, [1, 2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making it easier to key into the Q table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Q table will be implemented as a dictionary and only immutable types are allowed as keys. Therefore, we should convert the state, move pair into a tuple.\n",
    "\n",
    "An example with `state = [[2, 3], [], [1]]` and `move = [1, 2]` would convert to `smTuple = (((2, 3), (), (1,)), (1, 2))`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def stateMoveTuple(state, move):\n",
    "    '''\n",
    "    Returns the state and move as a tuple.\n",
    "    Use this tuple to key into the Q table dictionary.\n",
    "    :param state:\n",
    "    :param move:\n",
    "    :return:\n",
    "    '''\n",
    "    stateTuple = tuple([tuple(peg) for peg in state])\n",
    "    return (stateTuple, tuple(move))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(((2, 3), (), (1,)), (1, 2))"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stateMoveTuple([[2, 3], [], [1]], [1, 2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Are we done yet?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have solved the puzzle when all three disks are on the third peg. This is assuming we have only taken valid moves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def finished(state):\n",
    "    '''\n",
    "    Returns True if all disks are stacked correctly on the third peg.\n",
    "    Assumes only valid states are provided.\n",
    "    :param state:\n",
    "    :return:\n",
    "    '''\n",
    "    # if only valid states are provided, then the only time we are finished\n",
    "    # is if all disks are on the third peg\n",
    "    return len(state[2]) == 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "-----\n",
      "Finished? False\n",
      "\n",
      "    1\n",
      "    2\n",
      "    3\n",
      "-----\n",
      "Finished? True\n"
     ]
    }
   ],
   "source": [
    "printState(state)\n",
    "print('Finished? {}'.format(finished(state)))\n",
    "print()\n",
    "printState([[], [], [1, 2, 3]])\n",
    "print('Finished? {}'.format(finished([[], [], [1, 2, 3]])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have set up Towers of Hanoi, we can start learning how to solve it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Temporal Difference Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first think about reinforcement learning. To accomplish tasks better, we can reward the program for making desirable choices and penalize it for making unfavorable choices. If we have the program play the game many times, and reward and penalize its actions, then at the end, it will have (hopefully!) learned to play the game well. Temporal Difference (TD) learning is a reinforcement learning algorithm that seeks to do just that.\n",
    "\n",
    "TD learning keeps a Q table of values for each state, move pair according to the predicted return from the next state. In the equation below, $s_t$ and $a_t$ represent the state and action at time $t$. The value for this state, action pair is the one step reinforcement ($r_{t+1}$) plus the value of the next state, action pair ($\\text{value}(s_{t+1},a_{t+1})$).\n",
    "\n",
    "$$\\text{value}(s_t,a_t) \\approx r_{t+1} + \\text{value}(s_{t+1},a_{t+1})$$\n",
    "\n",
    "We add the value of the next state, action pair to the one step reinforcement because we assume the next state, action pair has a good idea of the return for the rest of the game by ending up there.\n",
    "\n",
    "In Towers of Hanoi, we can assign a move we haven't explored yet with a return of -1 because we want the program to find the shortest path to the goal. By assigning one step reinforcements of -1, more moves taken will result in a lower return. Therefore, TD learning will try to maximize the return recieved from each state, move pair.\n",
    "\n",
    "If we encounter a goal state, we can assign a return of 0 to the previous state, move pair because if we have that state and move to choose from at any point in the game, then it is guaranteed to \"win\" by executing that action from that state. For example, given the `state = [[1], [], [2, 3]]` and the `move = [1, 3]` below, `Q((((1,), (), (2, 3)), (1, 3)) = 0` because moving disk 1 onto peg 3 will result in a win:\n",
    "\n",
    "        2\n",
    "    1   3\n",
    "    -----\n",
    "\n",
    "How do we decide whether to explore more or to exploit the Q table for the greedy move? This is where we can use an `epsilonGreedy` function to decide. Epsilon will be in the range $0 \\leq epsilon \\leq 1$ and is the chance that we will choose a random move."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Epsilon - To Explore or not to Explore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initially epsilon is great because we haven't learned anything, so we want to explore a lot (i.e. choosing random moves). As we play more games, we want to decrease epsilon because we want to start relying on our prior experience a little more (exploiting the Q table). Eventually, we will have played so many games and learned so much that we are almost always exploiting the Q table instead of exploring random moves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import random as rand"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def epsilonGreedy(epsilon, Q, state, validMovesF):\n",
    "    '''\n",
    "    Returns a move based on epsilon.\n",
    "    If epsilon is high, more exploration and less exploitation is done.\n",
    "    If epsilon is low, less exploration and more exploitation is done.\n",
    "    :param epsilon:\n",
    "    :param Q:\n",
    "    :param state:\n",
    "    :param validMovesF:\n",
    "    :return:\n",
    "    '''\n",
    "    # np.where returns a tuple with an array of indices that evaluated True\n",
    "    # for the specified condition as the first item of the tuple\n",
    "    # Use [0] to get this array of indices\n",
    "    validMoves = validMovesF(state)\n",
    "    if np.random.uniform() < epsilon:\n",
    "        # choose a random move\n",
    "        return rand.choice(validMoves)\n",
    "    else:\n",
    "        # choose the greedy move\n",
    "        # Get the Q values for the valid moves in this state\n",
    "        Qs = np.array([Q.get(stateMoveTuple(state, m), 0) for m in validMoves])\n",
    "        return validMoves[np.argmax(Qs)] # retrieves max Q value for current state in Q table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train Q function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can train the Q values based on the number of trials we want (`nRepetitions`). To determine how much exploration vs exploitation we do, we specify the `epsilonDecayFactor`.\n",
    "\n",
    "Can we start learning now?\n",
    "\n",
    "Almost, we just need one more variable: the `learningRate`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Learning Rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `learningRate` determines how fast and accurate a program learns. A low `learningRate` means the program will slowly learn the true return for some state, action pair, but it could take a while if it is too low. A high `learningRate` means the program will update the Q values very dramatically which could lead to learning the true return faster, but can be very inaccurate also.\n",
    "\n",
    "It helps to picture finding the minimum of a parabola when thinking of `learningRate`. A low `learningRate` will slowly travel down one side of the parabola each iteration until it finds the minimum, though this could take a while. A high `learningRate` will travel down one side of the parabola very quickly each iteration, and could find the minimum quickly, but it will most likely overshoot the minimum and end up on the other side of it and continue to overshoot each iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def trainQ(nRepetitions, learningRate, epsilonDecayFactor, validMovesF, makeMoveF):\n",
    "    '''\n",
    "    Returns the Q values learned and steps to the goal for each trial.\n",
    "    :param nRepetitions:\n",
    "    :param learningRate:\n",
    "    :param epsilonDecayFactor:\n",
    "    :param validMovesF:\n",
    "    :param makeMoveF:\n",
    "    :return:\n",
    "    '''\n",
    "    epsilon = 1.0\n",
    "\n",
    "    outcomes = np.zeros(nRepetitions)\n",
    "    epsilons = np.zeros(nRepetitions)\n",
    "    # keep track of the number of steps to reach goal for each repetition\n",
    "    stepsToGoal = np.zeros(nRepetitions)\n",
    "    Q = {}\n",
    "\n",
    "    for nGame in range(nRepetitions):\n",
    "        epsilon *= epsilonDecayFactor\n",
    "        epsilons[nGame] = epsilon\n",
    "        step = 0\n",
    "        state = [[1, 2, 3], [], []] # towers of hanoi start state\n",
    "        done = False\n",
    "\n",
    "        while not done:\n",
    "            step += 1\n",
    "\n",
    "            # choose a move\n",
    "            move = epsilonGreedy(epsilon, Q, state, validMovesF)\n",
    "            newState = makeMoveF(state, move)\n",
    "\n",
    "            if (stateMoveTuple(state, move)) not in Q:\n",
    "                # initial Q value for state, move\n",
    "                # -1 b/c we want the program to learn to solve Towers of Hanoi quickly\n",
    "                # By assigning -1 to every move except for a move to the goal state,\n",
    "                # we can have program choose largest Q value as that will minimize # of steps taken\n",
    "                Q[stateMoveTuple(state, move)] = -1\n",
    "\n",
    "            if finished(newState): # goal state reached\n",
    "                # update Q value for previous state, move pair because we will always\n",
    "                # finish Towers of Hanoi from that state and action\n",
    "                Q[stateMoveTuple(state, move)] = 0\n",
    "                done = True\n",
    "                outcomes[nGame] = 0\n",
    "\n",
    "            if step > 1:\n",
    "                Q[stateMoveTuple(oldState, oldMove)] += learningRate * (-1 + Q[stateMoveTuple(state, move)]\n",
    "                                                                        - Q[stateMoveTuple(oldState, oldMove)])\n",
    "\n",
    "            # save this state; it will be the old state after next move\n",
    "            oldState, oldMove = state, move\n",
    "            state = newState\n",
    "        stepsToGoal[nGame] = step\n",
    "    return Q, stepsToGoal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Q, stepsToGoal = trainQ(50, 0.5, 0.7, validMoves, makeMove)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 110.,   30.,   75.,   45.,   53.,   25.,   31.,   24.,    8.,\n",
       "         12.,   15.,   20.,    7.,   13.,   14.,   18.,    7.,    7.,\n",
       "          7.,    7.,    7.,    7.,    7.,    7.,    7.,    7.,    7.,\n",
       "          7.,    7.,    7.,    7.,    7.,    7.,    7.,    7.,    7.,\n",
       "          7.,    7.,    7.,    7.,    7.,    7.,    7.,    7.,    7.,\n",
       "          7.,    7.,    7.,    7.,    7.])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stepsToGoal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing the Q table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to see how well the program learned to play Towers of Hanoi. We know that for a 3 disk environment, it takes 7 moves to solve. To find the shortest path to the goal, we can select the greedy moves from the Q table. Remember that a low epsilon will exploit more than it explores. Therefore, if $\\text{epsilon} = 0$, we will always be exploiting the Q table. From there, we choose the maximum return given the state we are in, which will also be the greedy move."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def testQ(Q, maxSteps, validMovesF, makeMoveF):\n",
    "    '''\n",
    "    Return path of states to goal.\n",
    "    Finds the greedy action each step until goal is found.\n",
    "    :param Q:\n",
    "    :param maxSteps:\n",
    "    :param validMovesF:\n",
    "    :param makeMoveF:\n",
    "    :return:\n",
    "    '''\n",
    "    state = [[1, 2, 3], [], []]\n",
    "    pathToGoal = [state]\n",
    "    done = False\n",
    "    while not done:\n",
    "        # pass in 0 as epsilon because we don't want to explore, Q is already trained with some good values\n",
    "        # therefore, we can just exploit the Q table for the best moves learned so far\n",
    "        move = epsilonGreedy(0, Q, state, validMovesF)\n",
    "        newState = makeMove(state, move)\n",
    "        if finished(newState):\n",
    "            done = True\n",
    "        pathToGoal.append(newState)\n",
    "        state = newState\n",
    "    return pathToGoal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path = testQ(Q, 20, validMoves, makeMove)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[[1, 2, 3], [], []],\n",
       " [[2, 3], [], [1]],\n",
       " [[3], [2], [1]],\n",
       " [[3], [1, 2], []],\n",
       " [[], [1, 2], [3]],\n",
       " [[1], [2], [3]],\n",
       " [[1], [], [2, 3]],\n",
       " [[], [], [1, 2, 3]]]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Path to goal is: (7 moves)\n",
      "1\n",
      "2\n",
      "3\n",
      "-----\n",
      "\n",
      "2\n",
      "3   1\n",
      "-----\n",
      "\n",
      "3 2 1\n",
      "-----\n",
      "\n",
      "  1\n",
      "3 2\n",
      "-----\n",
      "\n",
      "  1\n",
      "  2 3\n",
      "-----\n",
      "\n",
      "1 2 3\n",
      "-----\n",
      "\n",
      "    2\n",
      "1   3\n",
      "-----\n",
      "\n",
      "    1\n",
      "    2\n",
      "    3\n",
      "-----\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# len - 1 b/c start state is not a move, but is in the path\n",
    "print('Path to goal is: ({:d} moves)'.format(len(path) - 1))\n",
    "for p in path:\n",
    "    printState(p)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Can I see how well this thing learned?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can plot the number of moves to the goal for each trial using `matplotlib`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from IPython.display import display, clear_output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plotSteps(numTrials, stepsToGoal):\n",
    "    # plot number of steps to reach goal for each trial\n",
    "    trials = list(range(numTrials))\n",
    "    plt.plot(trials, stepsToGoal)\n",
    "    plt.xlabel('Trials')\n",
    "    plt.ylabel('Steps to Goal')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmgAAAJQCAYAAAA6xpiJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XmUXGd57/vfU8Pu7uqWetDQltoabEu2I8sDRIzGYAMm\nJDjgcHMIuYfEOZe7SM4lCZkDWcnlJOeyFuTmsJJzMtw4BDAhISEBY0PC4JjJNglEtjxPkgfJbk0t\nS2qpB1V1Vb33j713q9Wq7to17drV9f2spaWu6qrud6nC8e+87/s8jznnBAAAgORItXsBAAAAOBcB\nDQAAIGEIaAAAAAlDQAMAAEgYAhoAAEDCENAAAAAShoAGAACQMAQ0AACAhCGgAQAAJEym3QtoxNq1\na93WrVvbvQwAAICq7r///mPOuXVRXtvRAW3r1q3avXt3u5cBAABQlZntj/pajjgBAAAShoAGAACQ\nMAQ0AACAhCGgAQAAJAwBDQAAIGEIaAAAAAlDQAMAAEgYAhoAAEDCENAAAAAShoAGAACQMAQ0AACA\nhCGgAQAAJAwBDQAAIGEIaAAAAAlDQAMAAEgYAhoAAEDCENAAAAAShoAGAACQMAQ0AACAhCGgAQAA\nJAwBDQAAIGEIaAAAAAlDQFvG5Oyc3vQ/vq0v3P9iu5cCAAC6CAFtGemU6ZmJab00nW/3UgAAQBch\noC3DS/v/PIViuc0rAQAA3YSAtoxs2iQR0AAAQLwIaMswM3mZlPIlAhoAAIgPAa2KnnSKHTQAABAr\nAloVXoaABgAA4kVAq4KABgAA4kZAq8LLpFTgDhoAAIgRAa2KLHfQAABAzAhoVXjplObYQQMAADEi\noFXhZVLKs4MGAABiRECrgiIBAAAQNwJaFT0UCQAAgJgR0KrwKBIAAAAxI6BVwREnAACIGwGtCvqg\nAQCAuBHQquCIEwAAxI2AVgVHnAAAIG4EtCqYJAAAAOJGQKuiJ5NSnjtoAAAgRgS0KryMP+rJOdfu\npQAAgC5BQKvCS6fknFQsE9AAAEA8CGhVeBn/n4h7aAAAIC4EtCoIaAAAIG4EtCrmAxqFAgAAICYE\ntCq8NDtoAAAgXgS0KsIdtDwBDQAAxISAVkUPd9AAAEDMCGhVcAcNAADEjYBWRZY7aAAAIGYEtCrC\nIoE5dtAAAEBMCGhV0AcNAADEjYBWBVWcAAAgbgS0KnooEgAAADEjoFXhpdOSOOIEAADxIaBVwR00\nAAAQNwJaFWcDWqnNKwEAAN2CgFYFjWoBAEDcCGhVMCwdAADEjYBWRTZtkghoAAAgPgS0KsxMXjql\nPEecAAAgJgS0CLxMSnNF1+5lAACALkFAi8DLpFQoUcUJAADiQUCLwEunuIMGAABi07KAZmafNLOj\nZvbogudGzOwuM9sb/D284HsfMrN9ZvaUmf1Iq9ZVDy9DQAMAAPFp5Q7apyW9ddFzH5R0t3Nuu6S7\ng8cysx2S3i3piuA9f25m6RaurSb+EScBDQAAxKNlAc05911Jxxc9/Q5JtwVf3ybp5gXP/71zLu+c\ne07SPkmvbNXaasURJwAAiFPcd9BGnXOHgq8PSxoNvh6T9MKC170YPJcIXialPAENAADEpG1FAs45\nJ6nm3hVm9j4z221muycmJlqwsvNxBw0AAMQp7oB2xMw2SFLw99Hg+XFJmxa87sLgufM45251zu1y\nzu1at25dSxcb6uEOGgAAiFHcAe1OSbcEX98i6Y4Fz7/bzHrM7CJJ2yX9IOa1LSnLHTQAABCjTKt+\nsJl9TtL1ktaa2YuSPizpo5I+b2bvlbRf0rskyTn3mJl9XtLjkoqS3u+cS0xnWC+d0hw7aAAAICYt\nC2jOuZ9e4ltvWuL1H5H0kVatpxHcQQMAAHFikkAEBDQAABAnAloENKoFAABxIqBF4KXpgwYAAOJD\nQIughyNOAAAQIwJaBOERp99bFwAAoLUIaBF46ZSck4plAhoAAGg9AloEXsb/Z+KYEwAAxIGAFgEB\nDQAAxImAFkE27f8zMU0AAADEgYAWQbiDRqsNAAAQBwJaBD3hESc7aAAAIAYEtAi8NHfQAABAfAho\nEVAkAAAA4kRAi8DjiBMAAMSIgBYBR5wAACBOBLQIOOIEAABxIqBFQJsNAAAQJwJaBLTZAAAAcSKg\nReCl05I44gQAAPEgoEWQzZgkRj0BAIB4ENAioIoTAADEiYAWAVWcAAAgTgS0CGhUCwAA4kRAiyA8\n4qTNBgAAiAMBLQIzk5dOccQJAABiQUCLyMsQ0AAAQDwIaBF5mZQKpVK7lwEAALoAAS0ijjgBAEBc\nCGgRccQJAADiQkCLKJs2zZVcu5cBAAC6AAEtIi+Tps0GAACIBQEtIr9IgIAGAABaj4AWUU86pUKR\nKk4AANB6BLSIKBIAAABxIaBFxBEnAACICwEtIvqgAQCAuBDQIuKIEwAAxIWAFhEBDQAAxIWAFhF3\n0AAAQFwIaBF56RSNagEAQCwIaBF5mZTm2EEDAAAxIKBFRBUnAACICwEtIi+TUtlJRXbRAABAixHQ\nIvIy/j8VhQIAAKDVCGgReekgoHHMCQAAWoyAFtH8DhoBDQAAtBgBLaIwoNFqAwAAtBoBLaIe7qAB\nAICYENAi4g4aAACICwEtIu6gAQCAuBDQIgoDGtMEAABAqxHQIspyxAkAAGJCQItovoqTHTQAANBi\nBLSIKBIAAABxIaBF1EORAAAAiAkBLSKqOAEAQFwIaBExLB0AAMSFgBYRd9AAAEBcCGgRccQJAADi\nQkCLiCNOAAAQFwJaROERZ54dNAAA0GIEtIjMTNm0MeoJAAC0HAGtBl46xR00AADQcgS0GngZAhoA\nAGg9AloNCGgAACAOBLQaeJkUVZwAAKDlCGg14A4aAACIAwGtBl4mTZsNAADQcgS0GnDECQAA4kBA\nq0FPOqVCsdTuZQAAgBWOgFYDqjgBAEAcCGg18DIpzZVcu5cBAABWOAJaDbJpYwcNAAC0HAGtBl4m\nTZEAAABoOQJaDeiDBgAA4kBAq4GXSdEHDQAAtBwBrQY9GdpsAACA1iOg1YBGtQAAIA4EtBpwBw0A\nAMSBgFYDL5NS2UlFdtEAAEALEdBq4GX8fy6OOQEAQCsR0Grgpf1/rrki0wQAAEDrENBqEO6g5UtU\ncgIAgNYhoNUg3EGjUAAAALQSAa0G83fQCGgAAKCFCGg1oEgAAADEgYBWA444AQBAHNoS0MzsV83s\nMTN71Mw+Z2a9ZjZiZneZ2d7g7+F2rG05HHECAIA4xB7QzGxM0i9L2uWc2ykpLendkj4o6W7n3HZJ\ndwePE4WABgAA4tCuI86MpD4zy0jKSToo6R2Sbgu+f5ukm9u0tiWdbbNBQAMAAK0Te0Bzzo1L+iNJ\nByQdkjTpnPuGpFHn3KHgZYcljca9tmq4gwYAAOLQjiPOYfm7ZRdJ2iip38zes/A1zjknqWK7fjN7\nn5ntNrPdExMTLV/vQj0ccQIAgBi044jzzZKec85NOOfmJH1R0mslHTGzDZIU/H200pudc7c653Y5\n53atW7cutkVLZ4845zjiBAAALdSOgHZA0qvNLGdmJulNkp6QdKekW4LX3CLpjjasbVlZjjgBAEAM\nMnH/Qufc983snyQ9IKkoaY+kWyUNSPq8mb1X0n5J74p7bdXQqBYAAMQh9oAmSc65D0v68KKn8/J3\n0xKLNhsAACAOTBKoQVjFmSegAQCAFiKg1YA2GwAAIA4EtBqkUqZs2riDBgAAWoqAViMvnWIHDQAA\ntBQBrUZehoAGAABai4BWIwIaAABoNQJajbxMikkCAACgpQhoNfLSKeUJaAAAoIUIaDXKUiQAAABa\njIBWox7uoAEAgBYjoNWIIgEAANBqBLQaeZkUjWoBAEBLEdBqRKNaAADQagS0GnHECQAAWo2AViMv\nk+aIEwAAtBQBrUYccQIAgFYjoNXIy6SUJ6ABAIAWIqDVqIdRTwAAoMUIaDVqtEjgycOn9BN/fp+m\n8sUmrgoAAKwkBLQaZdPWUJHAgwdOas+Bk3r+2HQTVwUAAFYSAlqNvHRapbJTqezqev90oSRJmpyd\na+ayAADACkJAq5GX8f/J6j3mnA6ONk/MFJq2JgAAsLIQ0GrUcEAr+AHt5Aw7aAAAoDICWo3CgJYv\nlep6/0zef99JdtAAAMASCGg16kk354iTHTQAALAUAlqNmnbESZEAAABYAgGtRvMBrc5WGzMFjjgB\nAMDyCGg18oIjzrlinW02OOIEAABVENBqdHYHrb4igemgSIA2GwAAYCkEtBplgx20egemh3fQaFQL\nAACWQkCrUaNFAmfvoM3JufqOSQEAwMpGQKtRT4MBbSpfVCZlKpYdA9MBAEBFBLQaNVLFOVcqq1As\na8NQryQKBQAAQGUEtBp5DTSqDY83x4b6JBHQAABAZQS0GjVyB20mKBAYG8pJkk7OUskJAADOR0Cr\nUSNHnGEPtLHgiPMEO2gAAKACAlqNGtlBC3ugjQ37R5yT9EIDAAAVENBq5DXQBy3sgbaRO2gAAGAZ\nBLQazY96quOIcybYQRvq8zTQk+GIEwAAVERAq1EqZcqmrb4jzmAHLdeT1mBfliIBAABQEQGtDtl0\nqqE7aP1eRkO5LEecAACgIgJaHbxMqq4qzrDNRn9PWsM5TycpEgAAABUQ0Org1bmDFo52ynkZDeay\nOsnAdAAAUAEBrQ5epr6ANlMoqTebUjplGuaIEwAALIGAVgcvk1K+zka1Az0ZSX4l58mZgspl1+zl\nAQCADkdAq0O9R5wzhZJyXhDQclmVnXQ6OPYEAAAIEdDq0FPnEedUvqicl5YkDeU8SdIkx5wAAGAR\nAlod6r+DVlT//BFnVpJ0gkpOAACwCAGtDl4mVdckgel8aT6gDff7AY1KTgAAsBgBrQ5eur4+aNP5\novqDI87BPv+Ik15oAABgMQJaHeqdJLCwSGA4F+ygcQcNAAAsQkCrQ7130KYLRfX3hDtoBDQAAFAZ\nAa0OXialfD07aAvuoGXSKa3qyVAkAAAAzkNAq0NPHbM4C8WyCqXy/B00SRrqz2qSIgEAALAIAa0O\n9TSqDQelh3fQpLPTBAAAABYioNWhnjto04WSJM2PepL8aQInuIMGAAAWIaDVwavjiHMmGOmU61lw\nxJnzOOIEAADnIaDVwUunVSo7lWoYdD4VBLT+BUecw7ksRQIAAOA8BLQ6eBn/n62WaQIzwRFnbmGR\nQJ9fJFCuIegBAICVj4BWhzCg1dJqYzrcQVtwB20w58k56fSZYnMXCAAAOhoBrQ5hQKulUGC6cH5A\nC6cJcMwJAAAWIqDVwUubJNVUKDCd9484z+mDlmNgOgAAOB8BrQ717KDN90E7p82GPzCdHTQAALAQ\nAa0OXtrfBavpiDPYQctlzy0SkKRJeqEBAIAFCGh1qOsOWr6onJdWKmXzz7GDBgAAKiGg1WE+oJVK\nkd8zXSidM+ZJkgaDHbST7KABAIAFCGh18NK1t9mYKRQ1sGCKgCSlU6bVvRmmCQAAgHNklvqGmX1Z\n0pIdVJ1zb2/JijpAfUec5++gSdJwv8cRJwAAOMeSAU3SH8W2ig7TU+cdtP5FO2iSXyjAEScAAFho\nyYDmnPtOnAvpJGdHPUUf0TRTKM4XBSw0lPN0kh00AACwQNU7aGa23cz+ycweN7Nnwz9xLC6pwjto\ntRYJDPScn4eHclka1QIAgHNEKRL4lKS/kFSUdIOkz0j6bCsXlXSNtNlYjCNOAACwWJSA1uecu1uS\nOef2O+f+m6S3tXZZyZZN13sHrdIOmqdTZ+ZUKkc/LgUAACvbckUCobyZpSTtNbNflDQuaaC1y0q2\ncActapsN55xmCqXKRQK5rJyTTs3Oabj//DtqAACg+0TZQfuApJykX5b0w5J+RtItrVxU0s1XcUYc\nll4olVUsu8ptNpgmAAAAFqm6g+ac+w9JMjNJ+iXn3FSrF5V0Xo1HnOEczv4Kd9AGc8E0AQoFAABA\nIEoV55VmtkfSY5IeM7P7zWxn65eWXKmUKZOyGgJaUZKUq3QHbX7cEztoAADAF+WI8y8l/Zpzbotz\nboukX5d0a2uXlXxeJhU5oM0U/B20Sm02wiNOKjkBAEAoSkDrd859K3zgnPu2pP6WrahDeJlU5Dto\nU+EOWqU2GzkGpgMAgHNFqeJ81sx+T9LfBI/fI6mrG9VK/j20uYgBbabgB7RKbTZW92ZlxhEnAAA4\nK8oO2v8haZ2kLwZ/1gXPdTUvk4rcZiMsEqi0g5ZKmQb7mCYAAADOilLFeUJ+iw0sUNsdNH8HrdId\nNMm/h3aCI04AABBYcgfNzF5nZj+74PE/mdk3gz9vjGd5yeWlowe0+SrOCn3QJPk7aBxxAgCAwHI7\naL8v6ZcWPL5M0s/JLxD4HUnfbN2ykq+WIoHpoIqz0iQByS8UOD5NQAMAAL7l7qCtds49vuDxXufc\n/c6570pa1eJ1JV4tO2gz+aLMpL5s5YDmH3ES0AAAgG+5gDa08IFz7p0LHo62Zjmdo5Y7aNOFkvq9\nTDiN4Tz+ESd30AAAgG+5gPakmb1t8ZNmdpOkp1q3pM5Q0xFnvlixgjM0nPN0+kxRxYg/DwAArGzL\n3UH7VUn/bGY/KemB4LkflvRaSTc18kvNbEjSJyTtlOTkt+14StI/SNoq6XlJ7woqSBOppiKBQqli\nD7RQ2Kx2cnZOawZ6mrI+AADQuZbcQXPO7ZN0laR75IemrZK+K+kq59zTDf7eP5H0Nefc5ZKulvSE\npA9Kuts5t13S3cHjxKqpzUa+uGSBgLRgmgC90AAAgKr0QXPO5SV9spm/0MwGJb1efkWonHMFSQUz\ne4ek64OX3Sbp25J+u5m/u5lqaVQ7lS8u2WJDkobm53FSKAAAAKJNEmi2iyRNSPqUme0xs0+YWb+k\nUefcoeA1h5XwQoSeTC2jnkrqX+YO2lAf8zgBAMBZ7QhoGUkvl/QXzrmXSZrWouNM55yTfzftPGb2\nPjPbbWa7JyYmWr7YpXjpWvqgFZe9gzYc7KAxTQAAAEgRA5qZeWa2M/iTbfB3vijpRefc94PH/yQ/\nsB0xsw3B79sg6WilNzvnbnXO7XLO7Vq3bl2DS6lfbXfQ/DYbSxkM76BxxAkAABQhoJnZ9ZL2Svoz\nSX8u6Wkze329v9A5d1jSC2Z2WfDUmyQ9LulOSbcEz90i6Y56f0ccsjWOesotUySwujejdMo44gQA\nAJIiDEuX9D8kvcU595Qkmdmlkj4nv+VGvX5J0t+amSfpWUn/RX5Y/LyZvVfSfknvauDnt5yXSalY\ndiqXnVKpyg1oJck55x9xLrODZmZ+s9pZdtAAAEC0gJYNw5kkOeeebvSY0zn3oKRdFb71pkZ+bpy8\njL/5WCiV1ZtaencsXyyr7LTsHTTJLxRgBw0AAEjRAtpuM/uEpM8Gj/+zpN2tW1Jn8NJ+QMsXy+pd\nYsam5LfYkJYelB4ayhHQAACAL0pA+6+S3i/pl4PH98i/j9bVesIdtCr30GbyJUlatg+a5PdCO3r6\nTHMWBwAAOlqUgPYLzrmPS/p4+ISZfUD+NICutfCIcznThWAHbZk+aJK/g/bU4dPNWRwAAOhoUdps\n3FLhuZ9r8jo6jhd1By0MaFXvoHmaZNQTAADQMjtoZvbTkv53SReZ2Z0LvrVK0vFWLyzpvLS/I1Zt\nmsBUcMQZ5Q7aVL6ouVJZ2XQ7+gcDAICkWG5b53uSDklaK7/VRui0pIdbuahOEHkHLSgSqHYHbTh3\ndtzTulU9TVghAADoVEumBufcfvn9yF4T33I6RxjQqg1Mny74O2gDVY44B4NxT5OzBQIaAABdjrO0\nOoVtNqLeQctVKRIId9CYxwkAAAhodfIy/vSAalWcZ/ugVS8SkEQvNAAAUFtAM7NhM7uqVYvpJGGR\nQJQ+aCk72zdtKUPzO2iMewIAoNtFGZb+bTNbbWYjkh6Q9Fdm9vFq71vpohYJTBeK6u/JyGzpeZ3S\n2YA2yQ4aAABdL8oO2qBz7pSkd0r6jHPuVZLe3NplJd/ZRrWlZV83nV9+UHpooCejTMoYmA4AACIF\ntIyZbZD0LklfafF6Okb0HbSSclV6oEmSmWkol6VIAAAARApofyDp65Kecc79h5ldLGlva5eVfJGr\nOCPuoEnSYF+WI04AAFB9Fqdz7h8l/eOCx89K+t9auahOUEsftGpTBELDOY8iAQAAEKlI4GIz+7KZ\nTZjZUTO7I9hF62phVeZcyS37uqh30CS/UIA2GwAAIMoR599J+rykDZI2yt9N+1wrF9UJojeqLSlX\npQdaaJCB6QAAQNECWs459zfOuWLw57OSelu9sKRLpUyZlEWq4hyIfMSZ5YgTAABECmhfNbMPmtlW\nM9tiZr8l6V/MbCTojda1sulUtB20Go44Zwol5YvLhz4AALCyRUkO7wr+/vlFz79bkpPUtffRvMzy\nAc055zeqrTKHMzQUDkyfmdP61dHeAwAAVp4oVZwXxbGQTuRlUsvO4pydK8k5Rb6DFk4TODk7p/Wr\nu/4UGQCArhWlijNnZr9rZrcGj7eb2U2tX1ryeenUsm02pvP+UWW1Qemh4RwD0wEAQLQ7aJ+SVJD0\n2uDxuKT/p2Ur6iA9VY44p/NFSYp8xDnYx8B0AAAQLaBd4pz7Q0lzkuScm5G0/OTvLlHtDtp0wQ9o\ntRQJSAxMBwCg20UJaAUz65NfECAzu0RSvqWr6hDV7qDNFPwjzoEajzjZQQMAoLtFSQ7/TdLXJG0y\ns7+VdK2k/9LKRXUKL53S3DIBLTzijDIsXZJyXlrZtOkkzWoBAOhqUao4v2Fm90t6tfyjzQ845461\nfGUdoOoRZ1gkEPGI08w0lPN0kh00AAC6WpQqzrudcy855/7ZOfcV59wxM7s7jsUlXfQ7aNF7mg31\n1T6P8+ipM3pmYqqm9wAAgORacmvHzHol5SStNbNhnS0MWC1pLIa1JV61NhszwRFn1DtoUn0D0z/0\nxUd04PiM7vq1N9T0PgAAkEzLJYefl/Qr8gek36+zAe2UpD9t8bo6QrZKkcB0UCQQ9Q6a5E8TeOH4\nTOTX54slfe+Zl5RJU1gLAMBKsWRAc879iaQ/MbNfcs79rxjX1DF6qszinM4XlUmZvHSUYlnfUF9W\nj9ZQJPDA/pOanStJc9JcqaxsDb8LAAAk05L/NTezV5jZBWE4M7OfNbM7zOx/dvuQ9FC1O2j+oPS0\nzKLvbg33ezW12bh338T810wgAABgZVhuu+Uv5U8QkJm9XtJHJX1G0qSkW1u/tOSr1gdtOl+s6f6Z\n5E8TODNX1pm5UqTX37v3bEEt/dMAAFgZlgtoaefc8eDrn5J0q3PuC86535O0rfVLSz6v2hFnoRh5\nUHoobFY7GeGY88R0QQ+PT+qVF/kbmsenCWgAAKwEywY0MwvTxZskfXPB92pLHStUlD5oUedwhsJx\nT1F2w773zEtyTvrxqzdKEv3TAABYIZYLWp+T9B0zOyZpVtI9kmRm2+Qfc3Y9L5NSsexULjulUuff\nM5spFNVf4w7aUDAwPcp9snv3TWhVT0Y3XLZOknR8mjtoAACsBMtVcX4kaEi7QdI3nHMu+FZK0i/F\nsbik8zL+BmShVFZv6vydsul8SRuHvJp+5lBwxFltN8w5p3v2HtOrL1mjtQM9kriDBgDASrHs9o5z\n7t8rPPd065bTWcL2GYVSWb3ZCgGtUFR/DT3QpLNHnNV20Pa/NKMXT8zqfa+/WL3ZtHJeWie4gwYA\nwIpA06wG9IQ7aEvcQ5vOl5SLOIczFBYJnKgS0O7Z51dvvm7b2vn3HWcHDQCAFYGA1gCvSkCbKRQ1\nUOMOWm82JS+T0snZ5cPWvXsnNDbUp4vW9kuSRvo9dtAAAFghCGgNCLv2Vwpo5bILGtXWtoNmZhrq\ny2pymR20Yqms7z3zkl63be18E9yhXFbHaVQLAMCKQEBrwMIigcVmgkaztd5Bk/zjyuUu/D88PqnT\nZ4p63fa188+N9Hu02QAAYIUgoDXAW2YHbSZflKSad9AkaTCXXbZI4N69x2QmXbvtbEAbznk0qgUA\nYIUgoDUg3EHLVwho0wV/B63WUU+SNBwhoF2xcbVG+r0F7/F0+kxRc8uMngIAAJ2BgNaA5YoEpud3\n0Go/4hzq85YsEpjKF/XAgRN63bZ15zw/0h99AgEAAEg2AloDepa5gxYGtFonCUjSUP/SO2jff/Yl\nFctO1y24fyZJw/1hg1sKBQAA6HQEtAZ4aX93bK7SHbRCWCRQR0Dr85QvljUb/IyF7tl7TL3ZlH54\ny/A5z48E/dO4hwYAQOcjoDVguSrOqXAHrZ4jznCaQIVjznv3HdMrL1pz3uSCcEQUvdAAAOh8BLQG\nLHcHbaYQ3EGrs0hAkk4sGn5+aHJW+45O6bpta897T1gwUG0CAQAASD4CWgOWLxIIjjjr2EEb7Avu\nky3aQbt3bzDeafv5AS3cdaNIAACAzkdAa0A27Xfxz1dqVFuovw/acFCRuXiawL37jmntQI8uv2DV\nee/pzabV76W5gwYAwApAQGtAT1AkUGkHbSpfkpdOze+y1WKo7/zjynLZ6b59x/S6bWvmxzud974c\n8zgBAFgJCGgNqHYHLVfHmCepcpHAk4dP69hUQa/bvm6pt/kD0zniBACg4xHQGlDtDlp/Hcebkn9c\n2ZtNndPT7J69E5Kk11UoEAgN93sMTAcAYAUgoDUgnTKlU6ZC6fx+ZTOFYl2D0kNDfecOP7933zFt\nXz+gCwZ7l3zPcC7LEScAACsAAa1BXjq1xB20Yl0FAqGhXHb+DtqZuZJ+8NzxitWbCw1zBw0AgBWB\ngNYgL5PSXMmd9/xModTYDlouO1/Fufv5E8oXy+eNd1pspN/T6TwD0wEA6HQEtAZ5mZTySwxLr/cO\nmuTvhoVFAvfsm1A2bXrVRWuWf898s1p20QAA6GQEtAYtdcQ5XSjWNYcztPCI8969x/SyzcNVf95S\nEwgAAEBnIaA1qCeTqjiLcyZfUq6OKQKhwT5PkzNzemkqr8cOnqo43mmxcGA6O2gAAHQ2AlqDvExK\nheL5VZyN7qAN57IqlMr61yeOSJKuu3Tp/mfz7+lnYDoAACsBAa1B2QpHnKWy05m5ckN30MJmtV9+\n6JAG+7LDjFNSAAAgAElEQVS6cmyw6nvCgenH2UEDAKCjEdAa5FU44pwO5nA2VsXph63vPXNMr71k\njdKpyuOdzn1PeAeNgAYAQCcjoDWoUpHATN4/8myoD1qfH7bKTlX7n4V6Mv7A9BNMEwAAoKMR0Brk\n30Fr3Q6aJF23rfr9s9BwP81qAQDodAS0BlXqgxbuoDXWB83fQds8ktPmNbka3udxBw0AgA5HQGtQ\npTtoU3l/By3XwA7aYC4rs+jHmyF20AAA6Hz1b/FAktSTTp03WmkmPOJsYAetJ5PW//rpl+mHtwzX\n9L6RXFbPH5uu+/cCAID2I6A1qPIdtOCIs4E+aJJ001Uba34PO2gAAHQ+jjgbVDGg5RsvEqjXcM4f\nmF5p/BQAAOgMBLQGVWqzEQa0Rtps1CucJhAOWgcAAJ2HgNagSkUCM+ERZwOzOOs1P4+TgekAAHQs\nAlqDsumU5kpO5bKbf266UFRPJqVMOv5/3uF+vz3Hce6hAQDQsQhoDfIy/j/hwl206Xxjg9IbMRzu\noNELDQCAjkVAa1BPhYA2ky8p14bjTenswHQCGgAAnYuA1qD5HbQFhQLThaIG2rSDxsB0AAA6HwGt\nQV66QkBr4w5aODD9OEUCAAB0LAJag8IdtIXTBKYL7buDJvmtNk5yxAkAQMcioDWo0hFnO++gSf49\nNAamAwDQuQhoDQqPOPPFBO2g5Rj3BABAJyOgNWjJNhttmCIQGs5l2UEDAKCDEdAaVLmKs6RcG+Zw\nhob7PZ2kSAAAgI5FQGvQ4irOuVJZhWJZA23cQRthYDoAAB2NgNagxTto4RzOXBvvoA2FA9M55gQA\noCMR0Bq0+A7adL4oqT2D0kPhwHTuoQEA0JkIaA1afMQ5U/ADWjt30MKB6Se4hwYAQEcioDVo8RHn\ndN4/4hxoY5EA8zgBAOhsbQtoZpY2sz1m9pXg8YiZ3WVme4O/h9u1tlqEAS2/6Igz19Y2G8ERJ73Q\nAADoSO3cQfuApCcWPP6gpLudc9sl3R08TryetL9TNhfuoAVFAu3sgxYOTKdIAACAztSWgGZmF0p6\nm6RPLHj6HZJuC76+TdLNca+rHouLBM7eQWvfEWdPJq2BngwD0wEA6FDt2kH7Y0m/JWlho65R59yh\n4OvDkkZjX1Udlr6D1r4dNMkvFOAOGgAAnSn2gGZmN0k66py7f6nXOOecJLfE+99nZrvNbPfExESr\nlhlZOmVKp2xBQAvvoLVvB03y76FxBw0AgM7Ujh20ayW93cyel/T3kt5oZp+VdMTMNkhS8PfRSm92\nzt3qnNvlnNu1bt26uNa8LC+dOtsHrdD+IgHJD2jcQQMAoDPFHtCccx9yzl3onNsq6d2Svumce4+k\nOyXdErzsFkl3xL22emXTds4kgb5sWumUtXVNI/0ejWoBAOhQSeqD9lFJN5rZXklvDh53BC+TVj4I\naFP5ovrbWCAQGsplaVQLAECHaus5nHPu25K+HXz9kqQ3tXM99erJpM7uoOWLbT/elPxxT1PBwPSw\nkAEAAHQG/svdBF5m4R20UtsLBCRpmIHpAAB0LAJaE3jplApFv73GTKHY9hYb0tlxT9xDAwCg8xDQ\nmsDLpDRX8ruCTOVLbR2UHgqnCdBqAwCAzkNAawJv0R20/gQccY7MH3FSKAAAQKchoDWBf8R5ts1G\nUooEJHbQAADoRAS0JvAyKeUXNKodSESbDT+gnSCgAQDQcQhoTbDwiHM6X0zEHTQvk/IHplMkAABA\nxyGgNYEf0EoqFMuaK7lE3EGT/IHp3EEDAKDzENCaIJzFORPM4exPwA6a5N9D4w4aAACdh4DWBGGR\nwFQ+CGgJKBKQ/HtoJzjiBACg4xDQmiC8gzZT8JvV5hJQJCAFA9PZQQMAoOMQ0JogDGjTCdtBG855\n3EEDAKADEdCaIJzFGe6gJeYOWn9WU/mi8sEYKgAA0BkIaE3gpf1RT6fP+DtoSRiWLp3thcYuGgAA\nnYWA1gRexv9nnJz173slZwctaFZLoQAAAB2FgNYEPUFAOxHsVPUnpEhgmHFPAAB0JAJaE3jzAS3Y\nQUtKkUB/VpJ0YpojTgAAOgkBrQm8tP/PeDIIQn3ZZOygzQ9M54gTAICOQkBrgmwQ0I7PFJTz0kql\nrM0r8s0XCXDECQBARyGgNUF4xHlyppCYAgHJX9cqBqYDANBxCGhN4C0oEkjKoPTQUH9WJ9hBAwCg\noxDQmmA+oE0XlEtIgUBoJOfNV5cCAIDOQEBrgp6wSGB2LjEtNkLD/QxMBwCg0xDQmiDcQSuVXaLu\noEn+Dhp90AAA6CwEtCYIA5qUnB5ooaGcxx00AAA6DAGtCRYGtKTM4QyN9Gc1XSgxMB0AgA5CQGuC\nsFGtlJw5nKHhfgamAwDQaQhoTXDOEWfSigSYxwkAQMchoDXBuUecCdtBCwIa99AAAOgcBLQmOOeI\nM3F30IKAxhEnAAAdg4DWBOcecSZsB60/K4mB6QAAdBICWhMkuUhgqI8jTgAAOg0BrQky6ZRS5n+d\ntDYb4cB0pgkAANA5CGhNEh5zJm0HTQrGPbGDBgBAxyCgNUl4zJm0SQKSH9COUyQAAEDHIKA1iZfx\njzaT1gdNkoZzWXbQAADoIAS0JukJjjiT1gdN8gemcwcNAIDOQUBrkvAO2sAKuIPmnNOHvviw7nr8\nSAtXBQAAlkJAaxIvnZKZ1JtN3j/pcM4fmH5mLtrA9McOntLnfvCC/uq7z7Z4ZQAAoJLkpYkO5WVS\n6vcyMrN2L+U8tQ5Mv33PuCRp9/7jzPAEAKANCGhNkk1b4nqghUbCeZwR7qEVS2Xd+dBBXbyuX2Un\n3f0Ex5wAAMSNgNYkXiaVyPtn0tkdtCj30O575iVNnM7rN99ymTYO9nIPDQCANiCgNUm/l9Gq3oQG\ntGAHLco8zi/tGdfq3oxuuHy93rxjVPfsPRb57hoAAGgOAlqT/OZbL9Pvv2Nnu5dRUTgw/USVO2jT\n+aK+9uhhve2qDerNpnXjjlHNzpV0795jcSwTAAAECGhNcvkFq3XNpqF2L6OicAet2hHnXY8f0exc\nSTdfMyZJetVFa7SqJ8MxJwAAMSOgdYFsOqVVvZmqFZm37xnX2FCfXrF1RJJ/r+76y9fr7iePqFR2\ncSwVAACIgNY1hqtME5g4ndc9eyf0jms2KpU62yrkxh2jOjZV0IMvnIhjmQAAQAS0rjHc7y17B+3L\nDx1U2Uk/8bKxc55/w6XrlEmZ7nr8aKuXCAAAAgS0LjFSZWD6lx4c186x1do+uuqc5wf7snr1xWt0\n1+OHW71EAAAQIKB1ieGct+QdtH1Hp/Twi5PzxQGL3bhjVM9MTOvZialWLhEAAAQIaF3CP+KsHNDu\neHBcKZPefvXGit9/845RSaKaEwCAmBDQusRIv6eZCgPTnXO6fc+4rt22VutX91Z879hQn67YuJqA\nBgBATAhoXSLshbZ4YPr9+0/oxROz5xUHLHbjjlHdf+CEjk3lW7ZGAADgI6B1ieGcP01g8T202/eM\nqy+b1o9cccGy779xx6ick775BNWcAAC0GgGtS4QD008uuIdWKJb1lYcP6S1XjKq/yqD3HRtWa2yo\nT9/gmBMAgJYjoHWJkf7zB6Z/+6mjmpyd081Vjjclycx0445R3btvQrMFhqcDANBKBLQuMRQccS7s\nhXb7nnGt6fd03ba1kX7GjTtGdWaurHv2TrRkjQAAwEdA6xJhkcDxab9IYHJ2Tnc/cVQ/fvVGZdLR\n/s/glReNaFUvw9MBAGg1AlqXCAemh73QvvrIIRVK5arVm4t/xg2Xrdc3nzzK8HQAAFqIgNZFRhY0\nq719z7guXtuvqy4crOln3LhjVC9NF7TnAMPTAQBoFQJaFxkKxj2Nn5zV9587rptfNiYzq+lnXH/Z\nOmXTxjEnAAAtREDrIiO5rE7MFHTHg+OStOTszeWs6g2HpxPQAABoFQJaFxnu93Riek63PzCuXVuG\ntXlNrq6f85Ydo3r22LT2HWV4OgAArUBA6yIjOU/jJ2e19+hUpN5nS2F4OgAArUVA6yLhNIFs2vS2\nKzfU/XM2DPbpyrFB3fX44WYtDQAALEBA6yJhL7TrL1s/H9bqdeOOUe154aQmTjM8HQCAZiOgdZG1\nA34oq6c4YLFwePrdT3DMCQBAsxHQusgbLlunj7/rar115wUN/6zLL1ilC4f7uIcGAEALENC6SE8m\nrXe+/EKlU7X1Pqvk7PD0Y5opFJuwOgAAECKgoW43/tCo8sWyvvv0sXYvBQCAFYWAhrq94qIRre7N\n6F+5hwYAQFMR0FC3bDql6y9br+8+PdHupQAAsKIQ0NCQyy5YpaOn85rOcw8NAIBmIaChIVuCcVEv\nnJhp80oAAFg5CGhoyOYRP6Dtf4mABgBAsxDQ0JAtI/2SpBeOE9AAAGgWAhoaMpjLanVvhh00AACa\niICGhm1Z06/97KABANA0BDQ0bPNIjiNOAACaiICGhm1ek9OLJ2ZUKrt2LwUAgBWBgIaGbR7Jaa7k\ndGhytt1LAQBgRSCgoWFbglYbBygUAACgKQhoaNimMKBxDw0AgKYgoKFhG4f6lE0blZwAADQJAQ0N\nS6dMFw7n2EEDAKBJCGhoik0jOe6gAQDQJAQ0NMWWEXbQAABoltgDmpltMrNvmdnjZvaYmX0geH7E\nzO4ys73B38Nxrw312zyS0+TsnCZn5tq9FAAAOl47dtCKkn7dObdD0qslvd/Mdkj6oKS7nXPbJd0d\nPEaH2LyGSk4AAJol9oDmnDvknHsg+Pq0pCckjUl6h6TbgpfdJunmuNeG+m0OWm3sPz7d5pUAAND5\n2noHzcy2SnqZpO9LGnXOHQq+dVjSaJuWhTpsTnAvtGcmpvTy/36XHnrhZLuXAgBAJG0LaGY2IOkL\nkn7FOXdq4fecc05SxcGOZvY+M9ttZrsnJiZiWCmi6O/JaO2Al8hKzk/e+5yOTxf0/edeavdSAACI\npC0Bzcyy8sPZ3zrnvhg8fcTMNgTf3yDpaKX3Oududc7tcs7tWrduXTwLRiSbR3Lan7CANjkzpy8+\nMC5JevrIVJtXAwBANO2o4jRJfy3pCefcxxd8605JtwRf3yLpjrjXhsZsTmCrjX/YfUCzcyWNDfVp\n71ECGgCgM7RjB+1aST8j6Y1m9mDw58ckfVTSjWa2V9Kbg8foIJvX9OvQ5KwKxXK7lyJJKpWdbvve\nfr3qohHduGNU+46cln96DgBAsmXi/oXOuXsl2RLfflOca0FzbR7Jqeyk8ZOzumhtf7uXo7seP6Lx\nk7P6vZt26NhUXtOFkg5OntHYUF+7lwYAwLKYJICm2RL0Qtv/UjJabXz6e89pbKhPN+4Y1aWjqyRJ\nTx853eZVAQBQHQENTRO22nghAffQnjh0Sv/+7HHd8totSqdM29cPSJL2USgAAOgABDQ0zfpVPerJ\npBJRyfnp+55XXzatn9q1WZI03O9p7UAPO2gAgI5AQEPTmFkiKjmPTxf0pQfH9c6Xj2kwl51/fvv6\ngURXcj46PkmABABIIqChybasaX9A+9wPDihfLOvnXrv1nOcvHR3QvqNTiazkPHVmTj/z19/Xh774\nSLuXAgBIAAIammpTsIPWrhA0Vyrrb/5tv67bvlbbg8KA0LbRVZrKF3Vo8kxb1racv/zOMzoxM6fH\nDk6qWEpGmxIAQPsQ0NBUW0ZymimUdGyq0Jbf//XHDuvwqTPn7Z5J0qVBoUDSjhGPnDqjv773Oa3p\n93RmrqxnjyWjChYA0D4ENDTV5jXtHZr+qfue15Y1Od1w2frzvhfuqO1L2D20P/7XvSqVnf7wJ6+S\n5N9FAwB0NwIammrziN+g9sDx+HeBHn7xpO7ff0K3vGarUqnzeyGP9HtaO+Alagdt39EpfX73C/rP\nr9qi6y9br95sSo+On2r3sgAAbUZAQ1NdONwnM+nAS7Ox/+5P3/e8+r20fnLXhUu+ZlvCKjn/6OtP\nqTeT0i++cZvSKdOODav16EF20ACg2xHQ0FS92bQuWN2r/THvoB09fUZffvig/tOuTVrdm13ydZeO\nrtK+I8mo5HzgwAl97bHDet/rL9HagR5J0pVjg3r84CmVy+1fHwCgfQhoaLrNIzkdiLlZ7d99/4Dm\nSk4/+5oty75u+/oBnc4XdfhUeys5nXP66L88qbUDnv7P6y6af/6KsUFN5Yt6PiHjsgAA7UFAQ9PF\n3ay2UCzrs/9+QDdctk4XrxtY9rXb52dytveY81tPHdUPnj+uD7xpu/p7MvPP79w4KEl69CD30ACg\nmxHQ0HRb1uR09HRes4VSLL/vXx45pGNTef3ctRdVfW04k3NvGwsFSmWnj331KW1dk9O7X7n5nO9t\nHx2Ql0npMSo5AaCrEdDQdJvCoeknWr+L5pzTp+57Tpes69frt6+t+vo1Az1a0+9pbxt30G7fM66n\njpzWb/7I5cqmz/2fYDad0g9dsIpCAQDocgQ0NN2WNX6rjTiGpu954aQeenFSP/farTI7v7VGJX4l\nZ3t20M7MlfTxbzylqy8c1I9deUHF11wxNqhHx08lopABANAeBDQ03eaR+JrVfuq+57WqN6N3vnzp\n1hqLXTq6SnvbVMn5mX97Xgcnz+i3f/TyJQPlzo2Dmpyd04sn4m9VAgBIBgIamm44l9WqnowOtLgS\n8fDkGX31kUP6qV2bzrloX832Ub+S88ipfAtXd77JmTn92bee0RsuXafXXrL0cezOsdWSmCgAAN2M\ngIamM7P5oemt9Off3qeSc7qlwtzN5WxfH1ZyxnvM+RffeUanzszpt996+bKvu3R0lTIp4x4aAHQx\nAhpaYsuanPa3MKB968mj+sy/7dctr9k6X5QQ1fbRoJIzxokChyZn9an7ntPN14xpx8bVy762N5vW\npaOr9AgjnwCgaxHQ0BKbR3J68fhsSzriT5zO6zf/6SFdfsEqffBHl9+NqmTtQI9G+r1YW2388V17\n5Zz0azdeGun1O8dW67HxSQoFAKBLEdDQEpvX5FQolZvesb9cdvqNf3xIp88U9T9/+mXqzabr+jlx\nzuTce+S0/vH+F/SeV2+JvNu3c2xQL00X2j7xAADQHgQ0tESrKjk/9b3n9Z2nJ/S7N+3QpcFUgHpc\nOjqgp4+cbvkOlXNOH/vaU8p5Gf3iG7dFft8V4UQBjjkBoCsR0NASW0b8XmjNnMn52MFJfeyrT+rN\nPzSq97xqc/U3LGP7+lU6faaoo6dbU8lZLJX15YcO6sf/9F796xNH9F+vv0Qj/V7k9+/YsFopo5IT\nALpV9N4EQA02DPUqnbKm7aDNFkr65c/t0XB/Vn/4k1dFbkq7lLBQ4OkjpzW6urcZS5TkN6L9x90v\n6K/ueU4Hjs/o4rX9+ug7r9R/2rWppp/T56W1bf0AAQ0AuhQBDS2RTac0NtTXtErOP/jK43r22LQ+\n+95X1bQTtZSw1cbeI1O6bvu6hn/eyZmC/ubf9uvT33teL00XdM2mIf3Oj/2Q3rJjVKlUfWFy58ZB\n3ffMsYbXBgDoPAQ0tMzmJvVC+9qjh/S5HxzQL7zhEl27rfq8zSjWDngazmUbHvl08OSsPnHPc/r7\n/zigmUJJN1y2Tr/whkv0yotGGt7lu2JsUF/cM66jp89o/arm7fIBAJKPgIaW2bwmp68+cqihn3Fo\ncla//YVHdNWFg5FbVERhZtq+flXdQ9NfmsrrI//yhO588KAk6e1Xb9T73nCxLr9g+R5ntdgZ9Et7\n7OAprb+MgAYA3YSAhpbZPJLTiZk5nTozp9W92ZrfXyo7/eo/PKi5Ull/8u6Xycs0t6Zl++iAvvzQ\nQTnnat7t+qNvPK0vP3RQP/uarXrvdRdpbKivqWuT/B00SXr0xUndcNn6pv98AEByUcWJltkSttqo\ns5Lz//vOM/r3Z4/r999+hS5a29/MpUmStq8f0KkzRU3UWMmZL5b0zw8f1E1XbdT//eM7WhLOJGmg\nJ6OL1/Yz8gkAuhABDS2zqYFeaHsOnNDH73paN121QT/5wxc2e2mSNN9H7ekajzm/9eSETp0p6uaX\njbViWee4YmyQXmgA0IUIaGiZzWvqC2hT+aI+8PcP6oLVvfrIT1zZ8GX7pWybn8lZW6HAl/aMa+1A\nj669ZE0rlnWOnRtXa/zkrE5MF1r+uwAAyUFAQ8us7s1qOJfV/hqPOD98x2N68cSM/vjd12iwr/a7\na1GtG+jRUC5b0w7a5MycvvnkUb396o3KpFv/P5+dwT20xw6yiwYA3YSAhpbavKZfL9Swg/bAgRP6\nwgMv6hfecIlesXWkhSsLKzkHtK+GHbR/efSQCqWyfiKG403J74UmSY/QsBYAugoBDS21ZSSn/cen\nI73WOaePfvVJrR3w9H/dEH1uZSO2j67S00emIs/kvH3PuC5Z16+dY81rp7GcwVxWm0b6KBQAgC5D\nQENLbR7J6eDJM5orlau+9ttPTegHzx3XL79puwZ64ukAs339gCZn5zQxVb2S88UTM/rBc8f1zpdf\n2LJ7cZXs3Diox9hBA4CuQkBDS21ek1Op7HTw5OyyryuVnT72tSe1ZU1O735FY4PQaxFWckZpWHvH\ngqa0cdo5NqjnX5rRqTNzsf5eAED7ENDQUpsjttr40p5xPXn4tH7jLZc1vSHtcravDyo5jyx/D805\np9v3jOuVW0fm24fE5YpgosDjFAoAQNcgoKGltgStNpar5DwzV9LH73paV44N6m1XbohraZKkdat6\nNNiX1dNHl99Be+zgKe07OhVL77PFwkrORznmBICuQUBDS42u6pWXSS1byfnZf9+v8ZOz+uCPXq5U\nKr67XdKCSs4qR5y37xmXl07FHiAlae1AjzYM9hLQAKCLENDQUqmUadNw35I7aJOzc/rTb+3TddvX\n6tpta2NenW/76Co9ffT0kpWcxVJZdz50UDdcvk6Dudb1ZVvOFRsH9ShHnADQNQhoaLnNI7kl76D9\n5Xee0cmZOf32Wy+PeVVnbV8/oJMzczo2Vblb//eeeUkTp/Ox9T6rZOfYaj0zMaWZQrFtawAAxIeA\nhpbbsqZfB47PnLdDdXjyjD5533O6+ZqN8/es2uFsJWflQoEv7RnX6t6Mrr9sfZzLOseVY4NyjkIB\nAOgWBDS03KaRnKbyRR1fNE/yT+5+WqWy06+/5bI2rcy3fX4m5/n30GYKRX3tscN621Ub1JtNx720\neRQKAEB3IaCh5bZUaLWx7+iU/uE/XtB7Xr0l9rYVi61f1aPVvRk9XWEH7a7Hj2imUNLN17TveFPy\n17h2oId7aADQJQhoaLnNa84PaP/v159UzsvoF2Ma6bQcM9P20VUVd9Bu3zOusaG+ls8FrcbMtHNs\nNTtoANAlCGhouU3DQUALKjnv339CX3/siH7+9RdrzUBPO5c279LRAe09cm4l58TpvO7Ze0zvuGZj\n7O0/Ktm5cVB7j07pzFyp3UsBALQYAQ0t1+eltX5Vj/YHhQIf++qTWjvQo/ded1G7lzZv2/pVOjEz\np5cW3JP7ysMHVSq7tlZvLrRzbFClstOTh5efegAA6HwENMRiyxq/1cY3nzyqHzx/XL/y5u3KefEM\nRI/i0qBQYOE9tC/tGdcVG1dre1Dl2W47x/yRTxxzAsDKR0BDLDaN5PT8sWl97GtP6uK1/fqpV2xq\n95LOsX29H8L2BffQnpmY0kMvTiZm90ySxob6NJTL6rGDBDQAWOkIaIjFlpF+HT2d19NHpvSbP3KZ\nsulk/Z/e6OoerVpQyXnHnnGlTPrxqze2eWVnmZl2bhzUo+NUcgLASpes/0pixdq8pk+SdPWmIb11\n5wVtXs35wpmce49MyTmn2x8c17Xb1mp0dW+7l3aOK8ZW66nDp1Uoltu9FABACxHQEIurLxzSqt6M\nfvdtPySz9ldEVnJp0Grj/v0n9MLx2UQdb4auHBtUoVSu2LMNALByENAQi4vXDejhD7+l7f3ElrNt\n/YCOTxf01/c+p75sWj9yRfJ2+nZu9CcKcA8NAFY2Ahpik9Sds1A4k/Orjx7WW64YVX9PcqpMQ5tH\nclrVk+EeGgCscAQ0IBDO5JSkmxN4vClJqZRpx8bVepQdNABY0ZK3RQC0yQWre7WqJyMvk9J129a2\nezlLunJsUJ/5t/16/9890O6lAMCKsWPDar0/AeMHQwQ0IGBmes9rtmjDYK8yCWsDstCPXnmB7tl7\nTE8e4pgTAJpldW+23Us4hy2cPdhpdu3a5Xbv3t3uZQAAAFRlZvc753ZFeW1ytwkAAAC6FAENAAAg\nYQhoAAAACUNAAwAASBgCGgAAQMIQ0AAAABKGgAYAAJAwBDQAAICEIaABAAAkDAENAAAgYQhoAAAA\nCUNAAwAASBgCGgAAQMIQ0AAAABKGgAYAAJAwBDQAAICEIaABAAAkDAENAAAgYQhoAAAACUNAAwAA\nSBgCGgAAQMIQ0AAAABKGgAYAAJAw5pxr9xrqZmYTkvbH8KvWSjoWw+9B7fhsko3PJ7n4bJKNzye5\nGvlstjjn1kV5YUcHtLiY2W7n3K52rwPn47NJNj6f5OKzSTY+n+SK67PhiBMAACBhCGgAAAAJQ0CL\n5tZ2LwBL4rNJNj6f5OKzSTY+n+SK5bPhDhoAAEDCsIMGAACQMAS0ZZjZW83sKTPbZ2YfbPd6up2Z\nfdLMjprZowueGzGzu8xsb/D3cDvX2K3MbJOZfcvMHjezx8zsA8HzfD4JYGa9ZvYDM3so+Hx+P3ie\nzychzCxtZnvM7CvBYz6bhDCz583sETN70Mx2B8+1/PMhoC3BzNKS/kzSj0raIemnzWxHe1fV9T4t\n6a2LnvugpLudc9sl3R08RvyKkn7dObdD0qslvT/43wufTzLkJb3ROXe1pGskvdXMXi0+nyT5gKQn\nFjzms0mWG5xz1yxor9Hyz4eAtrRXStrnnHvWOVeQ9PeS3tHmNXU159x3JR1f9PQ7JN0WfH2bpJtj\nXRQkSc65Q865B4KvT8v/D82Y+HwSwfmmgofZ4I8Tn08imNmFkt4m6RMLnuazSbaWfz4EtKWNSXph\nwWUFqPIAAAOxSURBVOMXg+eQLKPOuUPB14cljbZzMZDMbKukl0n6vvh8EiM4QntQ0lFJdznn+HyS\n448l/Zak8oLn+GySw0n6VzO738zeFzzX8s8n0+wfCLSLc86ZGWXJbWRmA5K+IOlXnHOnzGz+e3w+\n7eWcK0m6xsyGJN1uZjsXfZ/Ppw3M7CZJR51z95vZ9ZVew2fTdq9zzo2b2XpJd5nZkwu/2arPhx20\npY1L2rTg8YXBc0iWI2a2QZKCv4+2eT1dy8yy8sPZ3zrnvhg8zeeTMM65k5K+Jf8+J59P+10r6e1m\n9rz8qzRvNLPPis8mMZxz48HfRyXdLv8KVMs/HwLa0v5D0nYzu8jMPEnvlnRnm9eE890p6Zbg61sk\n3dHGtXQt87fK/lrSE865jy/4Fp9PApjZumDnTGbWJ+lGSU+Kz6ftnHMfcs5d6JzbKv+/M990zr1H\nfDaJYGb9ZrYq/FrSWyQ9qhg+HxrVLsPMfkz+3YC0pE865z7S5iV1NTP7nKTrJa2VdETShyV9SdLn\nJW2WtF/Su5xziwsJ0GJm9jpJ90h6RGfv0fyO/HtofD5tZmZXyb/InJb//zH/vHPuD8xsjfh8EiM4\n4vwN59xNfDbJYGYXy981k/xrYX/nnPtIHJ8PAQ0AACBhOOIEAABIGAIaAABAwhDQAAAAEoaABgAA\nkDAENAAAgIQhoAFY0cxsjZk9GPw5bGbjCx57i1779bDn0TI/78WwpxgAtAptNgB0DTP7b5KmnHN/\ntOh5k///HpYrvvHc174oaWfQkR8AWoIdNABdycy2mdnjZva30v/f3h2rRhFFYRz/f50IYWsxXRoh\nxTYRBMEqhbG2lNhIsE3rE+x7pLYS8RFE2GbFNaQI+ABhg4gIKTwpdgPD7rZZLjv/Xzn3XDjTfZw7\nzGUKPOpOx5J8WlyOPE3ybs3+nSRfkkyS/EjyetPvIGl7eVm6pD57AhxX1Rige7k78LaqZkkeAuMk\nH6vqurP+CvhVVUeLvYNNNS1p+zlBk9Rnl3fhbI3TJBPgK7AL7C2tfwdeJhkleV5Vv++zUUn9YkCT\n1Gd/1z1Mcgi8AJ5V1ZB5GHvQramqc+CA+fHoKMmHe+5VUo94xClJqwbArKr+JdkHni4XJHkMXFXV\nWZI/wJtNNylpexnQJGnVZ+AkyU/gAvi2pmbIfHL2H7gB3m+wP0lbzt9sSJIkNcZv0CRJkhpjQJMk\nSWqMAU2SJKkxBjRJkqTGGNAkSZIaY0CTJElqjAFNkiSpMQY0SZKkxtwClIQmaVIbfX0AAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10ecd0438>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure(figsize=(10, 10))\n",
    "plotSteps(50, stepsToGoal)\n",
    "clear_output(wait=True)\n",
    "display(fig)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}