Seminar12/a2c_kungfu_dmia.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Playing atari with advantage actor-critic\n",
    "\n",
    "This time we're going to learn something harder then CartPole :)\n",
    "\n",
    "Gym atari games only allow raw image pixels as observation, hence demanding a more powerful agent network to find meaningful features. We shall use a convolutional neural network for such task.\n",
    "\n",
    "Most of the code in this notebook is written for you, however you are _strongly encouraged to experiment with it_ to find better agent configuration and/or learning algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "%matplotlib inline\n",
    "\n",
    "#setup theano/lasagne. Prefer GPU\n",
    "%env THEANO_FLAGS=device=gpu,floatX=float32\n",
    "\n",
    "#If you are running on a server, launch xvfb to record game videos\n",
    "#Please make sure you have xvfb installed (apt-get install xvfb, see gym readme on xvfb)\n",
    "import os\n",
    "if os.environ.get(\"DISPLAY\") is str and len(os.environ.get(\"DISPLAY\"))!=0:\n",
    "    !bash xvfb start\n",
    "    %env DISPLAY=:1\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Processing game image\n",
    "\n",
    "Raw atari images are large, 210x160x3 by default. However, we don't need that level of detail in order to learn them.\n",
    "\n",
    "We can thus save a lot of time by preprocessing game image, including\n",
    "* Resizing to a smaller shape\n",
    "* Converting to grayscale\n",
    "* Cropping irrelevant image parts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gym.core import ObservationWrapper\n",
    "from gym.spaces import Box\n",
    "from scipy.misc import imresize\n",
    "\n",
    "class PreprocessAtari(ObservationWrapper):\n",
    "    def __init__(self, env):\n",
    "        \"\"\"A gym wrapper that crops, scales image into the desired shapes and optionally grayscales it.\"\"\"\n",
    "        ObservationWrapper.__init__(self,env)\n",
    "        \n",
    "        self.img_size = (64, 64)\n",
    "        self.observation_space = Box(0.0, 1.0, self.img_size)\n",
    "\n",
    "    def _observation(self, img):\n",
    "        \"\"\"what happens to each observation\"\"\"\n",
    "        \n",
    "        # Here's what you need to do:\n",
    "        #  * crop image, remove irrelevant parts\n",
    "        #  * resize image to self.img_size \n",
    "        #     (use imresize imported above or any library you want,\n",
    "        #      e.g. opencv, skimage, PIL, keras)\n",
    "        #  * cast image to grayscale\n",
    "        #  * convert image pixels to (0,1) range, float32 type\n",
    "        \n",
    "        <Your code here>        \n",
    "        return <...>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-03-29 01:18:10,194] The cuda backend is deprecated and will be removed in the next release (v0.10).  Please switch to the gpuarray backend. You can get more information about how to switch at this URL:\n",
      " https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29\n",
      "\n",
      "Using gpu device 2: GeForce GTX 1080 (CNMeM is enabled with initial size: 22.0% of memory, cuDNN 5110)\n",
      "[2017-03-29 01:18:21,072] Making new env: KungFuMaster-v0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f7e270b5c10>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFA1JREFUeJzt3X+sHWWdx/H3p/f20t4C/UGhViGApkIqQYoNYNS1Baro\nGvEfiSZumg1J/3E3mJgo7CabuH/xl3H/2JA0ytpEV5e4sDREbUqhko2KvVCUllLqsiX059VKpS1t\nb398948z9/Cc2ftj7r3nzDnX5/NKbs4zM8858+30fM8888zMM4oIzCw/c7odgJl1h5PfLFNOfrNM\nOfnNMuXkN8uUk98sU05+s0zNKPkl3SNpr6TfS3qwXUGZWedpuhf5SOoDXgPWAQeAHcCXI+KV9oVn\nZp3SP4P33gb8PiJeB5D0Y+BeYNzkHxgYiMHBQQAWLFgwg1WPrb//3X/O/PnzW5alP3KXXXZZs3zm\nzJmWeu+8806zvHjx4pZlad05c95tNB0/fryl3vnz55vlCxcuVIp9NpDUMp1ub+sNx48f59SpU5q8\n5syS/33Am8n0AeD2id4wODjIJz7xCQBuv33CqtNy1VVXNcs33XRTy7KRkZFmec2aNc3y3r17W+rt\n2LGjWf7iF7/Ysiytm/6APP744y31jh071iyfOHGiSuizQjnZlyxZ0iyXfxisOx555JHKdTve4Sdp\ng6QhSUNpAppZd81kz38QuCaZvrqY1yIiNgIbARYtWtTRu4jOnj3bLPf19bUsO3nyZLP861//ullO\nm+/Qujf7xS9+0bIs3aPPnTu3WT59+nRLvXTdab3Z7i/p32Iz2/PvAFZIul7SAPAlYHN7wjKzTpv2\nnj8izkv6O2AL0Ac8GhG72xaZmXXUjLprI+KnwE/bFIuZ1ajWczVnzpzhtddeAzp/CuxnP/tZRz/f\nrBelfVuT8eW9Zply8ptlqtZmv6TmhSLz5s2rc9VmWSifup6wbgfjMLMe5uQ3y5ST3yxTtR7zX7x4\nsXln3F/SDS9mvWIqp9C95zfLlJPfLFO1NvvnzJnDwMAA0Li338zay6f6zGxSTn6zTE17AM9prUxq\nrmzhwoW1rdcsFydPnuT8+fOVxlTznt8sU05+s0w5+c0yVeupvgULFjSH1L755pvrXLVZFp544onK\ndb3nN8uUk98sU7U2++fPn99s9q9du7bOVZtl4emnn65c13t+s0w5+c0y5eQ3y1Ttz1geHcBz9O6+\nUen0xYsXW5alj7zuFendU+Vn2KXPDOxVl1xySbN87ty5lmXl7d8L0icEl+9c68UHwJafWpx+v8vx\ntvMS+6k8LXnSPb+kRyUNS9qVzFsiaaukfcXr4ok+w8x6T5Vm//eBe0rzHgS2RcQKYFsxbWazyKTN\n/oh4TtJ1pdn3AmuK8iZgO/DNyT7r3LlzHDp0CKD52K5RadO53Ozs9KO9pmO2N/vTZmj5sKrXm/3l\npm35sKUXlGNMvyOdPEyZyndvuh1+yyLicFE+Aiyb5ueYWZfMuLc/Gr0V4/ZYSNogaUjSUC92zJjl\narrJf1TScoDidXi8ihGxMSJWR8Tqcg9/qV7zb7ZJY58t8Zdj7vX4ez2+2Wi6yb8ZWF+U1wNPticc\nM6tLlVN9PwJ+Bdwg6YCk+4GHgXWS9gF3F9NmNotU6e3/8jiL7mpzLGZWo9qv8BtPeiw3G47rZnO8\nY033Om/v9vO1/WaZcvKbZapnmv1mVfViE3o28p7fLFNOfrNMOfnNMtUzx/xTGYSgF6TxzobYyzHO\nhphTabzlwTx68a7P2bC9vec3y5ST3yxTtTf7R5tsfX19LfPT6V4cTKIsbXpO9G/pVWmMs+JqtGR7\nl5v9vbi9y838ib4v3eI9v1mmnPxmmaq12T9v3jw++MEPAnDHHXe0LEub+rOhpzRV7m3ulWbdRNKY\nZ0O8E93YUz4M6EV1be9HH320ct3e32pm1hFOfrNMOfnNMlXrMf/Jkyd57rnnAHjrrbfqXLVZFo4d\nO1a5rvf8Zply8ptlqtZm/+DgIKtWrQJg3bp1da7aLAvPPvts5bre85tlyslvliknv1mmaj3mv3Dh\nAqdOnQLg+PHjda7aLAvlx61PpMrjuq6R9KykVyTtlvRAMX+JpK2S9hWvi2cQs5nVrEqz/zzw9YhY\nCdwBfFXSSuBBYFtErAC2FdNmNktUeVbfYeBwUT4haQ/wPuBeYE1RbROwHfjmRJ914cKFZnP/6NGj\n043ZzMbR1mZ/StJ1wCrgeWBZ8cMAcARYNpXPMrPuqpz8ki4F/hP4WkS8nS6Lxg3WY479JGmDpCFJ\nQyMjIzMK1szap1LyS5pLI/F/GBGPF7OPSlpeLF8ODI/13ojYGBGrI2L1wMBAO2I2szao0tsv4HvA\nnoj4drJoM7C+KK8Hnmx/eGbWKVXO838M+BvgZUkvFfP+AXgYeEzS/cAbwH2dCdHMOqFKb/9/A+MN\nondXe8Mxs7r48l6zTDn5zTLl5DfLlJPfLFNOfrNMOfnNMuXkN8uUk98sU05+s0w5+c0y5eQ3y5ST\n3yxTTn6zTDn5zTLl5DfLlJPfLFNOfrNMOfnNMuXkN8uUk98sU05+s0w5+c0y5eQ3y5ST3yxTTn6z\nTFV5Vt88Sb+R9FtJuyV9q5i/RNJWSfuK18WdD9fM2qXKnv8scGdEfBi4BbhH0h3Ag8C2iFgBbCum\nzWyWmDT5o+FkMTm3+AvgXmBTMX8T8IWORGhmHVHpmF9SX/GE3mFga0Q8DyyLiMNFlSPAsg7FaGYd\nUCn5I+JCRNwCXA3cJumm0vKg0Rr4fyRtkDQkaWhkZGTGAZtZe0yptz8ijgPPAvcARyUtByheh8d5\nz8aIWB0RqwcGBmYar5m1SZXe/islLSrK84F1wKvAZmB9UW098GSngjSz9uuvUGc5sElSH40fi8ci\n4ilJvwIek3Q/8AZwXwfjNLM2mzT5I+J3wKox5h8D7upEUGbWeb7CzyxTTn6zTDn5zTLl5DfLlJPf\nLFNOfrNMOfnNMuXkN8uUk98sU05+s0w5+c0y5eQ3y5ST3yxTTn6zTDn5zTLl5DfLlJPfLFNOfrNM\nOfnNMuXkN8uUk98sU05+s0w5+c0y5eQ3y5ST3yxTlZO/eEz3TklPFdNLJG2VtK94Xdy5MM2s3aay\n538A2JNMPwhsi4gVwLZi2sxmiUrJL+lq4K+B7yaz7wU2FeVNwBfaG5qZdVLVPf93gG8AF5N5yyLi\ncFE+AixrZ2Bm1lmTJr+kzwHDEfHCeHUiIoAY5/0bJA1JGhoZGZl+pGbWVpM+ohv4GPB5SZ8F5gGX\nS/oBcFTS8og4LGk5MDzWmyNiI7ARYNGiRWP+QJhZ/Sbd80fEQxFxdURcB3wJeCYivgJsBtYX1dYD\nT3YsSjNru5mc538YWCdpH3B3MW1ms0SVZn9TRGwHthflY8Bd7Q/JzOrgK/zMMuXkN8uUk98sU05+\ns0w5+c0y5eQ3y5ST3yxTTn6zTDn5zTLl5DfL1JQu720HSXWv0szG4D2/Waac/GaZcvKbZarWY/4L\nFy7w5z//GYDh4TEH/jGzGTh37lzlut7zm2XKyW+WqVqb/RcvXuTkyZMAHDt2rM5Vm2Xh/Pnzlet6\nz2+WKSe/WaZqbfb39/ezdOlSAK699to6V22WhUsuuaRyXe/5zTLl5DfLlJPfLFO1HvOPjIxw4MAB\nAAYHB+tctVkWTp8+XblupeSXtB84AVwAzkfEaklLgP8ArgP2A/dFxFtTjNXMumQqzf61EXFLRKwu\nph8EtkXECmBbMW1ms8RMmv33AmuK8iYaz/D75kRvWLBgAatXN3471q5dO4NVm9lYdu7cWblu1T1/\nAE9LekHShmLesog4XJSPAMuqh2hm3VZ1z//xiDgo6Spgq6RX04UREZJirDcWPxYboLHnN7PeUGnP\nHxEHi9dh4AngNuCopOUAxeuYN+hHxMaIWB0Rq+fNm9eeqM1sxibd80taAMyJiBNF+VPAPwObgfXA\nw8Xrk5N91qlTp9ixYwdAc1APM2uf48ePV65bpdm/DHiiGHW3H/j3iPi5pB3AY5LuB94A7ptGrGbW\nJZMmf0S8Dnx4jPnHgLs6EZSZdV7td/VdddVVAFx//fV1rtosC7/85S8r1/W1/WaZcvKbZcrJb5ap\n2u/qe/3114GJBxqcO3duy/TFixeb5Tlz5ow5HxrPBeiGcrxVTWWM9Xbr6+trltNtCuNv727Gm6r6\n/QCIePfas6kMbtlu/f3vplr5eZVpjOmy6Wzvd955p3Jd7/nNMuXkN8tUrc1+ScyfPx+ARYsWtSxb\nsmRJs/zJT36yZdkrr7zSLK9cubJZfvHFF1vqvfHGG22LdSruvvvulukTJ040y2lzD1oHWHzmmWc6\nG9gE0lOtq1atalm2e/fuZjnd3tu3b2+p99Zb9Q3fkA7+8ulPf7pl2d69e5vlD3zgAy3L9u/f3yy/\n/PLLnQmugtG7WQGuuOKKlmWHDh1qltOBbbds2dJSr8phQHo4Nxnv+c0y5eQ3y1Stzf6JDAwMNMtn\nz55tWTZ6qACtPbblJnW3lHuYJ+pJL093S7rtyr3g6fYeGRkZ8z11S7dp+axO+t1Je87L7+umib4T\n450t6vR3pTe+iWZWOye/Waac/GaZ6o2DZlqP1crHdOnxUnoM2ivKV2ylx9Dlf0svjmZU7mNJt3c3\nr4pLpd+P8pWdabzlcet7pY8ljbF8FV76/Sn3WXRSb2wZM6udk98sU6qzmdHf3x+XXXYZAO95z3vK\ny5rl9773vS3Lxjtdc/To0ZbpqdzU0E6jA5SMmmiU4vTqvz/+8Y8di2kyaYzl+FNps//w4cPjLuu0\n9DtQ/u6kp/rKjh071iy//fbb7Q+sovSK1sWLF49b78yZM81y+ftdPtwZy/79+zl9+rQmrYj3/GbZ\ncvKbZcrJb5apWo/5+/r6YvRU10THmeXTM2nd9Di5V05DlZXvWExNZVz1upQv2126dGmzfOTIkbrD\nmbJly959Ulx5+5ZPY/aCcp9Qejn1TPuBDh06xNmzZ33Mb2bjc/KbZarWK/zmzJnD6Km+8oAGqfIV\nc+PV7ZUx5crSZv9EV6P1ivJdZempqPTQqs5DxKlIB4Ipb99ebPanzXxoPQyY6TYeHh7zkZljqrTn\nl7RI0k8kvSppj6SPSloiaaukfcXr+CcvzaznVG32/wvw84i4kcaju/YADwLbImIFsK2YNrNZYtLe\nfkkLgZeA90dSWdJeYE1EHC4e0b09Im6Y6LMGBwfjhhsaVW688cZx61166aUt0+M1lXvlSb/l3vLR\nQ5uxpFeZdWuo8bKFCxeOuyxt9p86daqOcCZVbjanV/iVzxSlvf/dPGxJD2XLZ4PSuNKrVKdzE9uW\nLVv405/+1Lbe/uuBPwD/JmmnpO8Wj+peFhGj13seofE0XzObJaokfz9wK/BIRKwCTlFq4hctgjF/\nViVtkDQkaahXz8ub5ahK8h8ADkTE88X0T2j8GBwtmvsUr2N2M0bExohYHRGre2XMPTOrcKovIo5I\nelPSDRGxF7gLeKX4Ww88XLw+OenK+vu58sorgYkf0Z2OFQ+tx6Rp62Hnzp0t9brVskhPNQF86EMf\nGrfurl27muU6x70vS3+Ib7311pZlaR9Lesy8Z8+ezgdWQfm7s3z58nHrvvDCC81yN0/7pc8d+MhH\nPtKyLO37SZ8zkI7nX1X6XIjJVN0V/z3wQ0kDwOvA39JoNTwm6X7gDeC+KcZpZl1UKfkj4iVg9RiL\n7mpvOGZWl9qv8Bu9sWei02HlmxvSq+TSU2XlUz7dUr6KLx30onx6Ka070Tao04EDB1qmL7/88mY5\nHQyjV+ItD8qRXqGYDoYBracBJxr0o9PSU33l7Z0egqVjEE5ne/txXWY2KSe/Waac/GaZ6skT7+Wx\n1w8ePNilSKopX6ZbHnix16WDio413WvKl72WBxbtRWnfTzcHbk15z2+WKSe/WaZqHcNP0h9oXBC0\nFOiFto/jaOU4WvVCHFON4dqIuLJKxVqTv7lSaSgixrpoyHE4DsdRUwxu9ptlyslvlqluJf/GLq23\nzHG0chyteiGOjsXQlWN+M+s+N/vNMlVr8ku6R9JeSb+XVNtov5IelTQsaVcyr/ahxyVdI+lZSa9I\n2i3pgW7EImmepN9I+m0Rx7e6EUcST18xPuRT3YpD0n5JL0t6SdJQF+OobZj82pJfUh/wr8BngJXA\nlyWtnPhdbfN94J7SvG4MPX4e+HpErATuAL5abIO6YzkL3BkRHwZuAe6RdEcX4hj1AI3h4Ed1K461\nEXFLcmqtG3HUN0x+RNTyB3wU2JJMPwQ8VOP6rwN2JdN7geVFeTmwt65YkhieBNZ1MxZgEHgRuL0b\ncQBXF1/oO4GnuvV/A+wHlpbm1RoHsBD4X4q+uE7HUWez/33Am8n0gWJet3R16HFJ1wGrgOe7EUvR\n1H6JxsCrW6MxQGs3tsl3gG8A6Ygo3YgjgKclvSBpQ5fiqHWYfHf4MfHQ450g6VLgP4GvRUTLsDR1\nxRIRFyLiFhp73tsk3VR3HJI+BwxHxAvj1anx/+bjxfb4DI3Dsb/qQhwzGiZ/qupM/oPANcn01cW8\nbqk09Hi7SZpLI/F/GBGPdzMWgIg4DjxLo0+k7jg+Bnxe0n7gx8Cdkn7QhTiIiIPF6zDwBHBbF+KY\n0TD5U1Vn8u8AVki6vhgF+EvA5hrXX7aZxpDjUHHo8ZlSYyC37wF7IuLb3YpF0pWSFhXl+TT6HV6t\nO46IeCgiro6I62h8H56JiK/UHYekBZIuGy0DnwJ21R1HRBwB3pQ0+ti70WHyOxNHpztSSh0XnwVe\nA/4H+Mca1/sj4DBwjsav6/3AFTQ6mvYBTwNLaojj4zSabL+j8fzDl4ptUmsswM3AziKOXcA/FfNr\n3yZJTGt4t8Ov7u3xfuC3xd/u0e9ml74jtwBDxf/NfwGLOxWHr/Azy5Q7/Mwy5eQ3y5ST3yxTTn6z\nTDn5zTLl5DfLlJPfLFNOfrNM/R9SO1o9RoVp9gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f7e27bee590>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import gym\n",
    "#game maker consider https://gym.openai.com/envs\n",
    "def make_env():\n",
    "    env = gym.make(\"KungFuMaster-v0\")\n",
    "    return PreprocessAtari(env)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "#spawn game instance\n",
    "env = make_env()\n",
    "observation_shape = env.observation_space.shape\n",
    "n_actions = env.action_space.n\n",
    "\n",
    "obs = env.reset()\n",
    "\n",
    "plt.imshow(obs[0],interpolation='none',cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic agent setup\n",
    "Here we define a simple agent that maps game images into policy using simple convolutional neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import theano, lasagne\n",
    "import theano.tensor as T\n",
    "from lasagne.layers import *\n",
    "from agentnet.memory import WindowAugmentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#observation goes here\n",
    "observation_layer = InputLayer((None,)+observation_shape,)\n",
    "\n",
    "#4-tick window over images\n",
    "prev_wnd = InputLayer((None,4)+observation_shape,name='window from last tick')\n",
    "new_wnd = WindowAugmentation(observation_layer,prev_wnd,name='updated window')\n",
    "        \n",
    "#reshape to (frame, h,w). If you don't use grayscale, 4 should become 12.\n",
    "wnd_reshape = reshape(new_wnd, (-1,4*observation_shape[0])+observation_shape[1:])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Network body\n",
    "\n",
    "Here will need to build a convolutional network that consists of 4 layers:\n",
    "* 3 convolutional layers with 32 filters, 5x5 window size, 2x2 stride\n",
    " * Choose any nonlinearity but for softmax\n",
    " * You may want to increase number of filters for the last layer\n",
    "* Dense layer on top of all convolutions\n",
    " * anywhere between 100 and 512 neurons\n",
    "\n",
    "You may find a template for such network below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from lasagne.nonlinearities import rectify,elu,tanh,softmax\n",
    "\n",
    "#network body\n",
    "conv0 = Conv2DLayer(wnd_reshape,<...>)\n",
    "conv1 = <another convolutional layer, growing from conv0>\n",
    "conv2 = <yet another layer...>\n",
    "        \n",
    "dense = DenseLayer(<what is it's input?>,\n",
    "                   nonlinearity=tanh,\n",
    "                   name='dense \"neck\" layer')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Network head\n",
    "\n",
    "You will now need to build output layers.\n",
    "Since we're building advantage actor-critic algorithm, out network will require two outputs:\n",
    "* policy, $pi(a|s)$, defining action probabilities\n",
    "* state value, $V(s)$, defining expected reward from the given state\n",
    "\n",
    "Both those layers will grow from final dense layer from the network body."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#actor head\n",
    "logits_layer = DenseLayer(dense,n_actions,nonlinearity=None) \n",
    "#^^^ separately define pre-softmax policy logits to regularize them later\n",
    "policy_layer = NonlinearityLayer(logits_layer,softmax)\n",
    "\n",
    "#critic head\n",
    "V_layer = DenseLayer(dense,1,nonlinearity=None)\n",
    "\n",
    "#sample actions proportionally to policy_layer\n",
    "from agentnet.resolver import ProbabilisticResolver\n",
    "action_layer = ProbabilisticResolver(policy_layer)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Finally, agent\n",
    "We declare that this network is and MDP agent with such and such inputs, states and outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from agentnet.agent import Agent\n",
    "#all together\n",
    "agent = Agent(observation_layers=observation_layer,\n",
    "              policy_estimators=(logits_layer,V_layer),\n",
    "              agent_states={new_wnd:prev_wnd},\n",
    "              action_layers=action_layer)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[W,\n",
       " b,\n",
       " W,\n",
       " b,\n",
       " W,\n",
       " b,\n",
       " dense.W,\n",
       " dense.b,\n",
       " lstm.b_to_ingate,\n",
       " lstm.W_ctrl0_to_ingate,\n",
       " lstm.W_dense_to_ingate,\n",
       " lstm.b_to_forgetgate,\n",
       " lstm.W_ctrl0_to_forgetgate,\n",
       " lstm.W_dense_to_forgetgate,\n",
       " lstm.b_to_cell,\n",
       " lstm.W_ctrl0_to_cell,\n",
       " lstm.W_dense_to_cell,\n",
       " lstm.b_to_outgate,\n",
       " lstm.W_ctrl0_to_outgate,\n",
       " lstm.W_dense_to_outgate,\n",
       " lstm.W_cell_to_ingate_peephole.scales,\n",
       " lstm.W_cell_to_forgetgate_peephole.scales,\n",
       " lstm.W_cell_to_outgate_peephole.scales,\n",
       " W,\n",
       " b,\n",
       " W,\n",
       " b]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Since it's a single lasagne network, one can get it's weights, output, etc\n",
    "weights = lasagne.layers.get_all_params([V_layer,policy_layer],trainable=True)\n",
    "weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create and manage a pool of atari sessions to play with\n",
    "\n",
    "* To make training more stable, we shall have an entire batch of game sessions each happening independent of others\n",
    "* Why several parallel agents help training: http://arxiv.org/pdf/1602.01783v1.pdf\n",
    "* Alternative approach: store more sessions: https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-03-29 01:18:50,684] Making new env: KungFuMaster-v0\n",
      "[2017-03-29 01:18:50,721] Making new env: KungFuMaster-v0\n",
      "[2017-03-29 01:18:50,755] Making new env: KungFuMaster-v0\n",
      "[2017-03-29 01:18:50,790] Making new env: KungFuMaster-v0\n",
      "[2017-03-29 01:18:50,825] Making new env: KungFuMaster-v0\n",
      "[2017-03-29 01:18:50,859] Making new env: KungFuMaster-v0\n",
      "[2017-03-29 01:18:50,900] Making new env: KungFuMaster-v0\n",
      "[2017-03-29 01:18:50,939] Making new env: KungFuMaster-v0\n",
      "[2017-03-29 01:18:50,978] Making new env: KungFuMaster-v0\n",
      "[2017-03-29 01:18:51,017] Making new env: KungFuMaster-v0\n"
     ]
    }
   ],
   "source": [
    "from agentnet.experiments.openai_gym.pool import EnvPool\n",
    "\n",
    "#number of parallel agents \n",
    "N_AGENTS = 10\n",
    "\n",
    "pool = EnvPool(agent,make_env, N_AGENTS) #may need to adjust\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "actions:\n",
      "[10  1  5 12  8  1  6  0  5 12]\n",
      "rewards\n",
      "[ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]\n",
      "CPU times: user 261 ms, sys: 16.1 ms, total: 277 ms\n",
      "Wall time: 275 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#interact for 7 ticks\n",
    "_,action_log,reward_log,_,_,_  = pool.interact(10)\n",
    "\n",
    "print('actions:')\n",
    "print(action_log[0])\n",
    "print(\"rewards\")\n",
    "print(reward_log[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# batch sequence length (frames) \n",
    "SEQ_LENGTH = 25\n",
    "\n",
    "#load first sessions (this function calls interact and remembers sessions)\n",
    "pool.update(SEQ_LENGTH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advantage actor-critic\n",
    "\n",
    "* An agent has a method that produces symbolic environment interaction sessions\n",
    "* Such sessions are in sequences of observations, agent memory, actions, q-values,etc\n",
    "  * one has to pre-define maximum session length.\n",
    "\n",
    "* SessionPool also stores rewards, alive indicators, etc.\n",
    "* Code mostly copied from [here](https://github.com/yandexdataschool/tinyverse/blob/0b359aa6a5a9f666d2fa9eab97669c7930b7acb3/atari.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#get agent's Qvalues obtained via experience replay\n",
    "#we don't unroll scan here and propagate automatic updates\n",
    "#to speed up compilation at a cost of runtime speed\n",
    "replay = pool.experience_replay\n",
    "\n",
    "_,_,_,_,(logits_seq,V_seq) = agent.get_sessions(\n",
    "    replay,\n",
    "    session_length=SEQ_LENGTH,\n",
    "    experience_replay=True,\n",
    "    unroll_scan=False,\n",
    ")\n",
    "\n",
    "auto_updates = agent.get_automatic_updates()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "# compute pi(a|s) and log(pi(a|s)) manually [use logsoftmax]\n",
    "# we can't guarantee that theano optimizes logsoftmax automatically since it's still in dev\n",
    "logits_flat = logits_seq.reshape([-1,logits_seq.shape[-1]])\n",
    "policy_seq = T.nnet.softmax(logits_flat).reshape(logits_seq.shape)\n",
    "logpolicy_seq = T.nnet.logsoftmax(logits_flat).reshape(logits_seq.shape)\n",
    "        \n",
    "# get policy gradient\n",
    "from agentnet.learning import a2c\n",
    "elwise_actor_loss,elwise_critic_loss = a2c.get_elementwise_objective(policy=logpolicy_seq,\n",
    "                                                                     treat_policy_as_logpolicy=True,\n",
    "                                                                     state_values=V_seq[:,:,0],\n",
    "                                                                     actions=replay.actions[0],\n",
    "                                                                     rewards=replay.rewards/100.,\n",
    "                                                                     is_alive=replay.is_alive,\n",
    "                                                                     gamma_or_gammas=0.99,\n",
    "                                                                     n_steps=None,\n",
    "                                                                     return_separate=True)\n",
    "        \n",
    "# (you can change them more or less harmlessly, this usually just makes learning faster/slower)\n",
    "# also regularize to prioritize exploration\n",
    "reg_logits = T.mean(logits_seq**2)\n",
    "reg_entropy = T.mean(T.sum(policy_seq*logpolicy_seq,axis=-1))\n",
    "\n",
    "#add-up loss components with magic numbers \n",
    "loss = 0.1*elwise_actor_loss.mean() +\\\n",
    "       0.25*elwise_critic_loss.mean() +\\\n",
    "       1e-3*reg_entropy +\\\n",
    "       1e-3*reg_logits\n",
    "\n",
    "        \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Compute weight updates, clip by norm\n",
    "grads = T.grad(loss,weights)\n",
    "grads = lasagne.updates.total_norm_constraint(grads,10)\n",
    "\n",
    "updates = lasagne.updates.adam(grads, weights,1e-4)\n",
    "\n",
    "#compile train function\n",
    "train_step = theano.function([],loss,updates=auto_updates+updates)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demo run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-03-29 01:21:00,533] Making new env: KungFuMaster-v0\n",
      "[2017-03-29 01:21:00,584] Clearing 2 monitor files from previous run (because force=True was provided)\n",
      "[2017-03-29 01:21:00,601] Starting new video recorder writing to /home/justheuristic/agentnet/examples/records/openaigym.video.0.17080.video000000.mp4\n",
      "[2017-03-29 01:21:05,881] Finished writing results. You can upload them to the scoreboard via gym.upload('/home/justheuristic/agentnet/examples/records')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 1261 timesteps with reward=200.0\n"
     ]
    }
   ],
   "source": [
    "untrained_reward = np.mean(pool.evaluate(save_path=\"./records\",\n",
    "                                         record_video=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<video width=\"640\" height=\"480\" controls>\n",
       "  <source src=\"./records/openaigym.video.0.17080.video000000.mp4\" type=\"video/mp4\">\n",
       "</video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#show video\n",
    "from IPython.display import HTML\n",
    "import os\n",
    "\n",
    "video_names = list(filter(lambda s:s.endswith(\".mp4\"),os.listdir(\"./records/\")))\n",
    "\n",
    "HTML(\"\"\"\n",
    "<video width=\"640\" height=\"480\" controls>\n",
    "  <source src=\"{}\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\".format(\"./records/\"+video_names[-1])) #this may or may not be _last_ video. Try other indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#starting epoch\n",
    "epoch_counter = 1\n",
    "\n",
    "#full game rewards\n",
    "rewards = {}\n",
    "loss,reward_per_tick,reward =0,0,0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD8CAYAAACcjGjIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8leXd+PHPNztkkUUIhL03AiLuARYUH7WtWvRpxZZq\n+2htbf21dfR5qm1RrLW21Lq17kGtVByoFBRFWQFlrwBhhJBN9s71++O+z8lZIYeQ5OQk3/frlVfu\nc93jXBfjfM+1xRiDUkop5Sok0BlQSinV9WhwUEop5UWDg1JKKS8aHJRSSnnR4KCUUsqLBgellFJe\nNDgopZTyosFBKaWUFw0OSimlvIQFOgNtlZKSYgYPHhzobCilVFDZtGlToTEmtbXrgjY4DB48mMzM\nzEBnQymlgoqIHPLnOm1WUkop5UWDg1JKKS8aHJRSSnnR4KCUUspLq8FBRKJEZIOIbBGRHSJyv52e\nJCIrRGSf/TvR5Z67RSRLRPaIyGyX9Kkiss0+t1hExE6PFJE37fT1IjK4/YuqlFLKX/7UHGqBS4wx\nk4DJwBwRmQHcBaw0xowAVtqvEZGxwDxgHDAHeFxEQu1nPQHcDIywf+bY6QuAEmPMcOBR4KF2KJtS\nSqk2ajU4GEuF/TLc/jHAVcCLdvqLwNX28VXAG8aYWmPMQSALmC4i6UC8MWadsbafe8njHsez3gJm\nOmoVSimlOp9ffQ4iEioiXwP5wApjzHogzRiTa19yHEizj/sDR1xuP2qn9bePPdPd7jHGNAClQPIp\nl0YppTrYxuxiduWWBTobHc6v4GCMaTTGTAYysGoB4z3OG6zaRIcSkVtEJFNEMgsKCjr67ZRSysu1\nT67lsr9+HuhsdLhTGq1kjDkBfILVV5BnNxVh/863L8sBBrjclmGn5djHnulu94hIGJAAFPl4/6eN\nMdOMMdNSU1ud/a2UUqqN/BmtlCoive3jaOBSYDewDJhvXzYfeMc+XgbMs0cgDcHqeN5gN0GVicgM\nuz/hRo97HM+6Blhl10aUUkoFgD9rK6UDL9ojjkKAJcaY90RkLbBERBYAh4DrAIwxO0RkCbATaABu\nM8Y02s+6FXgBiAaW2z8AzwEvi0gWUIw12kkppbqUpqbm76zGGLrzuJlWg4MxZitwho/0ImBmC/cs\nBBb6SM8ExvtIrwGu9SO/SikVMFX1jc7j0up6eveKCGBuOpbOkFZKKT/ll9U4jwsragOYk46nwUEp\npfy041jzENYjJdWn/bymJuPWVNWVaHBQSik/HTvRHBCy8qy5wWv3F/H0Z/tZvi2XwXe9T355TUu3\ne/nm418w9J4PyCvz/57OErSb/SilVGfLK6ulV0QoDU2GwopaGpsM1z+zDoCMxGgA9hwvp09cVKvP\nMsaw5WgpAHMXf07mby7tuIy3gdYclFLKTwcKKxiQ2IuE6HBKq+vdaglH7WYmfwfhHy6uch4XVtS1\naz7bgwYHpZRqxfacUnYfL2PnsTImZCQ4g0NBuXendEVtg1/P3Hy4pL2z2a60WUkppXxY+pW1FNw3\nz8jgir+tASAiNITk2AgSosM5UVXvc8RSaXW9X8/fnVvu9vrChz9h1Z0XERrSNeZOaM1BKRVUck5U\nd8gw0mMnqjleajUTNTQ28fM3t/DzN7fw2d7mddzqGpvoFR5GWnwkeWU1/G1VFgDP3DjNec2JKv+C\nQ1lNPX3iIrnpnMEAHCqqYtXu/JPf1Ik0OCilupSWVs6prmvk129t5dxFqzhz4X/a/X3PWbSKGQ+u\n5EhxlVvT0I3Pb3C7Lr+8hvSEaA4UVvLV4RMAzBrThzdvmUGI+F9zKKtpIDYqjKq65vc6VFTp89rD\nRVXUNzadapFOiwYHpVSHu+3VzUz5/YpWrzPGMPH+j7lv2Q6vc+9vy+XNzCP2dW3PS0llndfQ0UqX\nYPDNx7+gvKblfoPhfWLp1zvaLU1EOGtoMkkxkZRWt965vPt4Ge9vzSUuKpzJA5ybaDprLg7GGP68\nYi8XPPwJ9y7d1upz25MGB6W6gY3ZxdzwzDq/vrXWNjSy41hpJ+Sq2fvbcimurGuxVuCw+3g55TUN\nvPBltjPtn5lH+J9XNlFS6f6he6ioklfXH2JjdvEp5eWKv63hrAdWuqXluc18ruO4R/A4c3AikzIS\n6N0rnPlnDyYmItR5zrWLIDkmgtzS1ucszPmLteR3XGQY884cwKbfzGJgUi8KPJrLtuWUsnjlPgD+\n/dUx/wrYTrRDWqluYMnGI3y5v4hlW47xvRmDvM5vzC7mUFEV10zN4OEP9/DsmoO8+IPpXDiyc5e+\nP1FVT2KM7/WINmYXc+2Ta73Sf/nWVgDWHnBfxf/Chz91Hr95ywy+PnKCWy4Y6rUYXllNPduPlnLO\n8BTA6rMAKKqoJTk2EoB8j1FHnvm4YEQqP7lkOGDVEsb1S3Ce+/T/Xew8PnNIIks359DQ2ERYaOvf\nvXv3CickREiOjaRvQhQ5HrOuK1xqMHV2s1JTkyGkEzqtteagVJBrajKEh1n/lXe2UCO49sm1/L9/\nbmHR8t18sM3awHH+8xvYl1fu8/qOsu6A1zYtTpsPuQ/trKlvZHtOc3lO1tH741c28eDy3W5zBxx+\n/+5Obnh2PVn57mXNL69lX145f/poj9vMZ1eDk3sB0DchChFxBp4JGc3BoW9C84S36UOSqaxrZGdu\nWYvLYrjWnmaO6eM8HtM3jl25Zc7zO46VcsOz693ufWzVPibe/zFHfJSzvWlwUCrIXfSnT3lt/WEA\nthw5eXPRk6v3c8yl2aPYo6mmtqHR85bTVtfQ3JHqmBHsS4jHN/7R//uhcwipq0XfmuD2OiI0hBI7\ncLy2wfpzKKqo5bX1hzlaUsUh+4N09d5Ct/uKKuq49qm1PPZJFl8fOYEI3Hi2e63rolHWh7drMPAU\nEdb8MTp1kNV/cOVj1rIYmw55N3k58gjQv3cv5/GApF5U1jWSlW8ty/HWpqNu98VEhPLIir1U1Dbw\n4fbjLeanvWhwUCqIldXUu31b9vXN+WSq6pqDwad78hn1mw/ZcuREu+UP4Mv9zR/KLY3GAfB3a4R5\n0wdyw1kDna/rXEbx5JXWcM/SbZyzaBX3LN3GeQ99wi57sbwVO4+zaPlu57XffW69szby0tpDGAO/\nu2o8f79hCgBp8ZHcc/kYPvjp+YzuG+9X3voluC+b4Rjq6urepdut958xkOlDkpzpqXFWE9elj37G\n3rxyIuxmqYjQEOaM60tMZBgCzB6Xxg/PH+JXfk6H9jkoFcQOFjR/2I5Nj2dnbhnlNfXERYU70x0d\nuf0SotxqDWAFl1+/tZWzhibxiyVbANibV87u42XMHJNGit0mfzJPfLqftzcf5aM7LnC2hVfXNfL5\nvgImZCRw0z82Oq892ezh6jr/ay3xdvkSe4U7aw1grX209oB7x225/Z7rDhSz7kDrndeXT+jLpt/M\ncvZHjO3nOzCsvPNCtz4BwKu/42QDBC4fn+72OtXlz/r7/9jo7BtZ8+uLeXL1AT7cYdUWzh6a3Cmb\nDGlwUCpINTYZvtzf3IY/d2I6O3PLOF5a4xYc/rnJGv55/1XjWZJ5hBU785znSqvreTPziHOIKEB2\nUSV//2Q/sI3sRXNbzcdDH1rfxo+UVDEoOQaA+5btcHsmWDUDR3AoKK+lsKKWMenNH7xlNfWEhwrn\nDU/hkz3NE8/e/+l5DEuNZdHy3c42+rgo66MrI7EXJVXNTVWendan4s/XTbLzKc7AcDLDUmNbvcZz\naKqr2Cj3j98pgxLpGx/F8bIaZ2AA6BMfRUpccyd+SlzreWsP2qykVJB6b+sx5wfzKwvOcjZRuA6l\nrKlv5IEPrGtmjenDw9dMdJ7rExfJ2v3eH6ZLN+c4j0/24eZpl8tyEAcKK9zOje4bx6Vj0pzzCc59\naBWX/fVzt2vKqhtIiongO2cOAKyg8MYtMxjXL4Go8FDuu3Ic54+wRlfFR1vBz7Uz+GRfpkelxZ00\n73dfNppvTclopYT+cTRLOZbYcPh0T77z7wugl8twWICo8FDW3eNzc03OcJkL4U9trj1ocFAqCL3w\nxUF+9sbXztdTBvWmb7z1Qen6ge6YA9DPHm2TYH+oXjs1g7OHJbNmn3snLeDW9DTjwZXOb/v55TUM\nvut91u4v4sUvszlQYAUAR9v4E6v38+Tq/QDUNbqP1Pm/K8YSGxVGcWU9RRW1zk5q12am0up64qPC\nmTM+nYMPXs64fgnMGJrss/yR9nvGRYXx1Pemcs/lo70mxo3u2xwQhqTE+HzOrDF9+PxXF/OjC4f5\nPN8Wcyems+wn57LgvCFU1zc6y3rTPzbyxKf7ndfFRoa39AgvZw9r/nNIie2crUm1WUmpTrLneDlD\nU2MI92P8e2ue/uyA2+vo8FDS7ODgWnP43nPW0g9v/uhswGoy2fOHOYSHhLD0qxze+dpqn48IDeHR\n70zmlXWHvJpmPtmdz+HiKmdTx6Mr9rIhu5j0hCjW/PoS6pusD78tR06w5cgJhqXGenVqTxmUyLtb\ncymsqOXZNQed6UeKq5xNS2U19c7g1VqbeqMdCcJChNnj+gIwIi2O77v0bwxNtQLC7uPlbs0yrr47\nYxADknr5PHc6Jmb0di6tUVZT7/Vtf1y/eLdajy93zBrBUJemq2+e0Z+lX+WQGtv6XhHtQWsOSnWC\nvLIaZv/lM3737s42P+OxVfu4+21rQli0R5OEiBARFkJyTATvbj3mthwEQJ/45g+nyLBQQkKEi0c3\nj7Ff9O0JzJ2YzpPfm+r1vre//hUPf7THOVx2g10byS2tobym3usb+80vZXo9Iyo8lBlDrWavTS7z\nGQ4XV7F45T7W7CuksKKWpBYmyHlydEj3TWhexuLiUX1YdeeFzteDk2OcQ0sHJvXi21MyONujJhIX\n1XHfjx2Bzlen9EWjWp98eMeskVw5qZ/z9aJvT+DDO84noZf/NY7ToTUHpTpBkb2Zy/Ltufz+6vF+\n3fPvr3Ioqarj++dawxb/9PFeAB781kRq6puHb37bpa28qLKOoso6Fq/cx92XjwGsIZmRYe7BBHD7\nIE63P2QTosPZeO8s/rxiL6+7jMdvyT8zj7Z47o/XTKS6rtHZtu7owN2VW0ZoiNDYZMgurOTPK6xy\nxUWFeX14t+Sy8X354zUTuXpyf7f0oamx7H/gcl5em8286QPZX1DBq+sPc8noNG65YBj//iqHtQeK\neOCbE4iPDmPKwETfb9AOHMGhrLqeRo8JcTed0/JQ1KmDEt0CqENkWKjfQ2rbgwYHpTpBSZUVHE62\noJunO960+hS+NSXD+UED8OH2XKrrm4d9PmKPsnH11GcHuH3mCABncDmZszzG2//+qnFMHpDAU6sP\ncKCw5bkJCz/Y5TM9NET41hn93ZaQiI20Pm7KaxoY3ieWE1V1bjOmy2sa/G7iCQkRrps2oMX3vsku\n87h+CW4jrq6a3I8+8ZHMGJLc4UtQxEdb5S2trnerPYxMiz1pv8HrN8/wCiaBoMFBqQ5mjOEpu4/A\n8z/9/oIK6hub6N87mtjIMJ9t7YeKKpmY0dv5+sevbG7xvax9Bqx1ghwdxql+jG7x/KAMCw3hO2cO\n5BG7thIbGXbSOQozhiax7kAxq+68kLT4KGIivT9aXIduzhnXl0/35rsNWQW4xKWpqyOICOcMS+nQ\n93BwBPRjJ2pIjbP6a/46bzJXTOx30j4V1xnXgdRqLkRkgIh8IiI7RWSHiPzMTr9PRHJE5Gv753KX\ne+4WkSwR2SMis13Sp4rINvvcYrH/hEQkUkTetNPXi8jg9i+qUoGx6VCJc8OYsFChxuVb/8xHVjPn\nL58z4b6Puf31r5zprtcsWr6bp1Y3j3Jx9YtLR7q9XnrruVw12Wqn3n3cGlqaepJx8Q9fM5H7rxzX\n4vmyGusb75Pfbe6L2HDvTJ6bP83tuj9+exJZCy9jaGqsz8AAzf0EAN+a0t9nW7znUtjBzDHc9p6l\n2/j9e1Zf0/A+sV1mp7fW+BOiGoA7jTFjgRnAbSIy1j73qDFmsv3zAYB9bh4wDpgDPC4ijgbPJ4Cb\ngRH2zxw7fQFQYowZDjwKPHT6RVOqa3D9EKypb2Jbju/1hd7bmus8dh3t8+X+Ih50WfbB4envTeWn\ndtORQ7/e0c4+iJ32shGundGerp02gPn2TmS+/O7K8SREhzMhI4GU2Ah+M3cMfeKiGN+/ea2hEX1i\nGZjcq9VVSCPCQnjmxml8Z9oAhqTE8MqCs7yuiQr37hsJVq41NsfM7PSE4Al+rQYHY0yuMWazfVwO\n7AL6n+SWq4A3jDG1xpiDQBYwXUTSgXhjzDpjLTv4EnC1yz0v2sdvATPlZPUupYKIY0vLP11r9Q14\n7kvgytHslOmjQxKspR0cMhJ9t88PtmcpO0YXtXSdP647cwBf/9+lJESHk/mbS/nh+UMBnMNmAbfl\nK1pz6dg0HrpmIiLCoOQYshfNZdlPzgXgnz8+u8357IpEhKc9Rn/1ju6ckUbt4ZQat+zmnjMAxzqy\nt4vIVhF5XkQc3f79Add580fttP72sWe62z3GmAagFPBv2IJSXVy+3QdwxkCr38Axa9ZzuClY6/c3\nNhne25pLn7hIXv2h+7frx/97qnNy17A+vid2DUzuRVR4iHNButgWmnn81dr3tBe+f+ZpPX9iRm+y\nF83lzMFJrV8cZL4xrq/b7OzO2IehvfgdHEQkFvgXcIcxpgyriWgoMBnIBR7pkBy65+EWEckUkcyC\ngoLWb1DqFJTX+P8N+FTkl9eSEB3u/Lb9q39tpa6hiS+yvGcnl9XU88KX2ezKLSO/vNZtHP59/2W1\n5r528wyW3nqOz+GpDjdM997wp6OMTe+84ZXBKCMxeJqSXPkVHEQkHCswvGqMeRvAGJNnjGk0xjQB\nzwDT7ctzANcxZhl2Wo597Jnudo+IhAEJgNeiL8aYp40x04wx01JTO3cHK9W9vb35KBPu+5g9x9tn\n85t/fHGQB5dbwzwLymvpExfptrXk3rxyHv/Uu5P5RFW928Yzjj0Opg9Ocg7PTIqJ4IxWxuf/Zq41\nx2FoC8tGtIe/3zCFuy4bHVTfhgPhl3NGseC8Iex/4PLWL+5C/BmtJMBzwC5jzJ9d0l3Xm/0msN0+\nXgbMs0cgDcHqeN5gjMkFykRkhv3MG4F3XO6Zbx9fA6wyrW02q1Q7uutta/P2gx4LxrXV/e/u5KnV\n1vDV/PIa+sRHIiK8drPVTHTF39bw9ZETjOjjvrLnoeJK0uwO5KW3nsPY9Hh+fOEw/jJv8im9f0iI\nsPqXF3VoO/7cien8uB3XJOquRveN53+vGBs0o5Qc/GmMPBf4HrBNRBwrfd0DXC8ikwEDZAM/AjDG\n7BCRJcBOrJFOtxljHOPybgVeAKKB5fYPWMHnZRHJAoqxRjsp1Wkci6MVV55+05LrvgRHS6rYfPgE\nF9h7NXsuABcVHsrBBy9nf0Ellz662rkLGFht8SEhwl2XjW5TPhzLZyvVFq0GB2PMGsBXyPvgJPcs\nBBb6SM8EvNYOMMbUANe2lhelOkJ+WfNCdUdLqjhUVEl8VDiJfq7z42l/QfMH/OKV+4DmNXx6R7s/\n89HvTEZEGN4nlv69o/nLf6zro8NDg+6bpupedIa06vH+6bJX787cMmb9eTUNTYYDD1zeph23HMtk\nQ/OQ1D9cZX0ncl0wL2vhZW5zA4akxHC0xOpvCA/VwKACq2vM01YqgByzkc8fkcKnewqobzQYA5Pu\n/7hNG7k7FtkDOFBQydwJ6T5rIZ6Txlw3hukCS+uoHk6Dg+rx9uaVk54Qxdaj7jOXy2oauO21ltcx\naonnGkT+LiZ36dg053FkF1lfR/Vc+i9Q9XiZ2SXMGJrs3N7RVWOTYcYDK0+6UXxlbQMT7/uIwXe9\nz7Itx1i5O4++8VFcbK/ZnxTjPit28fVncK+9nLarn1w83LmGUVdZfE31XPovUPV41fWNJMVEcN6I\nFG6/ZDgAX951Cf3tReCOl9WwN8+a/2CM4b2tx8gtraaitoHxv/2I217bTJm9FPdPX/+KI8XV1sY7\n9to6ngvRXTmpHzdfMNQrHyEhQqq9Y5nrInVKBYJ2SKsezRhDTX0jUeHW96Q7vzGKO78xCoB7547h\n1letZqVnPz/Au1uOkZ4QzUMf7mZYagx//+8pVNQ28Oke79n6eWU13H7JcI4UV3HFhH5e51sSFmLl\nY1x/nXWsAkuDg+rR6hsNTQaifCxFcenYNH46cwSLV+7jox15buf2F1SedOOe2oYmBiXHOPdu9tfE\njAT+fN0k5ozv2/rFSnUgbVZSPUpFbQN3LtlCpj3ctLbBGqnka6no8NAQr/0SXH2+t7nG4Dkl4Uc+\nmo38ISJ8a0oGvSL0e5sKLP0XqHqUH7+8iTVZhfxr81H2LbyMf399DICTTWf44q5LOHfRKufrfglR\nHCutYYe9XwJYK59eObkfZw5OYva4vkS0sreBUl2dBgfVoxwtqXIej7h3ufO4oLy2xXv6e+xOdv6I\nVN7MPMLe/OZF+spqGvjD1RPaMadKBZZ+vVE9SksriLpuy+nLL2ePch7PHNOHiLAQjhQ3r576X5P8\n73RWKhhozUH1KL422AG8ttv0dNvFwzl/RAqLV2ZxwchULh/f19kkdfDBti2zoVRXpjUHFbROVNXx\n4pfZnMrq7pW1jcw/u3kjnHd/ch7r75npnJNwMhMzevPs/GlEhYfyqznNK6VqYFDdkdYcVNC6d+l2\n3t+Wy8SMhFY3vwFoajJU1DaQEB3O4/89hQ0Hi5mQkdCm907xI5goFcw0OKiglV9uLbXtuZZRS6rs\nfoWYyDAun5DO5RPSW7mjZY7lLWaNSWvlSqWCkwYHFbTE3mbkUFEV55+8ywBo7m+IjWqff/a7fz+H\nMN1zQXVT2ueggtZxe5OeZz4/wKGiylavd9QwYiPbJzhEhYd6LbutVHeh/7JVUMovr+FwsTVn4VBR\nFRc+/Gmr9zhqDjE6+1ipVmlwUEFp40Frh7WrJzfPL2hqZYecCnstJM9VUpVS3jQ4qKBT39jE797b\nQUpshHMFVYAXvsxu8Z7qukZ+sWQL0Lyfs1KqZRocVNBZvHIfeWW13H/leAYk9SJ70VwSe4U791zw\n1NDYxJj/+9DZR6E1B6Vap/9LVJe0anceUwcmkdArnF+8+TWf7i3g5vOHctbQJP62Kou4yDAuc1nW\nuk9cFMWVdT6ftcFegdUhJtJ7BVallDutOagu50hxFT94IZO73t4KwNtf5VBcWcdDH+7mkY/3ANA/\nMdptnaSkmAiKK+t4a9NRiipqnc/ZnlPKnuPuNYr2Gq2kVHem/0tUl7MtpxSwRiF5Lo3xRVYRAM/O\nn+aWHh8dxkc78sg8VMKsMX14dv6ZnP/HT3w+P9rH3g1KKXet1hxEZICIfCIiO0Vkh4j8zE5PEpEV\nIrLP/p3ocs/dIpIlIntEZLZL+lQR2WafWyz2ojQiEikib9rp60VkcPsXVQWLL7IKAUiNi6Syzvdq\nqRmJvdxeuzYpbT58gmc/P+B1z+2XDCcmIlTXQlLKD/40KzUAdxpjxgIzgNtEZCxwF7DSGDMCWGm/\nxj43DxgHzAEeFxHHV7UngJuBEfbPHDt9AVBijBkOPAo81A5lU0HK8UGfX17Lyl3N23NePbkfH91x\nAc/eOM3rnoXfnOB2/x/e3+V2/pezrb2hd/xujuetSikfWg0OxphcY8xm+7gc2AX0B64CXrQvexG4\n2j6+CnjDGFNrjDkIZAHTRSQdiDfGrDNWW8FLHvc4nvUWMFP0612PVVpdD0BuaTUHCqyZzyt+fgEP\nXTORUX3jmDXWez2jkWlxZC+ay7RBvhfgi9fhq0qdklPqkLabe84A1gNpxphc+9RxwPE/tj9wxOW2\no3Zaf/vYM93tHmNMA1AKJJ9K3lRwe+frHEqr6ymtrifPHnJ6oqqev67cR0xEKCPS4ogMa72voE+8\n79VSq1vZzEcp5c7vr1MiEgv8C7jDGFPm+sXeGGNExP9F9dtIRG4BbgEYOHBgR7+d6iSHi6r42Rtf\nu6WlxEZSaI86unBUqt/P6hMX5Ty++fwhDE6J4d6l24mLCm+fzCrVQ/hVcxCRcKzA8Kox5m07Oc9u\nKsL+nW+n5wADXG7PsNNy7GPPdLd7RCQMSACKPPNhjHnaGDPNGDMtNdX/DwzVtVTXNXLTPzaw53g5\nNfWNXPCw96iiX1w60nl8/XT/vwgMSrY6qi8cmcq9c8dy/ZkDeeyGM7hu2oBW7lRKufJntJIAzwG7\njDF/djm1DJhvH88H3nFJn2ePQBqC1fG8wW6CKhORGfYzb/S4x/Gsa4BV5lS291Id4kBBRYvbap6O\njdnFfLqngNl/+YzR//uhz2vmuExwS47xf2Mdx6Y/ZwzsDVh7Rl8xsR+hurS2UqfEn2alc4HvAdtE\nxFH3vwdYBCwRkQXAIeA6AGPMDhFZAuzEGul0mzHG0eB7K/ACEA0st3/ACj4vi0gWUIw12kkFUHVd\nI5c8sprZ49J46nveo4Pa6susQm58foNXeliI0GAvnLfoWxNI7NXcDJQSG+H38ycP6M3yn53vrEEo\npdqm1eBgjFkDtPS1a2YL9ywEFvpIzwTG+0ivAa5tLS+q8+w4Zk1EW3eguJUr/X9eZnYJv122w+f5\nVXdexAUPf8Jf503mqsn93c75s7+zqzHp8W3Op1LKouP7lE85J6qB1mcTF1XU8rdVWfx6zmiiI1q+\ndu7iNe6vJ6ZTW9/Ef3blEREawsBkawE9V/dePoYT1XXaJKRUAGhwUD4dL7WGk4YILHx/J6+uP8zG\ne2d5rWj641c2sTG7hLLqet7blstnv7yYPnGRiNDiTORlPzmXiRlWn0B5TT1NTb7zcPMFQ9uvQEqp\nU6IL7ymfcu3gUFHbwDOfH6SqrpGvDp9wu+ZIcRUbs61Nd97+Koe6hiZmPLiSofd8wKMr9vp87uDk\nXs7AABAXFU5CLx1mqlRXo8FBuSmvqeejHcc5ZjcrldU0j1YqqLACRkllHV8dLmkxAAAsXpXlPK5w\nGfE0fUhSe2dZKdUBtFmpB3v28wP0igjjhrOa5xE88MFuXt9wGID+vaOdfQ8AeWXWpLQfvpTJpkMl\nhIUI54915JhLAAAXdklEQVRIISo8lBU78/C0em8BF45M5WiJtdfzTecM5q7LRndkkZRS7USDQw9V\nUdvgXJzONTgcKKhwuy48VKhvtIaYrjtQxOZDJWw6ZDUlNTQZfnj+UFbvKQDgvOEp/OnaSew+XsZN\n/9jI/Oc38D8XDeO5zw8CcNXkfkTpctlKBQUNDkHAGMPilVnMGd+XUX3jTvt5h4uquP6ZdV7pDY1N\nrD/YPHQ1Nc4aQuqoPXxqBwFXkwf0ZmRaLJsPl/DQNRPpmxBFo8v8xSc+3e88HpQcc9p5V0p1Dg0O\nQeCBD3bxzOcHWbHrOO/dfr7buU/25JMcE+HWyduahz7c7dZcVNvQSGRYKLtym3dM+/Wc0cydkE5+\neQ0HCir51b+2ej3np5cMJyE6nITocP5927nO9FSPeQkXj0rl15eNJinG/8lsSqnA0uDQha3eW8CX\n+wt5xm6WOVxU5Xa+sraB7/9jI4DXHIGTOVRc6fb6eGkNg5JjnHstP/ndqc7lKwYm92La4CR+/fZW\njIE7Zo3g+TUHWfGLC0mLj/J6NkBEmPs4h2dunEZYqI59UCqY6P/YLmrZlmPMf34DT61u3tGsrKaB\nrUet4aT3v7uDcb/9qE3Pzimxag297SGkR4qt12v3FzIouZfbukYOSb2sb/3fP3cIW377jRYDg8Oe\nPzRvqqOBQango/9ru6ifvv6V2+vzhqcAcOVjX9DQ2MQ/vsh2O9/U5N86hcYYSqvr+cnFw1nx8wsB\neOqz/VTUNvDpngJmjfHeSAfg9Vtm8MvZo4iPCvNrm03H3gvJ2pSkVFDSZqUuLjYyjAXnDeGOWSMY\ncvcHALyx8YjXdaXV9SR6fBA3NhnWHyzinGEpzrSCilqaDMRHh5EaF0nf+Cg2HyohM7uYhibD+SNS\nPB8NWDutjUw7tc7wdXfPbHX5DaVU16Q1hy6oxN5D+fvnDmb7/bP5+aUjERHeu/08AOecgtduPou/\nzpsMwN8/yaK8pt7tOS+tzeaGZ9azanfzHITpC1cCODe/+eM1E6msa+SFL7MBGJYa227l6JsQpbOf\nlQpSGhy6oA+2W7uvXj4h3S19RFosoSHC6r3WkNIpAxOdbf/PrjnIhPs+drv+qN238IMXMrlzyRa3\nc/F2cJhkj3JyDFPtm3DyvgSlVM+gzUpd0Nubc0iJjWTaoES39MiwUEb0iWX38XImZiQQFR7qtW9B\nTX2jc6JZjMsqqf/afJSvj5Q4Xw/vY9UQ4qPd/wmEa+exUgqtOXQ5pdX1bD5cwg3TB/js+L1kdB+g\neUJZWpz7N/3dx8spqqjl2ie/ZFtOqdu5/QXWENZzhiU7J9OJCBP6JwCw4Lwh7VsYpVTQ0ppDF/Dh\n9lyOFFez4LwhHC2pwpiWN6y5eHQfHv90P1X2YnYhLnsdhAis2p1PXUOTc7VUX269aLjb63dvP4+a\n+kYiw/S7glLKosEhgBoamxARfvzKZgCmDk7ki32FQPPSFZ6mDkzkzktHcuXkfs60v3xnMiEhwl//\ns5fFK/e5XT9lYG9+OXu023IZ5w5P9nqurnmklHKlwSFAqusaGfN/H3LdtAxn2vtbc3lujTUbOqWF\nrTFDQoTbZ45wS7v6DGtbzZe+zAbcZz+fqK7n7GHJjO8fjyC8a494Ukqpk9HgECCOZayXZB4FQKR5\niGpyTIRXR7M/HN/+5589iP/syifnRDVz7RFP79ymQUEp5T9tZO4E9Y1NPLh8F/nlNc60IyXu6yTN\nGpPG4WIr7Y/XTPRrFrKnugZrv80LR6Xyyg/P4tqpGfzkEqt/ITREdC9mpZTfNDh0gi+yCnlq9QHu\nX7bTmbZ4ZZbbNY7lMQDOHubdJ+CPB789gf+5aBjnDk9hSEoMD187ybmMhVJKnQptVupgZTX13GSv\nnJpXVkN2YSVXP/4FJ6rcZzOPSGuemdwrom1/LcNSY/n1HN1pTSl1+jQ4dLCPth93Htc3NvHjVzY5\nA8M5w5KpbWhi2uBEMnqfeh+DUkp1lFaDg4g8D1wB5Btjxttp9wE3A46twe4xxnxgn7sbWAA0Aj81\nxnxkp08FXgCigQ+AnxljjIhEAi8BU4Ei4DvGmOx2Kl/A1dj9AGEhwpajzZPSrpzUj0e/M9nZD1DX\n0ERkWAiTB/i/aY9SSnUUf/ocXgDm+Eh/1Bgz2f5xBIaxwDxgnH3P4yLiaPR+AiugjLB/HM9cAJQY\nY4YDjwIPtbEsXVJOSTXhocJ3ZwxypvXvHc0j101y6yCOCAth7d0zeXnBWYHIplJKuWm15mCM+UxE\nBvv5vKuAN4wxtcBBEckCpotINhBvjFkHICIvAVcDy+177rPvfwt4TETEGOPfBgVdXM6JatITopl/\nzmAOFlbyy9mjGG8vV+FJt9FUSnUVpzNa6XYR2Soiz4uIY4W4/oDrZgNH7bT+9rFnuts9xpgGoBRo\n23CdLuhAQQUZidEMSYnhxR9MbzEwKKVUV9LW4PAEMBSYDOQCj7Rbjk5CRG4RkUwRySwoKGj9hgB7\ncvV+dhwr81pdVSmluro2BQdjTJ4xptEY0wQ8A0y3T+UAA1wuzbDTcuxjz3S3e0QkDEjA6pj29b5P\nG2OmGWOmpaamtiXrnWpXbhkAt1w4LMA5UUqpU9Om4CAirrvQfBPYbh8vA+aJSKSIDMHqeN5gjMkF\nykRkhlhTf28E3nG5Z759fA2wqjv0N1TWNvDO18cAa6tPpZQKJv4MZX0duAhIEZGjwG+Bi0RkMmCA\nbOBHAMaYHSKyBNgJNAC3GWMa7UfdSvNQ1uX2D8BzwMt253Ux1minoPfq+kOBzoJSSrWZP6OVrveR\n/NxJrl8ILPSRngmM95FeA1zbWj6CzZf7rZaxx244I8A5UUqpU6drK3WQ3BM1zBqTxhUT+7V+sVJK\ndTEaHDqAMYajJVVkJEYHOitKKdUmGhw6QM6JairrGhmQpOslKaWCkwaHdlZcWcfv37OW5h7pstKq\nUkoFEx1j2c7+629ryDlRDcD5I7r+XAyllPJFaw7tqLym3hkYlFIqmGlwaEfbc8qcx+P7xwcwJ0op\ndXq0Wamd1NQ3cv0z6wD41/+cw6QMXWBPKRW8tObQTu5Zus15PCkjgbBQ/aNVSgUv/QRrB18dLuHt\nzdY6gmvvvkQDg1Iq6OmnWDt4a1PzVhXpCTrxTSkV/DQ4nKb3t+by6vrDADw3f1qAc6OUUu1Dg8Np\nenFtNgAzhiYxc0xaQPOilFLtRYPDaXhl3SE2HCzmvOEpPHbDlEBnRyml2o0Gh9Pwm39bexxdMDKF\nlNjIAOdGKaXajwaHNjLGkNgrnPBQ4YazBgU6O0op1a50ElwbZRdVUVJVzwPfnKDbgCqluh2tObTR\n8dIaAAYn67LcSqnuR4NDG5VU1QGQGBMR4JwopVT70+DQRp/vKwQgSYODUqob0uDQRgcKKggLER2l\npJTqljQ4tFFxZR2Xjk0jNEQCnRWllGp3GhzaqLiyTpuUlFLdlgaHNiitqqe4qo6+8VGBzopSSnWI\nVoODiDwvIvkist0lLUlEVojIPvt3osu5u0UkS0T2iMhsl/SpIrLNPrdYRMROjxSRN+309SIyuH2L\n2P42ZhdjDJw5JCnQWVFKqQ7hT83hBWCOR9pdwEpjzAhgpf0aERkLzAPG2fc8LiKh9j1PADcDI+wf\nxzMXACXGmOHAo8BDbS1MZ9ly9AShIcJE3e1NKdVNtRocjDGfAcUeyVcBL9rHLwJXu6S/YYypNcYc\nBLKA6SKSDsQbY9YZYwzwksc9jme9Bcx01Cq6qq+PnGBkWhy9InRmtFKqe2prn0OaMSbXPj4OONaq\n7g8ccbnuqJ3W3z72THe7xxjTAJQCyW3MV4crrKjl832FTB6gtQalVPd12h3Sdk3AtENeWiUit4hI\npohkFhQUdMZbelm1Kx+A6drfoJTqxtoaHPLspiLs3/l2eg4wwOW6DDstxz72THe7R0TCgASgyNeb\nGmOeNsZMM8ZMS01NbWPWT8+RkipCBK6Y2C8g76+UUp2hrcFhGTDfPp4PvOOSPs8egTQEq+N5g90E\nVSYiM+z+hBs97nE86xpglV0b6ZIOFlbSPzGa8FAdBayU6r5a7VEVkdeBi4AUETkK/BZYBCwRkQXA\nIeA6AGPMDhFZAuwEGoDbjDGN9qNuxRr5FA0st38AngNeFpEsrI7vee1Ssg6yK7eM0X3jA50NpZTq\nUK0GB2PM9S2cmtnC9QuBhT7SM4HxPtJrgGtby0dXUFPfyMHCSuZqk5JSqpvTtpFTsDevnCYDY9Pj\nAp0VpZTqUBocTsHmQyUAjEnXZiWlVPemweEUrMkqYkhKDAOTdPc3pVT3psHhFOSV1TAouRddfAK3\nUkqdNg0OfiqurGNbTqlu7qOU6hE0OPhp57EyAIakxAQ4J0op1fE0OPipqLIWgNnj0lq5Uimlgp8G\nBz8VV9YBkBSjzUpKqe5Pg4OfiirqCBFIiA4PdFaUUqrDaXDwU2FFLUkxkYSG6EglpVT3p8HBT4UV\ndaTERgQ6G0op1Sk0OPipsKKW1Djtb1BK9QwaHPxUWFFLcozWHJRSPYMGBz8VVdTpBDilVI+hwcEP\n5TX1VNc3arOSUqrH0ODgh7yyGgD6JkQFOCdKKdU5NDj44XipNTu6b7wGB6VUz6DBwQ+5pdWA1hyU\nUj2HBgc/OJqV0rTmoJTqITQ4tGJ/QQV/+ngvUeEhRIWHBjo7SinVKTQ4tOLltYcAdBirUqpH0eDQ\nivBQay2lF74/PcA5UUqpzqPBoRXHTtQwNCWG4X1iA50VpZTqNBocTuIv/9nL+9tyqW1oCnRWlFKq\nU51WcBCRbBHZJiJfi0imnZYkIitEZJ/9O9Hl+rtFJEtE9ojIbJf0qfZzskRksYh0iXWx1+wrBNBa\ng1Kqx2mPmsPFxpjJxphp9uu7gJXGmBHASvs1IjIWmAeMA+YAj4uIY/jPE8DNwAj7Z0475Ou0ldXU\nMyY9nsXzzgh0VpRSqlN1RLPSVcCL9vGLwNUu6W8YY2qNMQeBLGC6iKQD8caYdcYYA7zkck9AlVTV\nMykjgYReuvubUqpnOd3gYID/iMgmEbnFTkszxuTax8eBNPu4P3DE5d6jdlp/+9gzPaBq6hspqawj\nUZfpVkr1QGGnef95xpgcEekDrBCR3a4njTFGRMxpvoeTHYBuARg4cGB7PdanLUdO0NBkmDowsfWL\nlVKqmzmtmoMxJsf+nQ8sBaYDeXZTEfbvfPvyHGCAy+0ZdlqOfeyZ7uv9njbGTDPGTEtNTT2drLdq\nf0ElAKPT4zr0fZRSqitqc3AQkRgRiXMcA98AtgPLgPn2ZfOBd+zjZcA8EYkUkSFYHc8b7CaoMhGZ\nYY9SutHlnoA5WFhBZFgI/RKiA50VpZTqdKfTrJQGLLVHnYYBrxljPhSRjcASEVkAHAKuAzDG7BCR\nJcBOoAG4zRjTaD/rVuAFIBpYbv8E1IGCSoakxBAS0iVG1SqlVKdqc3AwxhwAJvlILwJmtnDPQmCh\nj/RMYHxb89IR9uSVM3lA70BnQymlAkJnSPvwyZ58jpZUMylDg4NSqmfS4ODDv7+y+sOvPiPgI2qV\nUiogNDh4MMbwztfHmDE0idQ4XaZbKdUzaXDwsCevHNBd35RSPZsGBw8bs0sA+NEFwwKcE6WUChwN\nDh4+3J5LRmI0I9N0JValVM+lwcFFVV0D246Wcs6wZMJC9Y9GKdVz6Segi0dX7KWspoGrJ+soJaVU\nz6bBwcXqvQVcNCqVc4anBDorSikVUBocbEdLqsjKr2Bcv/hAZ0UppQJOg4Ptva25NBm4btqA1i9W\nSqluToODbfm2XCZmJDAoOSbQWVFKqYDT4AA0NRn25JVz5uCkQGdFKaW6BA0OwIbsYmrqmxibrv0N\nSikFGhwAeG39YWIiQrl0XFrrFyulVA+gwQHILqpk6uAk4qPCA50VpZTqEnp8cDDGcLCwkkFJvQKd\nFaWU6jJ6fHA4UVVPeU0Dg5I1OCillEOPDw778isAGJqqQ1iVUsqhxweHLUdOADC+X0KAc6KUUl1H\njw8On+zJZ2RaLH10cx+llHLq0cGhvKaeDQeLuXh0n0BnRSmlupQeHRzW7CukoclwySgNDkop5apH\nB4dP9uQTFxXGlEGJgc6KUkp1KV0mOIjIHBHZIyJZInJXR79fTX0jq3bnc8HIVMJ11zellHLTJT4V\nRSQU+DtwGTAWuF5Exnbkez7y8R4KK+p0iW6llPKhSwQHYDqQZYw5YIypA94AruqoN3t1/SGe+fwg\nV0/ux4UjUzvqbZRSKmh1leDQHzji8vqondbu3t1yjHuXbuesIUk8dM3EjngLpZQKel0lOPhFRG4R\nkUwRySwoKGjTM5JjI7h0bBovLZhOZFhoO+dQKaW6h7BAZ8CWA7g2/mfYaW6MMU8DTwNMmzbNtOWN\nzhmWwjnDUtpyq1JK9RhdpeawERghIkNEJAKYBywLcJ6UUqrH6hI1B2NMg4j8BPgICAWeN8bsCHC2\nlFKqx+oSwQHAGPMB8EGg86GUUqrrNCsppZTqQjQ4KKWU8qLBQSmllBcNDkoppbxocFBKKeVFjGnT\nXLKAE5EC4FAbb08BCtsxO12Flit4dMcygZYrGAwyxrS6qFzQBofTISKZxphpgc5He9NyBY/uWCbQ\ncnUn2qyklFLKiwYHpZRSXnpqcHg60BnoIFqu4NEdywRarm6jR/Y5KKWUOrmeWnNQSil1Ej0qOIjI\nHBHZIyJZInJXoPPji4gMEJFPRGSniOwQkZ/Z6UkiskJE9tm/E13uudsu0x4Rme2SPlVEttnnFouI\n2OmRIvKmnb5eRAZ3UtlCReQrEXmvG5Wpt4i8JSK7RWSXiJzdTcr1c/vf33YReV1EooKxXCLyvIjk\ni8h2l7ROKYeIzLffY5+IzO+I8nUoY0yP+MFaCnw/MBSIALYAYwOdLx/5TAem2MdxwF5gLPBH4C47\n/S7gIft4rF2WSGCIXcZQ+9wGYAYgwHLgMjv9VuBJ+3ge8GYnle0XwGvAe/br7lCmF4Ef2scRQO9g\nLxfWFr0HgWj79RLgpmAsF3ABMAXY7pLW4eUAkoAD9u9E+zixM/5NttufXaAz0GkFhbOBj1xe3w3c\nHeh8+ZHvd4BLgT1Aup2WDuzxVQ6sPTHOtq/Z7ZJ+PfCU6zX2cRjW5B7p4HJkACuBS2gODsFepgSs\nD1HxSA/2cjn2dE+y3/M94BvBWi5gMO7BocPL4XqNfe4p4PqO/Htr75+e1Kzk+AfvcNRO67LsKuoZ\nwHogzRiTa586DqTZxy2Vq7997Jnudo8xpgEoBZLbvQDu/gL8CmhySQv2Mg0BCoB/2M1lz4pIDEFe\nLmNMDvAn4DCQC5QaYz4myMvlojPKEXSfN556UnAIKiISC/wLuMMYU+Z6zlhfRYJmmJmIXAHkG2M2\ntXRNsJXJFobVZPGEMeYMoBKrmcIpGMtlt8FfhRX8+gExIvJd12uCsVy+dJdydISeFBxygAEurzPs\ntC5HRMKxAsOrxpi37eQ8EUm3z6cD+XZ6S+XKsY89093uEZEwrOaRovYvidO5wJUikg28AVwiIq8Q\n3GUC69vgUWPMevv1W1jBItjLNQs4aIwpMMbUA28D5xD85XLojHIEzedNS3pScNgIjBCRISISgdV5\ntCzAefJij4J4DthljPmzy6llgGPEw3ysvghH+jx71MQQYASwwa42l4nIDPuZN3rc43jWNcAq+xtU\nhzDG3G2MyTDGDMb6c19ljPluMJfJLtdx4IiIjLKTZgI7g71cWM1JM0Skl52fmcCublAuh84ox0fA\nN0Qk0a6JfcNOCx6B7vTozB/gcqzRP/uBewOdnxbyeB5WNXcr8LX9czlWO+ZKYB/wHyDJ5Z577TLt\nwR5FYadPA7bb5x6jedJjFPBPIAtrFMbQTizfRTR3SAd9mYDJQKb99/VvrJEp3aFc9wO77Ty9jDWC\nJ+jKBbyO1W9Sj1XTW9BZ5QB+YKdnAd/vrP9j7fWjM6SVUkp56UnNSkoppfykwUEppZQXDQ5KKaW8\naHBQSinlRYODUkopLxoclFJKedHgoJRSyosGB6WUUl7+P1gkqRGCRFg9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f381ebf0a10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 76%|███████▌  | 113310/150000 [19:46:02<6:43:27,  1.52it/s] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=113310\tloss=0.349\treward/tick=7.746\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 76%|███████▌  | 113320/150000 [19:46:08<6:09:25,  1.65it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=113320\tloss=0.368\treward/tick=7.778\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 76%|███████▌  | 113330/150000 [19:46:14<6:19:58,  1.61it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=113330\tloss=0.371\treward/tick=7.789\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 76%|███████▌  | 113331/150000 [19:46:14<6:10:44,  1.65it/s]"
     ]
    }
   ],
   "source": [
    "from tqdm import trange\n",
    "from IPython.display import clear_output\n",
    "\n",
    "#the algorithm almost converges by 15k iterations, 50k is for full convergence\n",
    "for i in trange(150000):    \n",
    "    \n",
    "    #play\n",
    "    pool.update(SEQ_LENGTH)\n",
    "\n",
    "    #train\n",
    "    loss = 0.95*loss + 0.05*train_step()\n",
    "    \n",
    "    \n",
    "    if epoch_counter%10==0:\n",
    "        #average reward per game tick in current experience replay pool\n",
    "        reward_per_tick = 0.95*reward_per_tick + 0.05*pool.experience_replay.rewards.get_value().mean()\n",
    "        print(\"iter=%i\\tloss=%.3f\\treward/tick=%.3f\"%(epoch_counter,\n",
    "                                                      loss,\n",
    "                                                      reward_per_tick))\n",
    "        \n",
    "    ##record current learning progress and show learning curves\n",
    "    if epoch_counter%100 ==0:\n",
    "        reward = 0.95*reward + 0.05*np.mean(pool.evaluate(record_video=False))\n",
    "        rewards[epoch_counter] = reward\n",
    "        \n",
    "        clear_output(True)\n",
    "        plt.plot(*zip(*sorted(rewards.items(),key=lambda (t,r):t)))\n",
    "        plt.show()\n",
    "        \n",
    "\n",
    "    \n",
    "    epoch_counter  +=1\n",
    "\n",
    "    \n",
    "# Time to drink some coffee!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating results\n",
    " * Here we plot learning curves and sample testimonials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "plt.plot(*zip(*sorted(rewards.items(),key=lambda k:k[0])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from agentnet.utils.persistence import save\n",
    "save(action_layer,\"kung_fu.pcl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "###LOAD FROM HERE\n",
    "from agentnet.utils.persistence import load\n",
    "load(action_layer,\"kung_fu.pcl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-03-29 01:21:33,678] Making new env: KungFuMaster-v0\n",
      "[2017-03-29 01:21:33,717] Clearing 4 monitor files from previous run (because force=True was provided)\n",
      "[2017-03-29 01:21:33,734] Starting new video recorder writing to /home/justheuristic/agentnet/examples/records/openaigym.video.1.17080.video000000.mp4\n",
      "[2017-03-29 01:21:49,589] Starting new video recorder writing to /home/justheuristic/agentnet/examples/records/openaigym.video.1.17080.video000001.mp4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 3493 timesteps with reward=25700.0\n",
      "Episode finished after 4170 timesteps with reward=31900.0\n",
      "Episode finished after 5181 timesteps with reward=41700.0\n",
      "Episode finished after 4192 timesteps with reward=30200.0\n",
      "Episode finished after 3440 timesteps with reward=26100.0\n",
      "Episode finished after 2991 timesteps with reward=23500.0\n",
      "Episode finished after 3702 timesteps with reward=29600.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-03-29 01:23:28,925] Starting new video recorder writing to /home/justheuristic/agentnet/examples/records/openaigym.video.1.17080.video000008.mp4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 3242 timesteps with reward=24000.0\n",
      "Episode finished after 2358 timesteps with reward=18800.0\n",
      "Episode finished after 3979 timesteps with reward=29700.0\n",
      "Episode finished after 5510 timesteps with reward=39800.0\n",
      "Episode finished after 5083 timesteps with reward=43400.0\n",
      "Episode finished after 4671 timesteps with reward=42600.0\n",
      "Episode finished after 2915 timesteps with reward=26100.0\n",
      "Episode finished after 4510 timesteps with reward=36600.0\n",
      "Episode finished after 3877 timesteps with reward=36100.0\n",
      "Episode finished after 4209 timesteps with reward=35700.0\n",
      "Episode finished after 3996 timesteps with reward=31200.0\n",
      "Episode finished after 3774 timesteps with reward=29400.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-03-29 01:26:12,525] Finished writing results. You can upload them to the scoreboard via gym.upload('/home/justheuristic/agentnet/examples/records')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 4939 timesteps with reward=41600.0\n",
      "mean session score=32185.000000.5\n"
     ]
    }
   ],
   "source": [
    "rw = pool.evaluate(n_games=20,save_path=\"./records\",record_video=True)\n",
    "print(\"mean session score=%f.5\"%np.mean(rw))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<video width=\"640\" height=\"480\" controls>\n",
       "  <source src=\"./records/openaigym.video.1.17080.video000000.mp4\" type=\"video/mp4\">\n",
       "</video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#show video\n",
    "from IPython.display import HTML\n",
    "import os\n",
    "\n",
    "video_names = list(filter(lambda s:s.endswith(\".mp4\"),os.listdir(\"./records/\")))\n",
    "\n",
    "HTML(\"\"\"\n",
    "<video width=\"640\" height=\"480\" controls>\n",
    "  <source src=\"{}\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\".format(\"./records/\"+video_names[-1])) #this may or may not be _last_ video. Try other indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## How to enhance\n",
    "* More parallel agents\n",
    "* Different constructs for recurrent memory\n",
    "* Try PGQ-like algorithms\n",
    "* Maybe tune parameters for regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}