ddpg-pendulum/DDPG.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Deterministic Policy Gradients (DDPG)\n",
    "---\n",
    "In this notebook, we train DDPG with OpenAI Gym's Pendulum-v0 environment.\n",
    "\n",
    "### 1. Import the Necessary Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "import random\n",
    "import torch\n",
    "import numpy as np\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "from ddpg_agent import Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Instantiate the Environment and Agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('Pendulum-v0')\n",
    "env.seed(2)\n",
    "agent = Agent(state_size=3, action_size=1, random_seed=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Train the Agent with DDPG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 100\tAverage Score: -1241.72\n",
      "Episode 200\tAverage Score: -702.036\n",
      "Episode 300\tAverage Score: -452.37\n",
      "Episode 400\tAverage Score: -378.96\n",
      "Episode 500\tAverage Score: -346.99\n",
      "Episode 600\tAverage Score: -341.02\n",
      "Episode 700\tAverage Score: -342.22\n",
      "Episode 800\tAverage Score: -356.71\n",
      "Episode 900\tAverage Score: -369.13\n",
      "Episode 1000\tAverage Score: -526.35\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAAEKCAYAAADenhiQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnXe8VMXZx3/PrfQO0qWLIIJypagIigVsxC4aNTY00egb0zAaTYwmakx8YzRGNEZNjCWxvpaoKHZFQZAqSC8iXHq9l1ue9489Z/fs2VNmTtndu/f5fj4XdufMmZlTdp55nmfmGWJmCIIgCEKUFOW6AYIgCELhIcJFEARBiBwRLoIgCELkiHARBEEQIkeEiyAIghA5IlwEQRCEyBHhIgiCIESOCBdBEAQhckS4CIIgCJFTkusG5IoOHTpwr169ct0MQRCEBsXs2bM3M3NHv3yNVrj06tULs2bNynUzBEEQGhREtFoln5jFBEEQhMgR4SIIgiBEjggXQRAEIXJEuAiCIAiRI8JFEARBiJyCES5ENIGIlhDRMiKamuv2CIIgNGYKQrgQUTGABwBMBDAIwGQiGpTbVgmCIDReCkK4ABgBYBkzr2Dm/QCeBjApx23KGdv37sc7X20MVcYny7dg6579aWnMjI+XbUZ9vfPW2MyML9duT35fu3Vv2ndrvhlfbXItx0pdPePFOeuV8s5btx2bdlb55gOAVZv3YPbqrcnv1bV1+GT5Ft+2/N+X32DGkk2Yty7zumau2IJ3Lcc+XrYZ3+6owoL1O7Bq8560vCs378Hs1dtc66qqqcOcNe7HTT78ejNWVO5OXsuuqhq8sfDbtDwvf/kN3l7s/D58u6MKM5ZsAgC8tWgj3vlqI1ba2mpl9ZY9yXbV1tXjpbnr4bZV+tcbd+FXLy/E4g07Pa/ho2WbsXbr3uT3ffvr8OIc93LfW1qJHXtr0uqxt3n26q1Yv32fZ71hWbl5Dz5evhlfrNmGhd/siLWuhkihLKLsBmCt5fs6ACPtmYhoCoApANCzZ8/stCwHXPnELHy+ahvm3nIC2jQr0z6fmTH54U8xsHNL/Pd/jkmmv7VoI6b8YzZuOXUQLju6d8Z5//x0NX750kI8dukRGHdQJ4y5ewYAYNWdp6Tle2HOetzw7Jf4zXcOwUWjDvRsy98/WonbX12M/XX1OLeih2fe0+//CC3LSzD/1yf5XuO4e95Na9vvXvsKj328Cq9edzQGd23teM789Tvww6fmJL/br+u8aZ+mHbvgkZlo37wMWwwhbc1/rK1+O794YT6e/2I9Pp56HLq2aep6Hd/928zk5y9vPRE//feXeHPRRrz7k3Ho1aE5FqzfgeuMNjvVdfr9H2LTrmq8/9NjceUTqUXFbu0a+/tUu//63nLc8+ZSFBHhtKFdM/Je8cQsrN6yF499vMq1PAC48JGZKCkiLPvtyQCA37y6CP+auQadWzfBqD7t0/Lu2FuDSx79DCN6t8OzV40GAJxw7/sZbT7rwU88ryMKzGdoEmddDZFC0VyUYOZpzFzBzBUdO/pGL2iwmKO4mjr/0b4TppLw1be70tK/MUaCq7c4j2wXGSNUvxGjWc4GhZHlhh0JTcQ6UvViV3WtUj47yyt3AwC27N7vmmd/bb12uVv2uJfnxaJvEvdyxz7367Zrc/tr65PPvtpoa+Xuas96Nu1KHN9bo3/f1m+v8mzj6i17HdOdqLVci6l97qrKbFN1XR0AYEWlu3Yl5AeFIlzWA7AOa7sbaUIA3MwRyeMu6bWGMCspIp/yE/+TdzYACVMUABT7lBkWs/w6D/Nbvc99iRKzPbUeA4Sa+nRhRwTUsXm/4mubSX1Mz4bI41lk7xEIISkU4fI5gP5E1JuIygCcD+DlHLepweL2+yUfaWB2BiVF3q+VWT7Bv1NKllkcr3AxBWJNnbt2kkXZkmxPbb17e5w0U7PDL3J4Vn6DBl2SgkxllKCBKau82htxlUIMFITPhZlriehaAG8AKAbwKDMvzHGzGixBR+g1ioLALF5lwFvr0Vla8dI4VDAFolc5Op1z2I5cRZOqsZnpCKkO32kgwOzeKQdpbn2yLv1zvTCftVOTRHFpOBSEcAEAZn4NwGu5bkc+wQF/im4djV+HWWeMsv3MJPUadjGzTD9Tm5fGoUKxIRBrvISLRnlhlQRT2NV6CZc6u1mMYCo6TkKpnhlFLtpiIOESk1nMFC5egxxRXPKfQjGLCWmE++n5dTRupSv7XHzKsWL2n34dWFjhUprUFNzL0dHo6hTzuglsU/vz1Fwcjpn5ndrq1SbdgQgzJyd+RO9zSfzvdO3ZNE0K4RDhImQQVONR9bmYPYSfqStRpqG5+Jjags6MMyk22uxVjk7HpiqI3OorVvABeZnFkh2zpXivJul22jV1nKxL5TnqYF67V5vE55L/iHARMvBzX7jOFjPNJD6CwCxfpYNQ9bnUhtVcFDQFHc1FNaub8ChR8bnYzmWkTFVO53mVpeuzqqmrR11deLOYk+bmZRYLOvARso8IFyEDN1ON6mwxv9lDZgehZhZT04b2h/W5JKf+eswW0yhPtbN2Ey7FCj4X+zUzp7QJp47ZSzh61eNETV19srwwVjGnar3MYsk8MLUbETb5iggXIYOgE6/MjtJPI0nOFlPolVLrXPzqDtfJlBb7d+ZeHZn9mKqW47YwU0WTsq+BYUt+Z4e+ezt0NZf9FuESpn93uk/m4MSpXHuarlAUsocIl0Im6O8u4HlmB+XX2ej0B2E1AFVUFi3q+CxUr9FN41LyuWRoLimzmKPm4iWoPCYyONfNKUEWQro4PV+l2WJktiPccxfiQ4RLARLW2Rl0ncu2vfuVzk+axVzaae0Ea5OdpXfdVg3AybTl1rH+89PVqKmrT04YsHf2yyt349lZa/Hpii2ubaiqqcN0W2BIp/qmPjcPW2zhWOwaV1094+3FG/HKvA1pxzfs2JfsiHdX12LGV5uSIV5MGGxx6Ge20/pcHpixDB8v25z8bheqD7673DXMDwA8P3sdZiypTLZ5Z1VNwixXz/jliwvw9cb00EFrt+7FPW8swbptezHt/eXYsbcGX67djjlrUgFAq2rqsL+2Pvl+/O3DldhZVYP12/fhd68vxlOfrcHCb1JBMNds2Zv23Ct3VWNF5e6MgKsmG3bsw64q/zBCH3xdiU27EiFomBkPzFiGZZt2JesEgE9XZAY5ra9nzF69DTNXbMFX3+7Ef2avw7Oz1qZpteu27cUXa7ahclc15qzZhgXrCzfgJTVWm2VFRQXPmjXLP2MD5Ig7pqNyVzU6tixHi/ISXDCiJ+av34Hm5cWo3FWNaRdVJE1Sj364Ere9sghXjumNiUO64MF3l+O4gZ1w4/PzAQDjDuqIe84Zik+Wb0kL2njraYPwyAcrsX77PhzavTXaNivDe0sTnc11x/XDyi178X9ffgMAaFlegopebfH3S0cAAA791RvYWVWLqRMHYvKInrjssc/xx3OH4sD2zTFv3Xacfv9HAID+nVrg602JmF/3X3AYOrVsgsc/WYXFG3aieVkJjuzXHnNWb8dnq1LRjU26tWmKMw/vhtfmb8BVx/TFz5+fh++P7Yu/vLs8I2/HluWo3JXe6c/71Ym4/51lmPb+imTanWcOwVTjvpj85MQBePSjVRkd2ug+7fGJQwd0wqADUFVThw++TnTs/7h8BK57ag6evGIUfvXyQsdrcYMoXWP67KbxGHPXDFTX1uOpK0fhlXnf4Is125NRiT+7aTw6tWyCMXe/g7Vb/eO6TR7RE787cwjq6hk/fOoLvDb/W8/83zuyF56cuVrJRGkN6BmGLq2bJOPPjejVLuP+rbrzFDw5czVuemFBMq20mFBTx7jrrCE474hEANsde2sw9LY3cc2xffHAjMQ78tVvJuDLtduTAUn/dP4wXP/0XDx+2Qhc8uhnym28+ZSDccWYPrj9lUV45MOVGce7t22KD39+nN6F5xAims3MFX75CmYRpZBJ5a5qVO6qxh2vLU5L/2j5ZozpnwjcedsriwAAD3+wEq/N/xbrt+/DW4tSo/B3l1TiN68sShthAsCv/29R8vO8demjr/veWZb2fVd1LWYsqURdPaO4iLDTCEhIAN5c+C1mr96GP739Nf547rCkYAGQFCxAQnM596FP0sqd7zHqW799H/5stOPB95aDGY6CBUCGYAESI3erYAGcAzTe8+ZSxzKdBAuAtHsLAK/N34Bte2vwp7eXagkWwMFMx+kmpSdnrnHMryJYgNQ08OWVu30FCwA89vEqpXKB4AE97ZiCBYDr/bMKFiClDf78uflJ4bJic+JdMwULAHy2cit2WwKh/su4n3atzI9nZ63FFWP6OAoWAFi3Lf15zFyxBdv27seEQ7po1ZNviHBphOg6b6PymdbU1aO4qDj5nSjlW/DbryWMhl1dU6d9jpNpL07ncdgJCUDCVWbOkXCaKKBr7jQFlZcfqpCpZ067Z6mwRXp2Z938pqbU0EP4i89F8CUq06nd+VpElIqh5TsJIHgbqgKEynea1BA2fpljNUaRQcL5O5VldmRVDgJVt/2e0YkbAczpAyvzHdRd1xN3RO98RYRLI8Stq/AaYEWxItppdJ404/h0YJqTmdKISnPJe+ECTj6nqtrMa9aVz2afqDuTrKHgF6aorp7TBlZ1Ily0EOHSGAnQR0ahvDgFWlSJ/guE01yCLLB0ak4swsV4GGEXgQKJZ2Tez337M8vTbb/KlOCGjEqAVes9CxpLzW/xcaEiwqUACfoqu4ZjD9ySdDKEC1IdmN9aiTD9W02AMCW6gR+DYnZYUazXSPhc3M1i+j6XxP+F6nOxai5OV1hvN4spRqCwE/NWRHmLCJdGSJD4THGYxYo0HPpRjJ51BpxO1QXVXFqUu8+bidbnwslR8j5H4aJXXqH7XFQ0F+t7Z94HlcgSOvUUKiJcGiFu/bSraZ3jNIslPvtpBVH0bzqzdqL0uTQvL3Y9FrVZzOzHnPxMgWeLNQLh4vRm1DOnDXqCDnDELCY0euK2rdtH59bfXJw+F5NcCZdmZR4z/iPUXIDUNTprLsHMYoWruXh3fxlmMTMoqO6sO+2WFQYiXBohrpqL2wGKxizmNAI2B+y+IWOybBaL0qFf5hF109RY7KFcgmDVXKpqInDoFxW25uK7qZ2LWaxaU8ss1AkRfuSdcCGi3xPRV0Q0j4heIKI2RnovItpHRHONv79azhlORPOJaBkR3UeNVQ9VxO1Vd/0NRLiI0l6fVxRfK1H0bzqvhdPoNGgnW1biIVwMoWLf+CsIianI7pqLbh9n3q6we+XkK7o+F/Px605tL1TNz4+8Ey4A3gJwCDMfCmApgBstx5Yz8zDj72pL+oMArgTQ3/ibkLXW5iFBRWvcI6yd+2rw2vwNafWlTA3e50bRNp3b8vTnazPbELCTKPWYLmRqLFH5XL7dmQiHYsZ5s1JXz44BF90w90yJom35iJ9w+WZ7FW59eWHyu/kO/u/0r7XqaaSyJf+ECzO/ycxmQJ9PAXT3yk9EXQC0YuZPOWE7eQLAd2JuZt7y+aqt2LgzM1aWCpt3O8d7enX+Bqw2osGG4aYXFuAHT36R/L6vpi753c+h/+Lcb0LXH1ZAPTMrU+Co0Kt9c9dj+yMULuPueTc5SnaKlzbpgY9wvhFaRIV12/aivp4jCU2TK259aYHrMdMsVlVThy/Xbs84/vs3lqRpe+Zna7wxFVTeu2c/X5th+n1vaSVufH6+tuaTL5pmvscWuwzAM5bvvYloDoCdAG5m5g8AdAOwzpJnnZHWKLnvbf9RVa4iYZujahNr+HQAWFG5G244/fh1qcnREPKgzi3RummpY+BLU6jE9UguGNkzGXBRl1fmbUDfji3QtU2TiFvlzslDOisFyVTl8U9Wux4rKiIMu+1NbN/rH4Yf8J7C36NdU9eAoN3aNPUt+2fPzcPOqhrc/moqyKwZeXnbnv3460XDldq4dutejLl7Bn5/9qE4p6KH0jlxkRPNhYimE9ECh79Jljw3AagF8KSRtAFAT2Y+DMANAP5FRK00651CRLOIaFZlZabZoBDwc1IC0S2KLC4iXHpUr+R3XXOcdYZUEQGrPPYPiYJcjeiKiwjv//RYDOzcMuNYVLPE3PjtGUPw1+8eHvj8j5dvzqpZRzfIY7i6oCxYAG/TrVe72zYvUyr/vwuchep/F6oL22XGAM3cDyiX5ERzYebjvY4T0fcAnApgvGHqAjNXA6g2Ps8mouUABgBYj3TTWXcjzaneaQCmAYn9XMJdRX5S4rcfcIRYV9jD+Kyzit3q4M/GHIxc2b6JCK2blaJnu2b46tv0cO3VDjHAoqZVk9LA59bWc1ZnO2VTuESJ1y1S9dUVmm8r73wuRDQBwM8AnM7Mey3pHYmo2PjcBwnH/Qpm3gBgJxGNMmaJXQzgpRw0PS/wch6bRNVXEKU7yXXDYqQJlwjblW+YyqTT7Ylbc0nUG7zDrqvnrArlfF7NHtScrOozyca7kE3y0edyP4ByAG8ZP4pPjZlhxwC4jYhqANQDuJqZzd2BfgDgMQBNAbxu/DVK/BaGJYiut7CGwigqAqAxEE83i1EBC5fEPSKH+WpRrG/xI0yHXVvHWfXR5bPm4nUXvPwxqtq8CJeYYeZ+LunPAXjO5dgsAIfE2a6GQmkWR35k6y51NZf9lllIRFGKvPwi15pLmFfCHgIlbvJYcQk8+FG9f9kYaGSTvDOLCeEoyaJZDIQ0u5huQL+aDM2lMMWLaZZqiGax2gI2i+m+bl7aiVdZqjO5g/pcKndVZ3UAoIoIlwJDxaEfpWyxO/R12J/m0C9kzcXDLJYFJ26YDrsuyw79bAbX0N3mwKv/jsKhr7vyf/WWPdi4swpH3DEdf35nmda52UCES4GRVbOY3aGvq7nYZosVqOKSMvU0QLNYbX19qOeiW3cWJztqLw4Neh+UHfoawu7txRsx9vfv4h/GOp63Fke3NigqRLgUGNmcigyE01ysZrHEmYUpXVKaS27rD0JdXTjNRbduXb9dGHR9HEHvQxwO/S/WbAMALN2YmNpuryIffkkiXAqMbPpcCJTmR9CVa9aRWhEV8FTkItPnkhvxEka4hPW56Nadz2YxLw3Ey1+oahbTuc97qhMmNNNaYFafT/MhRLgUGKUKU5GD7ETpBFF6Z6A9W8wyUttZVYsNO6o8cjdckrPFclV/iF/5pl3VuOu/XwU+X1dWZNOhrxsjzCvCQ6lX5Ou6eqzdGj42n5W9+xNtN7ezrtxdjaUbdzlub50rRLgUGNn8cdomi2mPOndWpX7cs1dvw22vLIqmYXlGkcdsMTtjB3RM+/6Hc4aGrj+bpiY72maxLL6/2gEhPfJfdUxffHdUT/zp/GHo0zE9UOkHX2/GmLtnBGqjycPvr0jTjvbsTwiRGUsSYawqd1XjxHvfx9X//CLtvGWbdmlraFEhwqXAyOYK/T3769I6zDxe/4bfnjEkZ3WThuZy/hHpwQaPHdgpgvr9a27TrBTHHtTRN59+3fHmzyZ19YwRvdqhc6vMQJ4tmpTg9u8MwaRh6jFzO7QoV857x2uLMfW5+eg19VWs3boXby3c6HvON9v34fg/vo87Xl2MTbuqsGpzvLH77IhwKRAOu+1NnPmXj5Q6kihdG2Ec+tnioYuGo1tb/8i0QThh0AHKeVWejX1CRhQDeRVtoIgoluenW2JUbfDaoC0otfXsamJMu8UKP7DubZvi92cfqlW/ueXDGX/5SGlm2dY9iS00Zq7cihF3vI1x97yrVV9YRLgUCNv21uCLNeHD0uti/U3l6+rqI/u2j61tOuWqZLVPyIjCwa3SRkI8WoNu+6My4cW1qNBNUAdpt+6iY5OWioFIcz3WE+EihCIttliu32YXKKZROaCmFZCGXcw+ISMKoahy7Yk25oPmEk29cS38LCJy7LSDDAKCClJdv1SuIl+IcCkwVN7XuF62PJUtsY3KAb1OxWmFvh17xxGFUFQZISdm/oWuyqFgvexBR/N24oqG4taxB2l20Fl8qr9flfctTkS4CKFoCD4XIve2he3LVEafyQX6KppLcQzCRaGIuMyGufK5xIXb8w4yyy3otarIlnyI0yfCpcDI9milIcwWI7ibxcJOfY26U7Y79KO4p2oCMJ43J1cLR+PCTbOyvl8q3TpRcL+Q6lm5vvUiXAoMNbNYhPVZPufrqDOhuTgfC9tmVZMToOjQj8EsptLBF8VkFsvTVyIwbo87yHXq7NpqRVcryZUSI8KlwFB5x6NaoQ+kd375PEp1a1vYzlvHKauS1d6ebE1FJopHd8nfNyIYrrPFAjwo3UWcJqqn5frnKMJFCIX1Bc7XqchemotdU9BFqeM2uliVztvu5M2WzyUfJjwA+R9fzt13lz3hojs4jHIwqYMIlwLAarvNulmsITj0PXwuYWcnac0Ws2V1Emz2dkZxS9XMYs5TbEPXHX2ROcV9tlgWhYvmaUs37g5UT1jyTrgQ0a+IaD0RzTX+TrYcu5GIlhHREiI6yZI+wUhbRkRTc9Py3FFTrxc7KC6fS57KFgAxzhZT+AUlfS62ukodTra3JwpTo4p2delRvcTnooCbGdTdp+ecTqDAa3HC/H73aAbrDEPeCReDe5l5mPH3GgAQ0SAA5wMYDGACgL8QUTERFQN4AMBEAIMATDbyNhqsmx5l26xg/fHkaz/itYYjmz4X+x1y3h7Bu7yhPdpo1JfAT7YM69EGlx7VOxafi078LCA6E06Q+xQGqwZsFRolHotZjhkQfSw3K07Pc8o/ZsVap5V8FS5OTALwNDNXM/NKAMsAjDD+ljHzCmbeD+BpI2+jwRoKfNWWaEN7+5HPTnwTgofm4tPzEgF3nHGIx/Hg61zKXDSXN/7nGNeympcVp5ehEEPL6dpPG9oV1xzbFwDQskmJbxlBefyyEaHO79ZGLybcJaMPBAAce1BHDD+wbai6HSFgUJdWGcnW16hO0UzdrKwEd2vGFwPUog/MXbsdW3ZXZ6R/tGyLdn1ByVfhci0RzSOiR4nIfEO6AVhrybPOSHNLbzR89e2u5OenPlvjm9/v1Ty4Syt8fcdEpbrTfjwhBc2NEweGOh8AurbOjFhLRK6rof00j0O7t8EFI3q6HteZJWTP6aS5FBGhbXP32FF2QaGyrbWTcOnauglG9m7v3cAIOMAhgrAOI3q38zzep0N6eHtT2BMotq0G/jT5MM/jVuFiv/fDbBpVkBaq7Hu0q6oWFzwy0/FYthZY5kS4ENF0Ilrg8DcJwIMA+gIYBmADgD9EWO8UIppFRLMqKyujKjbnzF2rF7DS7+UiqL/01pF72J9yXGs6vDQX1Wm6brPKdGZiZTr0nTQX7ynBGWUoOH3sbRzVpx1uOHFAZtk+5ehqEUHQ7vdsjbbunRPXLKkW5SUY0ctd6KULl/RjzWyap4rm+3/XHo0zDnMfL582tKtvGVa2763Ryh+UnAgXZj6emQ9x+HuJmTcycx0z1wN4GAmzFwCsB2Dd7KK7keaW7lTvNGauYOaKjh3jtXdmkyYRhxfX6eOjHBtGMdB00lC8piKr4jpLSEtzSc/rZNLya6u9M1LZv8fe9rOH90B5SaqTS26R6/MAfnHywb51RY3f1bmtC4rbWJspuJx9Lvb2ZURgUKhrSPfWuOGEzMFAskzNl1t3AlBQ8s4sRkRdLF/PALDA+PwygPOJqJyIegPoD+AzAJ8D6E9EvYmoDAmn/8vZbHOu0R2f+eVPOMDVXtj0RZSaDcmoN4o1HQ6aC1Host1+wGqml9Ro2q9Mv3ufYVpTiH5oL899BpM3+biOyd4mU9jH7Qr0mkVc6+FzKbG1T7WdXr413QWc1glAcRKfJy84dxPRMCT6wFUArgIAZl5IRM8CWASgFsA1zFwHAER0LYA3ABQDeJSZF+ai4blC25Tgk19n1lCUP+I4w8uHnhUWwcpse07nqcjed99eXWmJ/nUFvRXZmLyR8Wr6VOm2LiiuiAMmXqblNLOY7YEFjXrt9K6Y6GoutVna9jjvhAszX+Rx7A4AdzikvwbgtTjblc/EMQ5RfV2jnIocRVfgPuU4XLluP26l2WKWDi+9TGeHvleHk1FGgLjtbuX7XUo+ai52rAIlzpXpds3Feu+si5rtmq39u6q89jJ/6msujdQsJuijHcjO50en53OJrseJai8Px7JdLkr13kW5A6FXmUU+synsh4LEtErOqLKdquvfyAZ+71ccsdjCYjWL2d+uoHu4RKm57K8t4NliQm7x6091XlXrbzus2SQSzcUtPWThrj4XlRX6Lm1wnOml6dBXmS1mxz2yr09Hno3eQnOgZG+yKWzi2ubYxKt0q1nMHuIl04wX3ixWrPlgRHMRlIl82rpGnKlI7fAxjoxdNRfF84tdzBJhdqJ00nqKLJMPnIrO8LkozBZzqiMIuVgw61dlpnBJ/F/P0WrVduwar7Umayh9X+GiWJ+Xhuoc6cEdES6CMlHblrU0l4DnOZEZV0u/DN3Q+qqCeYfL2gDVdTKJ//3rSfhcEp9VAlt6jWhd26OZ7lZ3PmLe6zrmWH0uXu+N9Vitbdqv/X2J4pZqT0XO0mwxES4FgK4FQG2nPLUXNkpTiX2kGWVn5vX7u358fzxycYXn+TurnAP+6fyu7Vn9NBMnwWU/J4jmElQDyYU/Q7dK852JexW6XXC53VN/zSX8TdUXLqK5CIro/o78YhMFdeiHlQVFBJwypAuuG98f3do0xd1nqcVdOrxnKqSGu8/FvXE/OmEAjh90AEb3ae+axw0Ve3e75mWO6U7+EusU2oMdY1ilrmNo99bKAvgnJw5ICiuzL+rTsQUA4JRDjaVlvrPFopUuLctLcPnRvXF0vw6RlZkyi8UrXOzrEN3uTG2GcEn837S0OO17GMTnIsSG9uZBMTn0xx3UKe1Ym2buMbLcynrgwsNxwwkD8NHU43DW8O5K5z2mEBzR7UdsvXdPTRnler7bCmm/e/XEZSOSARTtt729g9AhApqWFePJK0bise85XJelwpeuPdqxzhblmSsMrj2uP47smxCeppDo1qYplt4+Eecf0cMo2vtqnGTLwM4tPc8x6WyJMda3Y3O8dt0YzP/1SfjlqYPwzytGJo+FFQmLiyrEAAAgAElEQVSmb8zeqTsx8xfjA9djlt63YyK2mXUAcWD7ZsnPVx3TN+15mPewuZGmI6/dIkzr+lz++t5yrfxBEeFSAOgO0p7+fA3O/MtHrsf1NsBK5f3+2L5px/52ibepKaOsgCaCNL+P8cXe6YUddbuvc/E+zyuseseWmZ2F2c6j+nVAawfhbL+Oa4/tl5HHzSRkJlsHumUlRco+IacJCE9PGYX7Jh+Gq47pk5Y+4yfj8PdLjwAAfH3HRLx87VHJY0SEQV0ztTInrhrbF71twSmt2N8Zc91PrYJf4YBWTfTNikaxdYbqcutpg/HG/xyDHu1SAuXfV4/Gk1eMxKo7T8HUiQNxcJfUu1hVkzgvFWMsvf4BB7RwrXrWzcc7pqtORzc11M9XbVPKHxYRLg2YRd/sxLpt+iH2F6zfiS/WuAe71Pm5Wd/rzHUqej/coP2/k+DIXP8QTri42bV1SrX3+X6h21XqO7Jfh7SO26S8pCgj2KGppQX1uTh1YmUlRTh9aFf8cHz/tPTeHZrjWEOTLS0uSjtXp/Z+nVpgxk/GabfR6uv43ZlDXPMHtZ6ZwqtZWTEOsg1kOrVsgqMspr7mFs1l7/66RFqZs+byuzP1Q/A7vZumlmrFaYuHOMm7FfqCOiff9wGAhD09CFE4PXUi+PqWFXiKbGZ7MmblhPxduS3wDDM9d8ABmSYlPyHo1AynZ7DkdvctE4LOFnOcYGDGTfM5Ny16tkfmsK+kqYlY/Qo9LVpFRn26FRht32+Ur7KfjjX+276axMSQZuXF1uKSBFkUaz9n1Z2n4M7Xv8LHy7d45osb0VwKgKA/SLcpiVoOfY+8+jN9NE9I1uOguSjGcFK9d7ozclRQmQ2WeVz/HJOkWSxg+BenIJmqARjTwwTF18mZzu20Dbs88gcdYJmai8pUcKvprZ8xicL0T6oGFfXCNdKDPZ+lrmzs6SKaSwNj65792LpnP/p1StlmH/9kdaCy3GaN6Pz4dVaT+xE8oGLm54xouWFnskUgXOwTL5wX6Ps41cPU7ydcjNKblRUnzTdWvEa+/u1W01zCYg4C0iMTu1cYtIut0dFcLA/68APb4rrx/dHecM7bWxbEfOt0hlMx1oXA9QwEmMWuhWguDYwT730fx//xvbS0zQ7bmarw5ExnoaT3fnt1OHoEXzmemWZ3PsflcwkzAnT2FXmfo6K5qGyp4MWNLvu2eGlavrdXVbvS7O4zQ+oYwsUycPK6p6qP766z0v02SeGiornYGtDeMusrCkHr9E44vVvW34R9/U0ciHBpYAQVJE789rWvHNN1XnhvzUWzQQFRMYu5tUU5/EsUmoutMredKL1Q9bk41m869F2OJwWFy3Fn8wulnetGlDHovCjW1FxUsZdhlq+muVi1Nu9BT7CoFA5pDvmszy/udUCACBchJN4bW+maxaLTXKIwN1iJw+fitmum5zkR+FzcHoufFuK4uVnyf3VzXlR38uLRB2akmT4Q61TkaCaupGNqLio+F6tZLOPaM8y3QcxiTu9EZpr1+WVBtohwaajEuco2Op+LXr1B++/03TCdCwnr0I9jpo2zmcmnk47CjBJwvphXe4NMRHBCp9O7bdIhGWlOmksU2NtvToZR0VyspjP7bYjE5+I0uHIyFYvmIqhw+G/eiq3sqGaLadcbxSJK8/+MEaHb2eH2cwnzEw2zF0t6mtq5qm1163ecpyKn/+9G2jOK0Vxqjs7rLPFZnO7Zmz86JpL6VHwuVo3B/o5HMVvM6RRHf54IF0GFXS6BFKMgaGyxMOUAIaYiK/ghwtrddcxixw3s5JhuN88E2WhM5Vpd+w12L8Mr3cTRLGYk+Y24nWb0RYG9KNME5ae5OK0xCoLKCv8SDc0lyL1RnRhifX722GhxkHfChYieIaK5xt8qIpprpPcion2WY3+1nDOciOYT0TIiuo9ysfFEAdGyXD0mmP1O3zhxYOpYDCv07zlnqMN5VrNY4n8nf0a75mW4/OjeWm0y0TGrTTiks1KZQTQXR4e+subi7dBP5XOpO4xZzFKrlyAKO55OTkX2CP9ixlILgxkvTqWrOcwjsKr19CICurZpqt2WHg6LRP1mkDVKzYWZz2PmYcw8DMBzAJ63HF5uHmPmqy3pDwK4EkB/429C9lpceHiFy7Bjf4mvssQX0xfxmSekYjAl6Nq6SUYe55Iyy5p98/H45amD0tLcfmPTLhqe9n1gZ+dYWNW16kPAzC1vI3Le6pbh8mBMTcrNAe6lven4iv48+TCfFrrXPbBzS9x91qG4/TuZ/hYAyVAsk0f2TKZZr6dleQnuVIy27cXjl43AOz8eq5T3pMGdMdUYdNljqlmf54rfnYJmZfpLD0f3bZ9c93a18fsTn4sHhvZxLoCnfPJ1AdCKmT/lxFv0BIDvZKGJBUtblxDxTgRx6Pfp0NwxwqtTWYtum4AfHZ8Kb6PaiTk7OdW7YXtQyZ7tm2GpQ0iVXVWJTcR+fMIAnO0TxTlzKjLhpMEHKLcJUHfeBqHYZ9Rv17S+M6yrdh3lJUU4sL17IEo3zFhdL197NM49oge+OypzphiQiBy86s5TcPrQ9LbdfIrz2p2gtCgvSW5ZoMLVY/tixW9Pzrh2p2e36s5TXMuZ7RK8cki31gCA/oaQ8RuEZGGZS/4KFwBjAGxk5q8tab2JaA4RvUdEY4y0bgDWWfKsM9IERcb0D76fhqfPxeXYOz8Zh0O7t87MrxCWJBcbVpk4zQwyfV8tm/iPOO0LBIuI8OfJh7tGu3XC2aQUzUwss2i3Ua1duFzjEJEZSJmMguDWxge/ezje/NExmc9AQbISEc4NaArr0KIMfzw30xQbBEezovF/J9tgxsxqD9rZvkU5WhqBMA/p1irp8zHLYdv5aXVZ0go2/AsRTQfgZJi+iZlfMj5PRrrWsgFAT2beQkTDAbxIRIM1650CYAoA9OzZ0yd3/hHXCxFGRQ46Fdl5hot/HX6mJFOgdW+rZrsOe0dNzaVlE38/lX20WFxEKC0uct2nwwmne6oqcJPLXFzym2YxVeHiVs5z3z/S9dxThnTxb6gDzcpKInPC6/DQRRUYfmBb/Gf2Ov/MAdhXkwizM/7gdA22uIhQX8do18zBimDc9wcvHJ7ytxhpZh/hJ3PrClW4MLPnUI2ISgCcCSBp+GbmagDVxufZRLQcwAAA6wFY7RHdjTSneqcBmAYAFRUV2dlIOiJemfcNrv3XnFjKDvWeaQqQ5DGHt9+tHda8qh3peUf0xFOfrVXLHAJzJN2+hb8pMYrZYk6aS3lpul/KLYRKsuNxKTsVrt75eGZEAfX2lxYXYdbNx6N1U70N5KKAmZM+mwGKm5uZlCusYwnDMQM64rZJg3FuRbpmlXjO7BnNO20GnvEsUpqL97PJhlksXwNXHg/gK2ZODheIqCOArcxcR0R9kHDcr2DmrUS0k4hGAZgJ4GIAf85Jq2Pk9QXfxlZ2GOESdCqyk5BwG03phA7xC2ESNTedMggDO7fC2AEd8cq8DZ557dM/gzn0M1HtAH01lyJvzcXeXF3ZqKKh6cYWU6VZWQn+dcVIDO6aaY71QmWRZBhKi4tw8eheGeklRYRqOD9vM81xGwOP22fNX9+IY4udj0xH/jEA5hlTk/8D4Gpm3moc+wGARwAsA7AcwOvZami2sG4TGzVxmcW8uninjsnthbcKMNXRvvqCwXA/stZNS3HZ0b2VJgtE0XGWl2b+ZJWFS7J6l9liFof+gl+flHHcfo1xCPDykmL/TJqY7T7SZXdPL1TCu8RBUVLQZx4zryd9GwPb+T7vYzbCv+Sl5sLM33NIew6JqclO+WcBcJ6bWCDEOXUwlObiFVvMU3PJPOimuaT5XPw0F8+juSWKiD1NHDrfqEbX5r2tY07b9z2bXD22D+57+2v/jAZxP2/tbZAjomPL8sRkES9NxOHqU7uNepefjanIym8QER0NoD8z/90wUbVg5pXxNU2wEue7EJtD3/M8B+HiprmQ82cvVDWSbDreopiQ0bTMQbhENLpOmsUUTSZxrFUOss4jTuxT0rP1wvzz8pGYsWSTo6blNN3e/Oy3Z49JNhz6Sm8lEd0K4OcAbjSSSgH8M65GCdkllD/fc7aYnuRxEy7WH4pvYD/NDi8b5gGTKEaLduc9kHmf3aoJ63MBgJ9PsEZgKGxW3XlKLGY6Fbq2aYoLRzqv5THxcuj7/QzyaSryGQAOA/AFADDzN0SU/XmBjZh4zWLBy/YOua+HygZGTmFdnMjHqYDm5d02aTCO7Ns+UBnlYbQUn9liVrOYG98f1xdPzlyNddv2RRoj7P4LDsPuGOPlRUYeSVSn3T3NR+enVebTbLH9zMxExABARPpLbIVQxDnQiGkmsvY6FzcBSg6aC5HzPcnWb3/cQR3Rvnl5Rtp/Zq/D4K7OoWLM5rZuWop+nYKNzZwc+qqkRrVuDv3E/3UecbmsBI1i7cSph+qv9vdj3EGd8PmqbeiiGDKooWDe9TSHvilcPOLHWR97PvlcniWihwC0IaIrAVwG4OH4miXYifNlCDOKCbpZ2KJvdmakuUWytZbiN3vXXOkcdH3CkX3b4+PlW3wF7mOXjshIO/XQrjhuYCc0KyvByN7tMKJ3u7Tj5jMMs3FZGP9KclTrclxFcwEyO7Jc0qNdM8xdu93x2A/G9cV5R/TQWqTacEl/qr7rXPIlKjIz34PE9N/nABwE4BZmLri1JPlMnD/jcPvAux/zer9XbN6TkabmSCbLv5n8/pyhuPvsQ5XXM9iv/dHvHYHPb1IPx2LHdEg/c9Vo/PjEgxzr0hUu029IBUj00lyG9khE371iTG+t8k3sDv1BXVIa2AGtLPu+u9z968f3x1NXjgpUd1DuPHMIHrzwcACJBYlWiMhXsNx91qGxTvOPA3NA5/RrSTn0vcvIC82FiIoBTGfmYwHEt0OV4MgLc9bh5bnfoLOmat++eRm27NmvlDfMixbUNPK9I3vhudnrsKs6ZWdXmS1m/mgGHNASX327KyNv66alGauddWhSWowmpcVYu21v4DLcMEeLuopLv04t8OMTBuAPby1F66bOkQCW//ZkEIItzDRJbrRlvA9PXTkKa7buxdpte5OCy4sfnTDAN0/UNC8vwcQhXTyDPXpx7hE9cGS/9jj6rhnJtNOGdk3bbKwhkNIm079bObBdypuRF1GRmbkOQD0R6S1tFTKoqqnDK/O+0TrnR898iRlLKh0j6nox+5cnKNcRbp1LsGO/On0w5tsW6qmY+kuKivDkFSPx5BUjFVuYyfXj+wc+NwymGSlI///D8f2x8ncnu5r7iovIV7D0aJeIt9a83HkGVFEy/Euina2blWJI99Y4eUgXdAuwz0hDwW7a/dlJB+EvFw53yZ2fJK8gGVss/ZqevWo0Th7SOanl5ZNDfzeA+UT0FoCkPYOZr4ulVQXKHa8uxj8+XY2OLcoxso/ebKGtNi0kyj3C4xMuer1o347+80SIUuHXg/KjEwbgsqN6Y+htb7qaG+OYGGA+sqDrQ4go1Aytu846FKcP7eo6mcCMfqAya6+QsN/SIJu4ZRMvZz075Hn9+jE42DBxmuuk8sIsZvA80jftEjTZX1uPNVsTphbVLYqrjIipAPDmoo2xtAvQf9EuO6o3Hv0osX7Wy3+g8xOdfsPY5IZHWSEH/UdQn0tUtGxSigmHuEcl9go54kQ21wjFif1x5LtwMbHe/+Q6F4dFlOlmZTNfnggXZn6ciMqQiEIMAEuYuSa+ZhUeA25OhTtT7VsG/vK/MbUmHLecNigpXIKaxex4CZagP/X3fjoO2/bmz2ua1FxClBHl9F87ZsfjN7EiR7IxNuz3NFfCX5XvHdkLf3hradoeQvYmOy2wBCwzArPgUlISLkQ0DsDjAFYh8dvoQUSXMPP78TVNyBbhwr8Em4qcDQ5s3xwHulgfzcFpK4V9WKz89owhmLt2W6D2JKci52m4WLtDv7Fgf4X9/Jm55ofj++OHLn5DJ+3YaUJMPpnF/gDgRGZeAgBENACJqMUNy+slOBJqnYvXMcXfaAefvVDiiGHVskkpbps0GMce1EnrvAtG9sQFI4NtNKe6ejpXHHdwJwzt0QbXKU54KBQRZH8aYWbc5QvpmkuKIoUQP1GhKlxKTcECAMy8lIiyv+tPgaDat7QoL8Hu6vhDYoQL/+JxTOH8xbdNyJmZxWkfjTiJYhFlnPeqVZNSvHTNUf5tiK8JuaGB+lysmC12isLg7HOJv02qCvosInqEiMYZfw8DmBVnw4R4nW43WNYkxBVyX6UXalqWWFfSGPBa4PadYV2V1pII0WM33+a7WcwJsgmN9Cuw+lwS/+eT5vJ9ANcAMKcefwDgL7G0SEgS54zQNpZQ3rHFFotojLtxZ1Uk5eQa8wftdF/+9/zDlMrIJ4taNmYcZQP7Pc13h74X5hNx87mYQiif1rmUAPgTM/8RSK7abwwBe2JBtdONc3Rh3Q89XPgXD4d+RL9R+xqfhopqaI58J1s+o3d+PNY/UwQ0tHUuTnjPFkvRvW1TTJ04EL3bxx97WNUs9jYA6xLdpgCmR9+cxoFKwD9mxv4Y5wuWWHbYCxe40uNY8GLTuCEHYUXioN5l9bQOuZ6Bl036dMzOuifr87juuH4NUriYpGaLpdKs13dAqya4emxf9GzfLPa2qGouTZh5t/mFmXcTUfyty1P219Yrby27essePPKB/oadNXUcq9OtNE24xKO5OHHCoAO06+jUqgm6tWmK9dv3aZ8blMFdW+P4gw/AT06KTrAViuZSaJiPo3lZMW6wBRttKGT6jYosx3KDquayh4gON78QUQWAUL90IjqHiBYSUb1RnvXYjUS0jIiWENFJlvQJRtoyIppqSe9NRDON9GeMBZ+x8N7SSgy4+XXXMN92rnt6Lv7x6eq0NJXRZ3VtnW8eOxMP6ayc1/ryfX9cX+26VHAaoT9wweEOOfOPspIiPHJJBQZ2dt6bJQh1yXUu+TlbrLFSCPf04C6JkD69OyTMXW2bp7rAXF2fqnD5HwD/JqIPiOgDAE8DuDZk3QsAnAkgbSEmEQ0CcD6AwQAmAPgLERUbfp4HAEwEMAjAZCMvANwF4F5m7gdgG4DLQ7bNlXeXbAIAzFq1VSn/1j3VgeoJ4sQ+uIt7R9i+ebq8NTWXDi3KQm3U5PTiXj++P3560kGOIlRV4ytE6h1MFkHJZYdomo0Kw52vNthL+iXz9KLPHt4dr103BuMPTlgGrL/3XJlSPX/pRHQEEXVm5s8BDATwDIAaAP8FoG/rscDMi61rZyxMAvA0M1cz80oAywCMMP6WMfMKZt6PhICbRInh8XFI7DcDJCIJfCdM27zbnfhfxW5eX89Yu9VZwVv0zU4c/Mv/YsOOzOP79tfh3SWV2m3zMm9dcmSvtO+m5lLP4ToqJ7PYj04YgGuO7RdLB9iQR5m/mXQIRvVpp7zXTL7y8MUVuOLo3ujToUA2pG3A75QJEWGQZQfUdi3yX3N5CIA5VWc0gF8goT1sAzAtpjZ1A7DW8n2dkeaW3h7AdmautaVnQERTiGgWEc2qrNTvvNPKUsjj2tkT8M+Zq7Gvpg7TF2/KOHzWgx/j9lcXAwBevOYofPCzY7HsjokKGwClf//sF+PxPUOo2J2UpkOfmcPFuvJ06Mf/VgfdcTIXHNKtNZ6eMjrUup586Ad7d2iOm08dlLeRBnSxRxR2zmNkaiCX3LI85U7PVZRrP4d+MTOb9p/zAExj5ucAPEdEc/0KJ6LpAJwcATcx80t6TQ0PM0+DIRQrKioC3XFOzvjxz+v1TEuNzr7WYUbYog2pLYD7d2qB5saLkprD7lK2TZh1atUkuXOhvb3mdrlBX7sPfnYsiorI26Ef8w/xnR+PRaumEihCCIfKa3pk30SQugtGBAv9k22ICL07NMfKzXtyFivOV7gQUYmhFYwHMEXjXDBzkP1i1wOwbiXY3UiDS/oWAG0s7bTmjxyn/RLc8DJTlRqde43PdGPrKNcsrrS4CNW1mec5C5zEfxmzSYz66+s50Ai0R7vEZMFNHr6hKAe2V43tg1teWoh2Fltytqaq5hMFoizkFSrvf9c2TQPvdpkrkqvx81RzeQrAe0S0GYnZYR8AABH1A7Ajpja9DOBfRPRHAF0B9AfwGRL9eX8i6o2E8DgfwAXMzEQ0A8DZSPhhLgEQm1ak43PxsIolO/cay/aLX6zZhrte/yr5vay4yHHOvbtwyazQqu1YSZrFEDIEfET7ufhx8ehenrHA/nZJBdo0i22SoJBFpt8wFjv2ZW/hbKHK6+IcR7n2FC7MfAcRvQ2gC4A3ObWUuwjAD8NUTERnAPgzgI4AXiWiucx8EjMvJKJnASwCUAvgGmOrZRDRtQDeAFAM4FFmXmgU93MATxPR7QDmAPhbmLZ5YS6AVDOLeWkuiQKsmsstLy3AgvUpk5hp0nI7N7O+zDQ3M16pORU5pEM/yp0ow2DOkil8CrUrTJHVTeMKmNTeLXkoXACAmT91SFsatmJmfgHACy7H7gBwh0P6awBec0hfgcRssthxDgznjJdwMWdr1Vo0l86tmqYJFzfHr2lSy2yb+ktkai71zKEc796xxQRByBXZjIDsWH9uqm24JJ+T8eB6TX0VNzzrPLfBa8Bgdu419SnNpWPLdLNOmYsQcRUuTmkuPpdSi1ksDNmILSakkHsqqHL/BYfhvIoenuvf4kSEiyZmZ/3LFxck97h//gvn+QNumgRzSnDU1CbyLN24Czv3pe/dUuJi/nIziznVl9rfIT09tc6FYzOLpeqSHjFq5I5GT4EEeU7Sp2ML3HX2oTmLlaYaW0xIknoDZ6/23u7WTXOpZ04+cNPncuK9mTtGu70SbpqLs8/FuYzUOheXShTx8qtYZ7fV1uuHsxGEbCDaYDyI5qKJtTP2C9ro5nOpZ+D9rxOLOPfVuHe6bsLJLYSK82wx50JKLetc4tJc3LQmIThyK6PHtCJcMaZ3jltSWIhw0cTaf5sCwg034fL6/A3J8C779nsJF2/BYOfyoxM/jsuOSv1I3KZOm6YqDunQ9xKwTUuL0b1tU/zuzCGByxeEuCkpLsKqO0/BjxtoROR8Rcximlg1gQffXe6d10XzWL11b/Lznv21roucXE1aLjbU7m2buS70sp9RGlHYFLNcpyYVFxE+/PlxAIDrn/YN6CAoUCghV4TCR4SLJjpTxt00j/2WBZB799dh7ba9jvnciMJB16y0GFeN7YNTh3SNxGzlpk0JgtA4EeGiiY4D3E0QWRdO7q6qxdjfv+tyvp5ZzAm3RZTFRYQbJx4MwDm+mSpmuQNzNN2xsSF6i9BQEOGiicoWxSZu5i6rcFmz1V1rcRMuOptNucVCs5pXwphampWV4LFLj8DQ7m0ClyHoI+YxId8R4aJLFGYxq+ZSXeuYJ3G+c7qTbDnr8O6ebfASSGG7qXEHdQpZgqCKyBShoSCGck10loW4msVq1UpxW4TpNEPr3Ao34ZI65zeTBivVKwiCEBYRLproxO9S0Vy863JOd3Lou2kmpmmuiAgXuUQVltGwIAhRI8JFEx3NZfwf3nNMr3EIl++Em3BqXpYZ0NLN6mWW4TUHQOz3DYdc7YcuCLqIz0UTVcXFS8NR1VzczGrNyzMfm5uAMKvKtQA5r6IHhvdqm9M2FBIiYoR8R4SLJqqai1dYF7/dJ01cNRcH4eK2Uj6pueRYuNx19qE5rb9QECVTaCiIWUwTN43ku4/MTAu9//biTa5lqC7EdFN+mpc5aC6udZlmMemVBEHIHiJcNHGTCx8u25wWev+HT80JX5er5uLkc/HenVJGvIIgZBMRLrooah0VB4b3L7hpON3aNM1IcxMe1tligiAI2SInwoWIziGihURUT0QVlvQTiGg2Ec03/j/OcuxdIlpCRHONv05GejkRPUNEy4hoJhH1irPtqiv0D49EuDjX1aZZGb689cS0tGqXGWhiFissZIwgNBRy5dBfAOBMAA/Z0jcDOI2ZvyGiQwC8AaCb5fiFzDzLds7lALYxcz8iOh/AXQDOi6ndyrPF3EK/aNXlkk4EtG5ampa2s6rGuR3mCn2jU7r0qF4Ft+OeIAj5R06ECzMvBjKnxzKz1VGxEEBTIipn5mqP4iYB+JXx+T8A7iciYp3VjhooT0WOpDLnZKfBa4+2maYyIHMq8q2nySr9QkA0GCHfyWefy1kAvrAJlr8bJrFfUkoydQOwFgCYuRbADgDt42qUillsx94a/O3DlaHrcjOLmZc+uGsiEvGXt5yIfp1aOublPJmKLERDrtcrCYIqsWkuRDQdQGeHQzcx80s+5w5GwrxldSxcyMzriaglgOcAXATgCc02TQEwBQB69uypc2oSFc3lizXbApVtxzUqstG/PD1lFLbs3o/WzUod81nLKMrnYYQgCAVHbMKFmY8Pch4RdQfwAoCLmTm51SMzrzf+30VE/wIwAgnhsh5ADwDriKgEQGsAW1zaNA3ANACoqKgIZLlSOak2An8L4D5bzBy8tmxSipZN3AULANS5bHMsNEzkKQoNhbwazxJRGwCvApjKzB9Z0kuIqIPxuRTAqUhMCgCAlwFcYnw+G8A7cflbADXNpS4i4eKOxn4uimYx2edeEIQoydVU5DOIaB2A0QBeJaI3jEPXAugH4BbblONyAG8Q0TwAc5HQVh42zvkbgPZEtAzADQCmxtt6f8GhK1x0ZwnrKCF1iutcJo8IZiYUsosooEJDIVezxV5AwvRlT78dwO0upw13KasKwDnRtc4bJc1FU3FqUV6CnVXum4bZ0elfdHwuD100XGtLAUEQBDfyyizWEFDpeuvq9fakb+EQiHJo99b40/nDHPPr+E+sm4X5cdLgzphwSBfHY0tvn6hcpxAfEnJfaCiIcNFEZWRfW6c3+j/l0MwO/aVrj8akYd0ccuuZ0czFnLJCXxCEbCLCRRM1zUVduMz8xXhMnXiwVht0Rq91rOZz8dR8FHMAAA9BSURBVK1TZFNeIM9BaCjIfi6aRO1zOaBVE+026HQwKbOYdjVCHiPmMSHfEc1Fk1tOG+SbJ/6pyOpEFRWZAAzq0gqThnWNoFWCIBQ6orloUua1Gb1BEOFy6VG98MWa7fhy7XbfvHqaS3Q+l9euHxO6DCEcoq8IDQURLpqodOxBhIsZULLX1Fcd92tJa4NGF2NuiVxSHNbnIt2aIAjqiHDRRKWTDRP+ZfFtE3zXpOjECbt/8mF4Yc56HHSAc2BLoYEhMl5oIIhw0UTltx3G59K0LHML48w2qPcwnVo1wVVj+wZuT6pOQRAEdcShr0kYs9joPu0j8X2IharxIrPEhIaCCBdNVH7cbmaxYwd29PWnqLUh+4hAyzPkeQh5jggXTVQ6WbctjqOaoSwdfeNFnr3QUBDhoonKb9tNc4lq/UsuZm7JbDFBEHQQ4aKLks/FOXBlVBGHpZsXBCHfEeGiSRifS51esGT3NogW0WiRJy80FES4aBLG56K7z4trGyIpRRAEIT5EuETMpp1VrppLZGYxkS6NFtFahYaCLKLUxO+nPeK3b6N7W+fpxnX1DFYK2u/XBulgGjvyBgj5Tk40FyI6h4gWElE9EVVY0nsR0T4immv8/dVybDgRzSeiZUR0HxlDOCJqR0RvEdHXxv9tY267b5512/Y5pstUZCEs8uiFhkKuzGILAJwJ4H2HY8uZeZjxd7Ul/UEAVwLob/xNMNKnAnibmfsDeNv4Hhthftz1YhYTBKGRkBPhwsyLmXmJan4i6gKgFTN/ygnHxRMAvmMcngTgcePz45b0WAjTsbs5+vXbINKlsSKPXmgo5KNDvzcRzSGi94jI3ECkG4B1ljzrjDQAOICZNxifvwVwQJyNC+PvsM4WKysuwk0n621vnGqDIAhCfhObQ5+IpgPo7HDoJmZ+yeW0DQB6MvMWIhoO4EUiGqxaJzMzEbmqB0Q0BcAUAOjZs6dqsbZCgp0GpG+R/O+rR2NojzbBmiDSpdEj74CQ78QmXJj5+ADnVAOoNj7PJqLlAAYAWA+guyVrdyMNADYSURdm3mCYzzZ5lD8NwDQAqKioCGSjCvOjtoZ/CbPtsMwWa7zIsxcaCnllFiOijkRUbHzug4TjfoVh9tpJRKOMWWIXAzC1n5cBXGJ8vsSSHk8bQ5xrNYuFEVIRRO0XBEGIlVxNRT6DiNYBGA3gVSJ6wzh0DIB5RDQXwH8AXM3MW41jPwDwCIBlAJYDeN1IvxPACUT0NYDjje9xtj3wuRyRcJHBayNGnr3QQMjJIkpmfgHACw7pzwF4zuWcWQAOcUjfAmB81G10I5TmYjGLhTFviGlEEIR8J6/MYg2BUFORLV6eohB3Xpy5jRd59kJDQYSLJmG0hvrINBdBEIT8RoSLJuE0l2h8LrKIsvEiT15oKIhwyRLHHtQRPznpoOT3MDO+ZLaYIH43Id+RqMiauCkNZxzWDS/MWZ+RPv2GsaiqqcMh3VrbSwreBulYBEHIc0S4aOLWsRe7qBP9OrVwTA+lfYhsabSISVRoKIhZTBO333aJprQI00lI/yIIQr4jwkUTt369pFitxzd9+mE0F5EtjRd59kJDQYSLJm4aR4nmwpVQU5FFdREEIc8R4aKJq+aSxSlcMltMEIR8R4SLJm5KQ7GiWSySNohxRBCEPEeEiyZWk1SX1k3Qojwx4S6bmotYxQRByHdEuITEXHWv63MRhCCYMR5kgCHkO9IjhsSc/VWaTbOYdCyCIOQ5IlxCQADYGEsW684WCzUVWaSLIAj5jQiXiJDZYoIgCClEuITENIupLqKMAlnnIghCviPCJQRElHSwlhRn71aKaBEEId/JiXAhonOIaCER1RNRhSX9QiKaa/mrJ6JhxrF3iWiJ5VgnI72ciJ4homVENJOIesXd/ltOHZT6YmouMhVZEAQhSa40lwUAzgTwvjWRmZ9k5mHMPAzARQBWMvNcS5YLzePMvMlIuxzANmbuB+BeAHfF3fjJI3oCAK4Y0zuZtr+2Pu5qk4hZrPFijmHaNivLbUMEwYecCBdmXszMS3yyTQbwtEJxkwA8bnz+D4DxFHPv27SsGKvuPAWXHtUbtfUJoXJ4z7YZ+U4f2jUj7ZFLKjB5RA90a9NUu95bTh2Esiya3/515UhcPPpATBjcOWt1Ct40KyvBHWccgqenjMp1UwTBE2LL1rtZr5zoXQA/YeZZDseWA5jEzAssedsDqAPwHIDbmZmJaAGACcy8znLeSGbe7FV3RUUFz5qVUa02vaa+CgB4/6fHon2LMgy+9Q0AwJj+HfCPy0eGLl8QBCGfIKLZzFzhly+2YTARTSeiBQ5/kxTOHQlgrylYDC5k5iEAxhh/FwVo0xQimkVEsyorK3VP96RJaRGal5fgiqN7gwh44MLDIy1fEAShIRHbTpTMfHyI088H8JStvPXG/7uI6F8ARgB4AsB6AD0ArCOiEgCtAWxxadM0ANOAhOYSon0ZlJcUAwBuPnUQbrY6/AVBEBoheTcVmYiKAJwLi7+FiEqIqIPxuRTAqUhMCgCAlwFcYnw+G8A7nANbX3lp3t1KQRCEnBGb5uIFEZ0B4M8AOgJ4lYjmMvNJxuFjAKxl5hWWU8oBvGEIlmIA0wE8bBz7G4B/ENEyAFuR0HqyTnmJCBdBEASTnAgXZn4BwAsux94FMMqWtgfAcJf8VQDOibiJ2sj0YEEQhBQy3BYEQRAiR4SLIAiCEDk5MYsVEv++ejRWVu7JdTMEQRDyChEuITmiVzsc0atdrpshCIKQV4hZTBAEQYgcES6CIAhC5IhwEQRBECJHhIsgCIIQOSJcBEEQhMgR4SIIgiBEjggXQRAEIXJEuAiCIAiRk9OdKHMJEVUCWB3w9A4APHe6LEDkmhsHcs2NgzDXfCAzd/TL1GiFSxiIaJbKNp+FhFxz40CuuXGQjWsWs5ggCIIQOSJcBEEQhMgR4RKMabluQA6Qa24cyDU3DmK/ZvG5CIIgCJEjmosgCIIQOSJcNCCiCUS0hIiWEdHUXLcnKoioBxHNIKJFRLSQiK430tsR0VtE9LXxf1sjnYjoPuM+zCOiw3N7BcEhomIimkNErxjfexPRTOPaniGiMiO93Pi+zDjeK5ftDgoRtSGi/xDRV0S0mIhGF/pzJqIfGe/1AiJ6ioiaFNpzJqJHiWgTES2wpGk/VyK6xMj/NRFdEqZNIlwUIaJiAA8AmAhgEIDJRDQot62KjFoAP2bmQQBGAbjGuLapAN5m5v4A3ja+A4l70N/4mwLgwew3OTKuB7DY8v0uAPcycz8A2wBcbqRfDmCbkX6vka8h8icA/2XmgQCGInHtBfuciagbgOsAVDDzIQCKAZyPwnvOjwGYYEvTeq5E1A7ArQBGAhgB4FZTIAWCmeVP4Q/AaABvWL7fCODGXLcrpmt9CcAJAJYA6GKkdQGwxPj8EIDJlvzJfA3pD0B340d3HIBXABASC8tK7M8cwBsARhufS4x8lOtr0Lze1gBW2ttdyM8ZQDcAawG0M57bKwBOKsTnDKAXgAVBnyuAyQAesqSn5dP9E81FHfMlNVlnpBUUhhngMAAzARzAzBuMQ98COMD4XCj34n8B/AxAvfG9PYDtzFxrfLdeV/KajeM7jPwNid4AKgH83TAFPkJEzVHAz5mZ1wO4B8AaABuQeG6zUdjP2UT3uUb6vEW4CEmIqAWA5wD8DzPvtB7jxFCmYKYWEtGpADYx8+xctyWLlAA4HMCDzHwYgD1ImUoAFORzbgtgEhKCtSuA5sg0HxU8uXiuIlzUWQ+gh+V7dyOtICCiUiQEy5PM/LyRvJGIuhjHuwDYZKQXwr04CsDpRLQKwNNImMb+BKANEZUYeazXlbxm43hrAFuy2eAIWAdgHTPPNL7/BwlhU8jP+XgAK5m5kplrADyPxLMv5OdsovtcI33eIlzU+RxAf2OWSRkSTsGXc9ymSCAiAvA3AIuZ+Y+WQy8DMGeMXIKEL8ZMv9iYdTIKwA6L+t0gYOYbmbk7M/dC4lm+w8wXApgB4Gwjm/2azXtxtpG/QY3wmflbAGuJ6CAjaTyARSjg54yEOWwUETUz3nPzmgv2OVvQfa5vADiRiNoaGt+JRlowcu2Eakh/AE4GsBTAcgA35bo9EV7X0UiozPMAzDX+TkbC1vw2gK8BTAfQzshPSMycWw5gPhIzcXJ+HSGufxyAV4zPfQB8BmAZgH8DKDfSmxjflxnH++S63QGvdRiAWcazfhFA20J/zgB+DeArAAsA/ANAeaE9ZwBPIeFTqkFCQ708yHMFcJlx7csAXBqmTbJCXxAEQYgcMYsJgiAIkSPCRRAEQYgcES6CIAhC5IhwEQRBECJHhIsgCIIQOSJcBEETIqojormWP88I2UR0NRFdHEG9q4ioQ4DzTiKiXxtRcl8P2w5BUKHEP4sgCDb2MfMw1czM/Nc4G6PAGCQWDY4B8GGO2yI0EkRzEYSIMDSLu4loPhF9RkT9jPRfEdFPjM/XUWLfnHlE9LSR1o6IXjTSPiWiQ4309kT0prEXySNILH4z6/quUcdcInrI2BLC3p7ziGguEiHn/xfAwwAuJaKCiCwh5DciXARBn6Y2s9h5lmM7mHkIgPuR6NDtTAVwGDMfCuBqI+3XAOYYab8A8ISRfiuAD5l5MIAXAPQEACI6GMB5AI4yNKg6ABfaK2LmZ5CIcL3AaNN8o+7Tw1y8IKggZjFB0MfLLPaU5f97HY7PA/AkEb2IRPgVIBF+5ywAYOZ3DI2lFYBjAJxppL9KRNuM/OMBDAfweSJcFpoiFZTQzgAAK4zPzZl5l8L1CUJoRLgIQrSwy2eTU5AQGqcBuImIhgSogwA8zsw3emYimgWgA4ASIloEoIthJvshM38QoF5BUEbMYoIQLedZ/v/EeoCIigD0YOYZAH6ORDj3FgA+gGHWIqJxADZzYj+d9wFcYKRPRCLIJJAIRng2EXUyjrUjogPtDWHmCgCvIrGfyd1IBFsdJoJFyAaiuQiCPk0NDcDkv8xsTkduS0TzAFQjsW2slWIA/ySi1khoH/cx83Yi+hWAR43z9iIVJv3XAJ4iooUAPkYifDyYeRER3QzgTUNg1QC4BsBqh7YejoRD/wcA/uhwXBBiQaIiC0JEGBuPVTDz5ly3RRByjZjFBEEQhMgRzUUQBEGIHNFcBEEQhMgR4SIIgiBEjggXQRAEIXJEuAiCIAiRI8JFEARBiBwRLoIgCELk/D/hS3rBl/18WwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def ddpg(n_episodes=1000, max_t=300, print_every=100):\n",
    "    scores_deque = deque(maxlen=print_every)\n",
    "    scores = []\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        state = env.reset()\n",
    "        agent.reset()\n",
    "        score = 0\n",
    "        for t in range(max_t):\n",
    "            action = agent.act(state)\n",
    "            next_state, reward, done, _ = env.step(action)\n",
    "            agent.step(state, action, reward, next_state, done)\n",
    "            state = next_state\n",
    "            score += reward\n",
    "            if done:\n",
    "                break \n",
    "        scores_deque.append(score)\n",
    "        scores.append(score)\n",
    "        print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_deque)), end=\"\")\n",
    "        torch.save(agent.actor_local.state_dict(), 'checkpoint_actor.pth')\n",
    "        torch.save(agent.critic_local.state_dict(), 'checkpoint_critic.pth')\n",
    "        if i_episode % print_every == 0:\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_deque)))\n",
    "            \n",
    "    return scores\n",
    "\n",
    "scores = ddpg()\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(1, len(scores)+1), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Watch a Smart Agent!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent.actor_local.load_state_dict(torch.load('checkpoint_actor.pth'))\n",
    "agent.critic_local.load_state_dict(torch.load('checkpoint_critic.pth'))\n",
    "\n",
    "state = env.reset()\n",
    "for t in range(200):\n",
    "    action = agent.act(state, add_noise=False)\n",
    "    env.render()\n",
    "    state, reward, done, _ = env.step(action)\n",
    "    if done:\n",
    "        break \n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Explore\n",
    "\n",
    "In this exercise, we have provided a sample DDPG agent and demonstrated how to use it to solve an OpenAI Gym environment.  To continue your learning, you are encouraged to complete any (or all!) of the following tasks:\n",
    "- Amend the various hyperparameters and network architecture to see if you can get your agent to solve the environment faster than this benchmark implementation.  Once you build intuition for the hyperparameters that work well with this environment, try solving a different OpenAI Gym task!\n",
    "- Write your own DDPG implementation.  Use this code as reference only when needed -- try as much as you can to write your own algorithm from scratch.\n",
    "- You may also like to implement prioritized experience replay, to see if it speeds learning.  \n",
    "- The current implementation adds Ornsetein-Uhlenbeck noise to the action space.  However, it has [been shown](https://blog.openai.com/better-exploration-with-parameter-noise/) that adding noise to the parameters of the neural network policy can improve performance.  Make this change to the code, to verify it for yourself!\n",
    "- Write a blog post explaining the intuition behind the DDPG algorithm and demonstrating how to use it to solve an RL environment of your choosing.  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}