dqn-explained.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Deep Reinforcement Learning\n",
    "\n",
    "* [Deep Q Learning Nature Paper - Human-level control through deep reinforcement\n",
    "learning](https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf)\n",
    "* [Playing Atari with Deep Reinforcement Learning](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf)\n",
    "* [LET’S MAKE A DQN: FULL DQN](https://jaromiru.com/2016/10/21/lets-make-a-dqn-full-dqn/)\n",
    "\n",
    "\n",
    "**Temporal Difference Learning**\n",
    "\n",
    "* [An Introduction to Temporal Difference Learning](http://www.ias.informatik.tu-darmstadt.de/uploads/Teaching/AutonomousLearningSystems/Kunz_ALS_2013.pdf)\n",
    "* [Temporal-Difference Learning](https://www.tu-chemnitz.de/informatik/KI/scripts/ws0910/ml09_6.pdf)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "%pylab inline\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from scipy.special import huber, pseudo_huber\n",
    "from sklearn.metrics import mean_absolute_error, mean_squared_error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Deep Q Network의 특징 \n",
    "\n",
    "Q-network에서 neural network를 사용하면 DQN은 다음과 같은 특징을 통해 향상을 시켰습니다. \n",
    "\n",
    "1. Multi-layer convolutional network를 통해서 더 복잡한 문제를 해결함 \n",
    "2. Experience Replay를 통해 더 stable한 모델을 만들수 있게 됨\n",
    "3. target network 를 사용 (즉 2개의 neural network를 사용)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Future Discounted Return \n",
    "\n",
    "Agent의 목표는 future reward를 최대치로 하는 actions을 선택하는 것입니다.<br>\n",
    "여기서 future reward란 $ \\gamma $ 배수만큼 (a factor of $ \\gamma $ per time-step) discounted 되는 것을 의미합니다. <br>\n",
    "쉽게 이야기해서 먼미래의 reward일수록, 더 적은 reward로 계산하겠다는 뜻입니다.\n",
    "\n",
    "### $$ R_t = \\sum^T_{t^{\\prime} = t} \\gamma^{t^{\\prime} - t} r_{t^{\\prime}} $$\n",
    "\n",
    "$ T $는 게임이 끝나는 시점의 time-step을 의미하며 $ \\gamma \\in [0, 1] $ 의 값을 갖습니다.\n",
    "\n",
    "아래의 코드에서는 gamma 값이 시간의 흐름에 따른 변화를 그렸습니다.<br>\n",
    "먼 미래의 reward값일수록 0에 가까운 gamma값과 곱해야 하기 때문에<br>\n",
    "코앞의 reward는 가중치가 높고, 먼 미래는 reward는 가중치가 낮게 됩니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XecVPW5x/HPM7ONrZRdlt6bgIKAiH0xFkxsN2osiTGm\nEKPkmsTEaG5iEuO9qTeaRFNMNHaxxCh6rVHQWFBAQQVEeu9tWWBhy3P/OGfXcd1lZxdmZ3fn+369\nzmtOP89v5sx5zvmdZu6OiIgIQCTZAYiISOuhpCAiIrWUFEREpJaSgoiI1FJSEBGRWkoKIiJSS0kh\nBZlZiZmtSdC8y8xsQCLm3dLM7Etm9upBzuMEM1t0gOH9zMzNLO1gliNyqCgpyCHl7rnuvizZcbQW\n7v5vdx9a021mK8zslObOz8zuMrObDk107YOZjTazOWa2J/wcfYBxDzOzl8xsp5ktMbP/aMlYwxhW\nmNnecAeqzMyerzP822a2wcxKzexOM8tsyfiUFKRVauk9ZzOLtuTy5NAwswzgCeA+oBNwN/BE2L/u\nuGnhuE8BnYHJwH1mNqTlIq51VrgDlevup8XEeDpwHfApoC8wAPhpi0bm7inXAGOAd4BdwCPAQ8BN\n4bBOBCvNZmB72N4rZtoZwE3A60AZ8CTQBbgfKAVmAf1ixnfgSmBxuLyfAQPD6UuBh4GMeJZdpwzf\nBx6t0+93wO/D9suBheEylwFfjxmvBFhTJ8ZBMd131XwfYfeZwFxgRxj3EQf4bmvnFc7nNuD/wjje\nBAY2MF2/cNqvAKuAV8L+E8Jl7gDmASUx0/QHXgnn/a9wWffFDH8E2ADsDMcbUaeMfwKeBnYDp4S/\n47Twd3kr/K1ebSDeu4FrwvaeYexXhd0DgW0EO1213zVwL1AN7A3XnWtjyn1ZWO4twH81sMzJQAWw\nP5z+yTjX92uB9cA64Kt1fqPPEPwXSoHVwE/q+U0uD4dtB64AjgLeDX+TW2PG/xLwGnBzOGwZcGzY\nfzWwCbgsZvwGl92E//JpwFrAYvqtAibVM+7I8HuLHfd54GdxLusu4lyfG5nPCuCUBoY9APxPTPen\ngA1NXcbBNC22oNbSABnASuBqIB34bPgnq0kKXYDzgGwgj2DD8njM9DOAJeEfvwBYAHxIsFFJA+4B\n/h4zvhPsneQDI4B9wIsEewA1018Wz7LrlKMvsAfIC7uj4R9/Qtj9mTBGA04Kxx0TDishzqQAHBn+\nmY8Ol3FZuFJnNhBX3aSwFRgffjf3A1MbmK5fOO09QA7QgWBjuxX4NMEG9tSwuyic5g3gN+FvejzB\nxiU2KXw5/B4zgVuAuXXKuBM4Lpx3FjCVIEnnEGxA1tJwUvgy4UYZuARYCjwUM+yJBr7rFcRsEGLK\n/dewzKPCdeSwBpZb+9vEub5PIkiMI8L16r46v1EJcHj4HRwBbATOrRPbn8Pv5zSgHHgc6Br+PpuA\nk8LxvwRUEiSRKMHO0yqCDWlmOP0uILexZYfDdxyguS4c59vAM3XK/BRhwq7Tv76k8ALwzzi/y7s4\nwPrMR4myvuaPddaBjQQ7f88Do2KGzQMujOkuDH+DLi22jWypBbWWBjiRT+5ZvNrQHw0YDWyP6Z5B\nzJ4c8L+xKyVwFh/f+DhwXEz3HOD7daa/JZ5l1zP8VeCLYfupwNIDjPs4cHXYXkL8SeFP1NmTAhYR\nbgjqWU7dpPC3mGGfBj5oYLp+4bQDYvp9H7i3znjPESSmPgQboOyYYfcRkxTqTNcxnH9BTGz3xAyP\nEuyFD4vp9z80nBQGEuw5Rwg2ml/noyOCu4HvNPBdr6D+pBB7NPoWcFEDy639beJc3+8Efh7TPaju\n711n/FuAm+vE1jNm+FY+vtH6B/CtsP1LwOKYYYeH0xfXmX50Y8tuQvl+RJ0dDYKN9U/qGTed4Ojl\n2rD9NIIdwufiXFbc63Mj8zmOYAcgG7ieIGl3DIctJeYoJ4zTial9SHSTiucUegBrPfzGQ6trWsws\n28z+YmYrzayUoNqhY506540x7Xvr6c6ts8y4xo9z2bEeAC4O2y8Ju2vKcYaZzTSzbWa2g2AFLmxg\nPgfSF7jGzHbUNEBvgu8xHhti2vfwye+mrtUx7X2BC+os+3ige7j8be6+p75pzSxqZr8ws6Xhd7ki\nHFRY3/hAEcHeX2y/lQ0F6e5LCaqdRgMnEOydrjOzoQRHZi83Us66mvo9xasHHy9TbDtmdrSZTTez\nzWa2k6B6qO560pT1ve4w3L2h9T2eZTemjOAoPFY+wRHJx7h7BXAuwVH0BuAagiPDplyJd9C/k7u/\n5u573X2Pu/+c4EjihHBw3fLUtH+iPImSiklhPdDTzCymX++Y9muAocDR7p5PcGQBQTVMojV12Y8A\nJWbWC/gPwqQQXq3wD4KqlWJ370hQd97QfPYQ7LXU6BbTvhr4b3fvGNNku/uDTS9eXOom63vrLDvH\n3X9B8Dt2NrPYuGN/x0uAcwiq9QoI9nrh499B7LI2Exx5xM6jTyOxvgycT3BOaG3YfRnBuaG5cZSv\nOZo6/XqgV0x37zrDHyA4j9Lb3QsIjnpaYl1vdNkxV+fU1/wgHG0+cESd//MRYf9PcPd33f0kd+/i\n7qcTVOO+dSgKY2bzDxDvnw8wqfNRuecTVCHWGAVsdPethyLGeKRiUngDqAKmmFmamZ1DUEdYI49g\nb2aHmXUGftyCsTVp2e6+maA66+/AcndfGA7KIKjD3QxUmtkZBIfKDZkLXBLuXU8i2NOt8VfginCv\nzswsx8w+Y2Z5zShfU90HnGVmp4exZYX3WPRy95XAbOAnZpZhZscQVN3VyCOom99KkPD+50ALcvcq\n4LFwftlmNpxgA38gLwNTCI7oIPgtphBUOVU1MM1Ggg1RczV1+oeBy8NLMbMJqlti5REccZWb2XiC\nZNpSDrhs/+jqnPqamt9zBsH/+T/NLNPMpoT9X6pvgWZ2RLgeZZvZdwmOOu+KGe5mVtKcwrj7iAPE\ne0U4/z5mdly4zmaZ2fcIjo5eC2dzD/AVMxtuZh2BH8bG1xJSLim4+36Ck8tfIThs+wLBof++cJRb\nCOr7tgAzgWdbMLzmLPsBgr3h2qojd98F/CfBBmE7wZ9t2gHmcTXBBnUH8HmC8w8185oNfA24NZzX\nEoK644Rz99UEe/s/IEhwq4Hv8dF6+3ngGIIN/00EV5HV/I73EFT/rCU4mT8zjkVOIagO2EDwR/x7\nI+O/TLBhq0kKrxIkoFcanAJ+DvwwrA77bhwx1XUHMDyc/nEAM3smZs/5Y9z9GeD3wHSC367me6j5\nnq4EbjSzXcANBOtMSznoZYf/53OBLxKsv18mOFm9H8DMfmBmz8RMcinB0dMmgit7TnX3feG4vQmq\nad5rdokal0dwnm47wbo5CTij5kjA3Z8FfkXwe60iWIdbcsc0ONma6szsTeDP7t7YRkBaMTN7iODE\nX4v+idoSMzsMeJ/g6rHKZMfTmpjZFwguW74+2bEkU0omBTM7ieAKmi0Ee5t/JrjqZX1SA5MmMbOj\nCO4HWE5QPfY4cIy7v5PUwFoZC+7afZrgKOZuoNrdz01uVNJaperzVoby0fXoy4DzlRDapG4E5wG6\nEFxB8g0lhHp9naA6rIqgyuvKpEYjrVpKHimIiEj9Uu5Es4iINKzNVR8VFhZ6v379mjXt7t27ycnJ\nObQBtSGpXP5ULjukdvlV9qDsc+bM2eLuRY1N0+aSQr9+/Zg9e3azpp0xYwYlJSWHNqA2JJXLn8pl\nh9Quv8peAoCZNXiHfixVH4mISC0lBRERqaWkICIitZQURESklpKCiIjUUlIQEZFaSgoiIlIrZZLC\n26u28/Ci/eixHiIiDUuZpDB/7U6eXl7Bkk1lyQ5FRKTVSpmkcOrw4A2Tz83f0MiYIiKpK2WSQreC\nLAYWRHhWSUFEpEEpkxQAxnaL8v7aUtZs35PsUEREWqXUSgpdg+f/PTd/Y5IjERFpnVIqKRTnRBjW\nLU/nFUREGpBSSQHgtBHdmLViG1vK9iU7FBGRViflksKkEd1wh38tUBWSiEhdKZcUDuueR5/O2boK\nSUSkHimXFMyM00cU89qSLZSWVyQ7HBGRViXlkgLApJHdqKhypn+wKdmhiIi0KimZFI7s3YmivExd\nhSQiUkdKJoVIxDhteDEzFm2mvKIq2eGIiLQaKZkUAE4f0Y09+6v49+ItyQ5FRKTVSNmkMGFAF/Kz\n0lSFJCISI2WTQkZahFMOK+ZfCzdSUVWd7HBERFqFlE0KENzdvGNPBW8t35bsUEREWoWUTgonDSki\nKz2iKiQRkVBKJ4UOGVFOGlLEc/M3UF2t13SKiCQ0KZjZJDNbZGZLzOy6eob3MbPpZvaOmb1rZp9O\nZDz1mTSyGxtL9zF3zY6WXrSISKuTsKRgZlHgNuAMYDhwsZkNrzPaD4GH3f1I4CLgj4mKpyEnDysm\nIxrh/95d39KLFhFpdRJ5pDAeWOLuy9x9PzAVOKfOOA7kh+0FwLoExlOvgg7plAwt4sl566hSFZKI\npDhzT8yG0MzOBya5+1fD7kuBo919Ssw43YHngU5ADnCKu8+pZ16TgckAxcXFY6dOndqsmMrKysjN\nzf1E/1kbKrlt7j6+Ny6LEYXRZs27LWio/KkglcsOqV1+lT0o+8SJE+e4+7jGpklLeFQHdjFwl7v/\nr5kdA9xrZiPd/WM3Drj77cDtAOPGjfOSkpJmLWzGjBnUN+2EiiruWfgvVnghV5WMata824KGyp8K\nUrnskNrlV9lLmjRNIquP1gK9Y7p7hf1ifQV4GMDd3wCygMIExlSvrPQop4/sxrPvb9CzkEQkpSUy\nKcwCBptZfzPLIDiRPK3OOKuATwGY2WEESWFzAmNq0Lmje7JrXyUv6XHaIpLCEpYU3L0SmAI8Bywk\nuMpovpndaGZnh6NdA3zNzOYBDwJf8kSd5GjEMQO7UJSXyRNz6x7MiIikjoSeU3D3p4Gn6/S7IaZ9\nAXBcImOIVzRinHVED+6buZKdeyooyE5PdkgiIi0upe9oruvcI3uwv6qaZ97XPQsikpqUFGIc3rOA\nAYU5PDG3xW+XEBFpFZQUYpgZZ4/uwczlW9mwszzZ4YiItDglhTrOHd0Td3hyno4WRCT1KCnU0a8w\nh1G9O/K4rkISkRSkpFCPc0b1YP66UpZs2pXsUEREWpSSQj3OHNWdiKETziKScpQU6tE1L4vjBhXy\nxNx1JOleOhGRpFBSaMA5o3uyatse3l6ll++ISOpQUmjA6SOKyUyL6LEXIpJSlBQakJeVzukjuvHE\n3HV6cqqIpAwlhQO46Kje7NxbwXPzNyQ7FBGRFqGkcAATBnShT+dsHnxrVbJDERFpEUoKBxCJGBce\n1ZuZy7axfMvuZIcjIpJwSgqNuGBsL6IR46FZq5MdiohIwikpNKJrfhafGtaVR+esoaKquvEJRETa\nMCWFOFw0vjdbyvbx4sKNyQ5FRCShlBTicNKQrnQvyOLBt1SFJCLtm5JCHKIR44JxvXll8WbW7tib\n7HBERBJGSSFOnxvXC4CHdcJZRNoxJYU49eqUzQmDi3hk9mqqqvWQPBFpn5QUmuDio3qzbmc5r3y4\nOdmhiIgkhJJCE3zqsGIKczN0h7OItFtKCk2QkRbhvDG9ePGDTWzaVZ7scEREDjklhSa68KjeVFU7\nj85Zk+xQREQOOSWFJhpQlMvR/Tvz0KzVVOuEs4i0M0oKzXDJ0X1YuXUPryzWCWcRaV+UFJrhjJHd\nKc7P5M7XViQ7FBGRQ0pJoRky0iJcOqEvr3y4mcUbdyU7HBGRQ0ZJoZkuHt+HzLQIf399RbJDERE5\nZJQUmqlLbib/cWRPHnt7Ddt37092OCIih4SSwkG4/Lj+lFdU8+As3cwmIu2DksJBGNotj+MHFXLP\n6yv1Ah4RaReUFA7Sl4/vx4bScp55f0OyQxEROWhKCgepZEhX+hfmcOery5MdiojIQUtoUjCzSWa2\nyMyWmNl1DYzzOTNbYGbzzeyBRMaTCJGIcflx/Zi7egdvr9qe7HBERA5KwpKCmUWB24AzgOHAxWY2\nvM44g4HrgePcfQTwrUTFk0jnjelFXlaajhZEpM1L5JHCeGCJuy9z9/3AVOCcOuN8DbjN3bcDuPum\nBMaTMDmZaVw8vg/PvL+BdXpdp4i0YYlMCj2B2HdXrgn7xRoCDDGz18xspplNSmA8CfXFY/ri7tzz\nxspkhyIi0mxpjY1gZsXA/wA93P2MsAroGHe/4xAtfzBQAvQCXjGzw919R50YJgOTAYqLi5kxY0az\nFlZWVtbsaeMxpmuUe19fypHp68lMs4Qtp7kSXf7WLJXLDqldfpV9RtMmcvcDNsAzwOeAeWF3GvBe\nHNMdAzwX0309cH2dcf4MXB7T/SJw1IHmO3bsWG+u6dOnN3vaeMxavtX7fv8pv/v15QldTnMluvyt\nWSqX3T21y6+yB4DZ3sh2293jqj4qdPeHgeowiVQCVXFMNwsYbGb9zSwDuAiYVmecxwmOEjCzQoLq\npGVxzLtVGtu3E2P7duIvLy9jf6VuZhORtieepLDbzLoADmBmE4CdjU0UJo8pwHPAQuBhd59vZjea\n2dnhaM8BW81sATAd+J67b21GOVoFM2PKyYNYu2Mv/3xHb2YTkban0XMKwHcI9vAHmtlrQBFwfjwz\nd/engafr9Lshpt3D+X8n3oBbu5IhRRzes4A/zljKeWN6kRbV/YEi0nY0usVy97eBk4Bjga8DI9z9\n3UQH1lbVHC2s3LqHp95dn+xwRESaJJ6rj75Yp9cYM8Pd70lQTG3eqYcVM7Q4j1unL+HsUT2IRFrf\nlUgiIvWJp27jqJjmBOAnwNkHmiDVRSLB0cKSTWU8O18PyhORtiOe6qNvxjRfA8YAuYkPrW379OHd\nGVCYwx9eWlJzua2ISKvXnLOgu4H+hzqQ9iYaMa6cOIiF60t5cWGbfHqHiKSgRpOCmT1pZtPC5ilg\nEfDPxIfW9p0zuge9O3fgD9N1tCAibUM8l6T+Jqa9Eljp7roIPw7p0QjfOGkQP/jne7y6ZAsnDC5K\ndkgiIgcUzzmFl2Oa15QQmua8sT3pXpDFH15ckuxQREQa1WBSMLNdZlZaT7PLzEpbMsi2LDMtytdP\nHMBbK7bx5rI2e7O2iKSIBpOCu+e5e349TZ6757dkkG3dReP7UJibyS3/WqxzCyLSqsV99ZGZdTWz\nPjVNIoNqb7LSo1w1cSBvLNvKvxdvSXY4IiINiufqo7PNbDGwHHgZWEHwOG1pgkuO7kPvzh34xTMf\nUF2towURaZ3iOVL4GTAB+NDd+wOfAmYmNKp2KDMtyndPG8qC9aVMm7cu2eGIiNQrnqRQET7OOmJm\nEXefDoxLcFzt0llH9GBEj3x+8/wi9lXG80oKEZGWFU9S2GFmucArwP1m9juCu5qliSIR47ozhrFm\n+17un7kq2eGIiHxCPEnhHGAP8G3gWWApcFYig2rPThhcxPGDCvnDS4spLa9IdjgiIh8TT1L4OtDd\n3Svd/W53/31bfjtaa/D9ScPYvqeC219us28eFZF2Kp6kkAc8b2b/NrMpZlac6KDau8N7FXDWqB78\n7dVlbCotT3Y4IiK14nnMxU/dfQRwFdAdeNnM/pXwyNq57542hKpq55YXFyc7FBGRWk15dPYmYAOw\nFeiamHBSR98uOXz+6L48NGs1SzeXJTscEREgvpvXrjSzGcCLQBfga+5+RKIDSwVTTh5EVlqEXz+7\nKNmhiIgA8R0p9Aa+5e4j3P0n7r4g0UGlisLcTCafOJBn529gph6WJyKtQDznFK5397ktEUwqmnzi\nAHp27MCPn5hPRVV1ssMRkRTXnNdxyiHUISPKDWcNZ9HGXdzzxspkhyMiKU5JoRU4bXgxJUOLuPmF\nD3WJqogklZJCK2Bm/OSsEeyvrObnz3yQ7HBEJIXFc/XRBDObZWZlZrbfzKr05rVDr19hDl8/aQD/\nfGet3tAmIkkTz5HCrcDFwGKgA/BV4LZEBpWqriwZRM+OHbhBJ51FJEniqj5y9yVA1N2r3P3vwKTE\nhpWadNJZRJItnqSwx8wygLlm9isz+3ac00kznDa8mJOGFHGLTjqLSBLEs3G/FIgCUwjeo9AbOC+R\nQaUyM+MnZ49gn046i0gSxHPz2kp33+vupeHD8b4TVidJgvSPOemsO51FpCXFc/XRmWb2jpltM7NS\nM9ulq48S78qSQfTu3IHr/vEue/fr1Z0i0jLiqT66BbgM6OLu+e6e5+75CY4r5XXIiPKr80axYuse\nfv2cHpgnIi0jnqSwGnjf3T3RwcjHHTOwC188pi9/f305by3fluxwRCQFxJMUrgWeNrPrzew7NU2i\nA5PA9ycNo1enDlz76DxVI4lIwsWTFP4b2ANkEbyas6ZplJlNMrNFZrbEzK47wHjnmZmb2bh45ptK\ncjLTVI0kIi0mLY5xerj7yKbO2MyiBHc+nwqsAWaZ2bS672MwszzgauDNpi4jVcRWI51xeDeO6tc5\n2SGJSDsVz5HC02Z2WjPmPR5Y4u7L3H0/MBU4p57xfgb8EtCdWgdQU430vUdUjSQiiWONnT82s11A\nDrAPqAAM8MauQDKz84FJ7v7VsPtS4Gh3nxIzzhjgv9z9vPCVn99199n1zGsyMBmguLh47NSpU+Mv\nYYyysjJyc3ObNW1rsHBrFb+cVc5pfdO45LDMJk/f1st/MFK57JDa5VfZg7JPnDhxjrs3WkXfaPWR\nu8d1/qCpzCwC/Bb4Uhwx3A7cDjBu3DgvKSlp1jJnzJhBc6dtDUqAddH3ue/NlXztjKOaXI3U1st/\nMFK57JDa5VfZS5o0TVzPMDKzI8zsbDP7bE0Tx2RrCR6JUaNX2K9GHjASmGFmK4AJwDSdbD6w684I\nqpGueXgepeUVyQ5HRNqZeO5ovhO4k+B5R2eFzZlxzHsWMNjM+ocP1LsImFYz0N13unuhu/dz937A\nTODs+qqP5CM5mWnccuFo1u7Yy/WPvYduHxGRQymeq48muPvwps7Y3SvNbArwHMED9e509/lmdiMw\n292nHXgO0pCxfTtzzWlD+NWzizhuYCGXHN0n2SGJSDsRT1J4w8yG172UNB7u/jTwdJ1+NzQwbklT\n55/KrjhxIG8s3cpPn5zPmL4dGdZNTx4RkYMXzzmFewgSwyIze9fM3jOzdxMdmBxYJGLcfOFo8juk\nc9X9b7Nnf2WyQxKRdiCepHAHwTsVJvHR+YSzEhmUxKcwN5PfXTiaZVt2c8MT85Mdjoi0A/Ekhc3u\nPs3dl4fvVljp7npXZCtx7KBCvjlxEI/OWcNjb69Jdjgi0sbFc07hHTN7AHiS4AY2ANz9sYRFJU3y\nn58azMzl2/jh4+8zqndHBhal5o06InLw4jlS6ECQDE6jaZekSgtJi0b4/UVHkpkW4ar739ZjMESk\n2eK5o/nylghEDk63gix++7nRfPnuWVz7j3f5/UWjMbNkhyUibUyjScHMsoCvACMIHp8NgLt/OYFx\nSTNMHNaV750+lF89u4hh3fK4auKgZIckIm1MPNVH9wLdgNOBlwkeV7ErkUFJ833jpIGcM7oHv35u\nEc/P35DscESkjYknKQxy9x8Bu939buAzwNGJDUuay8z45XlHMKpXAd9+aC4fbChNdkgi0obEkxRq\nnrq2w8xGAgVA18SFJAcrKz3KXy4dR05mGl+9ezbbdu9Pdkgi0kbEkxRuN7NOwI8IHmi3APhVQqOS\ng9atIIvbvziOTbv28Y375rC/sjrZIYlIG9BoUnD3v7n7dnd/2d0HuHtXd/9zSwQnB2d074786rwj\neHP5Nn76pO54FpHGxXP10Xfq6b0TmOPucw99SHIonXtkTz7YsIs/v7yU6mEZlCQ7IBFp1eKpPhoH\nXAH0DJuvEzwH6a9mdm0CY5ND5HunD+X0EcU8+MF+npi7tvEJRCRlxZMUegFj3P0ad78GGEtwovlE\n4niVpiRfNGL87qIjGdopwjUPz2PGok3JDklEWql4kkJXYp55RHA1UrG7763TX1qxrPQoV4/JYkhx\nHt+4723eWbU92SGJSCsUT1K4H3jTzH5sZj8GXgMeMLMcgiuRpI3ITjfu/vJ4uuZncvlds1iySfcg\nisjHxXP10c+AycCOsLnC3W90993u/vlEByiHVlFeJvd++WjSoxEuveMt1u3Ym+yQRKQViedIAXef\n7e6/C5vZiQ5KEqtPl2zuvnw8ZeWVXHrHm2zXzW0iEoorKUj7M7xHPn+7bBxrtu/lsr+/xc49FY1P\nJCLtnpJCCjt6QBf+9IUxfLB+F5+/YyY79uiIQSTVKSmkuJOHFfOXS8fy4cYyLvnrm3pOkkiKU1IQ\nJg7ryl+/OI6lm8u45K8z2VqmK41FUpWSggBw0pAi7rjsKFZs3c3Ff53J5l1KDCKpSElBah0/uJA7\nv3QUq7ft5aLb32BTaXmyQxKRFqakIB9z7MBC7rr8KNbvLOei22eyZvueZIckIi1ISUE+4egBXbjn\ny+PZUraPz/7xdeav25nskESkhSgpSL3G9evMo984lrSIceFfZvLvxZuTHZKItAAlBWnQkOI8Hrvy\nOHp16sDlf5/FY2+vSXZIIpJgSgpyQN0Ksnj4imMY378z33l4HrdNX4K7JzssEUkQJQVpVH5WOndd\nPp5zRvfg188t4kdPvE9VtRKDSHvU6Os4RQAy0iLc/LnRdC/owJ9fXsqqbXv5/UWj6ZidkezQROQQ\n0pGCxC0SMa47Yxg//+zhvLF0C2ff+hoL15cmOywROYSUFKTJLh7fh6mTj6G8oorP/vF1npy3Ltkh\nicghoqQgzTK2byee+ubxjOiRzzcffIefP72QyqrqZIclIgcpoUnBzCaZ2SIzW2Jm19Uz/DtmtsDM\n3jWzF82sbyLjkUOra34WD3xtAl+Y0Ie/vLKML/19ll7YI9LGJSwpmFkUuA04AxgOXGxmw+uM9g4w\nzt2PAB4FfpWoeCQxMtIi3HTu4fzyvMN5a/k2zvzDq7y1fFuywxKRZkrkkcJ4YIm7L3P3/cBU4JzY\nEdx9urvXPFxnJtArgfFIAl14VB8eueIY0qLGRbe/wW9f+FDVSSJtkCXqRiQzOx+Y5O5fDbsvBY52\n9ykNjH/Z1rymAAAQ0UlEQVQrsMHdb6pn2GRgMkBxcfHYqVOnNiumsrIycnNzmzVte9AS5d9b6dy3\nYD+vratkUMcIXz8ik6Ls5J+60m+fuuVX2YOyT5w4cY67j2t0IndPSAOcD/wtpvtS4NYGxv0CwZFC\nZmPzHTt2rDfX9OnTmz1te9CS5X9i7lofecOzPvKGZ/3xd9a02HIbot9+erJDSBqVPQDM9ji23Ync\nhVsL9I7p7hX2+xgzOwX4L+Bsd9ebXdqJs0f14OmrT2BotzyunjqXbz80l517KpIdlog0IpFJYRYw\n2Mz6m1kGcBEwLXYEMzsS+AtBQtiUwFgkCXp3zmbq5Al865TBTJu3jlNufpln31+f7LBE5AASlhTc\nvRKYAjwHLAQedvf5ZnajmZ0djvZrIBd4xMzmmtm0BmYnbVRaNMK3ThnCE1cdR1FuJlfc9zZX3DtH\nb3UTaaUS+uwjd38aeLpOvxti2k9J5PKl9RjZs4AnphzHX/+9jFv+tZjXf7uFH35mOBeM64WZJTs8\nEQkl/7IQSRnp0QhXlgzimatPYFi3fK79x7t84Y43WbFld7JDE5GQkoK0uIFFuUydPIGbzh3JvNU7\nOfXml/n5Mwsp21eZ7NBEUp6SgiRFJGJ8YUJfXrrmJM4Z3ZO/vLyMib+ZwaNz1lCtdzWIJI2SgiRV\n1/wsfnPBKB6/6jh6duzAdx+Zx2f/9DpzV+9IdmgiKUlJQVqF0b078tg3juV/LxjF2h17Ofe21/jO\nQ3NZvW1P4xOLyCGjN69JqxGJGOeN7cXpI7tx2/Ql3Pnqcp58dx0Xj+/DlImD6JqflewQRdo9HSlI\nq5Obmcb3Jw3jlWsncuFRvXngzVWc+Ovp/PyZhXo0t0iCKSlIq1Wcn8VN5x7Oi9ecxBkju3P7K8s4\n8VfT+f2Li9m5V4/MEEkEJQVp9fp2yeHmC0fz7NUncszALvz2hQ85/hcv8YtnPmDTLt0ZLXIo6ZyC\ntBlDu+Vx+xfHMX/dTv40Yym3v7KUO19bzgVjezH5xAH07ZKT7BBF2jwlBWlzRvQo4NZLxrBiy27+\n8soyHpm9hgffWsWZR/TgaycM4PBeBckOUaTNUlKQNqtfYQ4//+zhfPuUwdzx6nLum7mSafPWMaZP\nRy47th9njOxORppqSEWaQv8YafO65mdx/acP440ffIofnzWc7XsquHrqXI79xUv89oUP2agnsorE\nTUcK0m7kZ6Vz+XH9ueyYfvx7yRbufn0Ff3hpMX+cvoQju0awHps5flAh0YieyirSECUFaXciEeOk\nIUWcNKSIlVt3c+8bK5n65nIuu/Mtuhdkcd6YXpw/thf9CnViWqQuJQVp1/p2yeGHZw7n6OyNVBQN\n4+HZq/njjCXcOn0J4/t35oKxvZg0sht5WenJDlWkVVBSkJSQHjFOPbw7nz68Oxt2lvOPt9fwyOzV\nfO/Rd/mvx9/n5KFdOWtUD04e1pUOGdFkhyuSNEoKknK6FWRx1cRBXFkykLdXbefJeev5v/fW8+z8\nDWRnRDnlsGLOGtWDE4cUkpmmBCGpRUlBUpaZMbZvZ8b27cyPzhzOm8u38uS89Tz7/nqmzVtHTkaU\nk4YWcerwYk4eWkxBtqqYpP1TUhABohHj2IGFHDuwkBvPGcFrS7bw/IKN/GvBRp5+bwPRiDG+X2dO\nHV7MqcOL6d05O9khiySEkoJIHenRCCVDu1IytCs3nTOSd9fu5IUFG3h+/kZufGoBNz61gAGFOZw4\npIgThxQyYUAXsjP0V5L2QWuyyAFEIsbo3h0Z3bsj3zt9GCu27Gb6ok288uFmps5axV2vryAjGmFc\nv06cOKSIYwd2YXj3fNKiui9U2iYlBZEm6FeYw+WF/bn8uP6UV1Qxe8V2Xlm8mVc+3MwvnvkAgLzM\nNI7q35kJAzozYYCShLQtSgoizZSVHuX4wYUcP7iQH3z6MDbtKufNZduYuWwrM5dt5aUPNgFBkhjT\ntxNj+nRibN9OjOpdoPsipNVSUhA5RLrmZXHWqB6cNaoHAJtKy5m5PEgSb6/czi0vfog7RAyGFOcx\ntm8njuzTiSN6FTCwKFeP35BWQUlBJEG65mdx9qgenB0midLyCuau2sHbq7YzZ+V2ps1dx/1vrgIg\nOyPKiB75HN6zI0f0KmBkzwL6F+YoUUiLU1IQaSH5WenhFUtFAFRVO8s2l/He2p28u2Yn763dyQNv\nreTO16oB6JAeZUi3PIZ3z2NYt3wO657PsO555KvqSRJISUEkSaIRY3BxHoOL8/jsmF4AVFZVs2Rz\nGe+u2ckH63excH0pz7y/gQffWl07XY+CLAYV5zG4a27QFOcyqChPN9fJIaGkINKKpEUjDOuWz7Bu\n+bX93J2NpftYuKGUhetLWbyxjMWbdnH/m1spr6iuHa8oL5P+hTkMKMyhf9gMKMrRjXbSJEoKIq2c\nmdGtIItuBVlMHNq1tn91tbNm+14Wb9rF4k1lLNlUxootu3lhwUa27t5fO17EoFOmMfjDN+jTOZs+\nnbPpHX726pRNYW4GZjp3IQElBZE2KhIx+nTJpk+XbD51WPHHhu3cW8GKLbtZvmU3y7bsZvbC5eyv\ncmYs2symXfs+Nm5GWoSeHTvQs2MHenTMomfHbHp0zKJ7QQe6FWTSraADuZnaVKQK/dIi7VBBh3RG\n9e7IqN4dAZiRvo6SkmMB2Lu/ijXb97By6x7W7tgbNNv3smbHXqYv2szmOkkDgnstiguy6F6QRVFe\nJl3zaj7DJj/ozsmI6qijjVNSEEkxHTKitSe461NeUcWGneVsKC1nY2k563eWB907y1lfWs6yzbvZ\ntKuciir/xLSZaREKczPpkpsRfOZk0CU3k8456XTKzqBzzkdNp5wM8jLTlERaGSUFEfmYrPQo/Qpz\nDvi6Undnx54KNu3ax+Zd+9i0q5wtZfvYWrafzeHnpl3lLFhXytbd++pNIBBcgVXQIZ2OHdIpyA4+\nO2ZnUNAhnfwO6eRnpZHfIT3ozkonv0Ma+Vnp5GWlkZuZpseHJICSgog0mZnRKdzbH9qt/iOOGu5O\n2b5Ktu+uYNue/WzfvZ9tYbNzbwXb9+xnx94Kdu6pYHPZPj7cWEZpeQW7yisbjaNDejRIEGGSyM1M\nI6f2M0pOZhqb1+1nadpysjOiZGdEyclIIzszSnZGGtkZUTqkR+kQDstKixJJ8RsGE5oUzGwS8Dsg\nCvzN3X9RZ3gmcA8wFtgKXOjuKxIZk4i0LDMjLyudvKx0+nSJ//LYqmqnrLyS0vIKdu6toLS8gtK9\nlewKE0bZvqC9bF8lpeWV7N5XSVl5Jdt272H3/kp276uirLyS/VXVPLZ4QdzLrUkSWWkRssJE0SFM\nHlnpETLTomSmR8hKj5KZFnxmhf0yohEyw3Ey0iJkpkWCz2jwWdPUDE+PGhnRCOnh8LSIJb06LWFJ\nwcyiwG3AqcAaYJaZTXP32F/nK8B2dx9kZhcBvwQuTFRMItJ2RCNGQXZQrdT7IObzr5emc9SE49lT\nESSKPfs/+txbUcWe/VXs3R9+VlSxN+xfXlHN3ooq9lVUheNVsm13Nfsqg2Gxnw1VjzVHkCSM9LQI\naZEIGVEjLez3rVOG1D5bK1ESeaQwHlji7ssAzGwqcA4QmxTOAX4Stj8K3Gpm5u6H7hsWkZSWVpNc\nSNwd31XVzv7KIEHsq6yubS+vqGZ/VdAd9Avbq6qoqPTaYRVVQRMMcyprumPaK6qcji1w17olavtr\nZucDk9z9q2H3pcDR7j4lZpz3w3HWhN1Lw3G21JnXZGAyQHFx8dipU6c2K6aysjJyc3ObNW17kMrl\nT+WyQ2qXX2UPyj5x4sQ57j6usWnaxIlmd78duB1g3LhxXlJS0qz5zJgxg+ZO2x6kcvlTueyQ2uVX\n2UuaNE0ir+daCx+rCuwV9qt3HDNLAwoITjiLiEgSJDIpzAIGm1l/M8sALgKm1RlnGnBZ2H4+8JLO\nJ4iIJE/Cqo/cvdLMpgDPEVySeqe7zzezG4HZ7j4NuAO418yWANsIEoeIiCRJQs8puPvTwNN1+t0Q\n014OXJDIGEREJH66R1xERGopKYiISC0lBRERqZWwm9cSxcw2AyubOXkhsKXRsdqvVC5/KpcdUrv8\nKnugr7sXNTZBm0sKB8PMZsdzR197lcrlT+WyQ2qXX2VvWtlVfSQiIrWUFEREpFaqJYXbkx1AkqVy\n+VO57JDa5VfZmyClzimIiMiBpdqRgoiIHICSgoiI1EqZpGBmk8xskZktMbPrkh1PopnZnWa2KXyR\nUU2/zmb2gpktDj87JTPGRDGz3mY23cwWmNl8M7s67N/uy29mWWb2lpnNC8v+07B/fzN7M1z/Hwqf\nXNwumVnUzN4xs6fC7lQq+woze8/M5prZ7LBfk9b7lEgKMe+LPgMYDlxsZsOTG1XC3QVMqtPvOuBF\ndx8MvBh2t0eVwDXuPhyYAFwV/t6pUP59wMnuPgoYDUwyswkE7z+/2d0HAdsJ3o/eXl0NLIzpTqWy\nA0x099Ex9yc0ab1PiaRAzPui3X0/UPO+6HbL3V8heBx5rHOAu8P2u4FzWzSoFuLu69397bB9F8EG\noicpUH4PlIWd6WHjwMkE70GHdlp2ADPrBXwG+FvYbaRI2Q+gSet9qiSFnsDqmO41Yb9UU+zu68P2\nDUBxMoNpCWbWDzgSeJMUKX9YfTIX2AS8ACwFdrh7ZThKe17/bwGuBarD7i6kTtkh2AF43szmhO+2\nhyau923iHc1y6Lm7m1m7vh7ZzHKBfwDfcvfSYKcx0J7L7+5VwGgz6wj8ExiW5JBahJmdCWxy9zlm\nVpLseJLkeHdfa2ZdgRfM7IPYgfGs96lypBDP+6JTwUYz6w4Qfm5KcjwJY2bpBAnhfnd/LOydMuUH\ncPcdwHTgGKBj+B50aL/r/3HA2Wa2gqCK+GTgd6RG2QFw97Xh5yaCHYLxNHG9T5WkEM/7olNB7Dux\nLwOeSGIsCRPWI98BLHT338YMavflN7Oi8AgBM+sAnEpwTmU6wXvQoZ2W3d2vd/de7t6P4D/+krt/\nnhQoO4CZ5ZhZXk07cBrwPk1c71PmjmYz+zRBfWPN+6L/O8khJZSZPQiUEDw6dyPwY+Bx4GGgD8Hj\nxz/n7nVPRrd5ZnY88G/gPT6qW/4BwXmFdl1+MzuC4GRilGCn72F3v9HMBhDsPXcG3gG+4O77khdp\nYoXVR9919zNTpexhOf8ZdqYBD7j7f5tZF5qw3qdMUhARkcalSvWRiIjEQUlBRERqKSmIiEgtJQUR\nEamlpCAiIrWUFETqMLOOZnZlTHcPM3v0QNM0czklZnbsoZ6vyMFQUhD5pI5AbVJw93Xufv4Bxm+u\nEkBJQVoV3acgUoeZ1TxFdxHBA+VuA55y95Fm9iWCp0zmAIOB3wAZwKUEj63+tLtvM7OB4XRFwB7g\na+7+Qcwy+gEzgSpgM/BNd/93S5RP5ED0QDyRT7oOGOnuo6F2Ax5rJMGTV7OAJcD33f1IM7sZ+CLB\nnfO3A1e4+2IzOxr4I8GzeABw9xVm9megzN1/k+DyiMRNSUGk6aaH72nYZWY7gSfD/u8BR4RPZz0W\neCTmyayZLR+mSNMpKYg0Xexzc6pjuqsJ/lMRgmf4j27pwEQOlk40i3zSLiCvuRO7eymw3MwugOCp\nrWY26lAvRyQRlBRE6nD3rcBrZva+mf26mbP5PPAVM5sHzKf+178+CfxH+JL1E5q5HJFDSlcfiYhI\nLR0piIhILSUFERGppaQgIiK1lBRERKSWkoKIiNRSUhARkVpKCiIiUuv/AVJagYxcbuQpAAAAAElF\nTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f3bc1095eb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def gamma_values(gamma=0.9, n=50):\n",
    "    return [gamma**i for i in range(1, n)]\n",
    "\n",
    "plot(gamma_values())\n",
    "title('gamma value in regard with t. gamma=0.9, n=50')\n",
    "xlabel('time t')\n",
    "ylabel('gamma value')\n",
    "grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Optimal action-value Function and Bellman Equation \n",
    "\n",
    "기본적으로 Reinforcement Learning의 목표는 expected return을 최대치로 하는 policy를 학습하는 것이며,<br>\n",
    "아래의 공식과 같은 <span style=\"color:red\">**Optimal action-value function $ Q^{*}(s, a) $**</span> 을 사용합니다. <br>\n",
    "즉 어떤 특정 state $ s_t $ 에서 특정 action $ a_t $ 를 취했을때 얻게되는 **expected return** 을 나타냅니다.\n",
    "\n",
    "$$ \\begin{align}\n",
    "Q^{*} (s, a) &= \\max_{\\pi} \\mathbb{E} \\left[ r_t + \\gamma r_{t+1} + \\gamma^2 r_{t+2} + ... \\big|\\ s_t =s, a_t = a, \\pi \\right] \\\\\n",
    "&= \\max_{\\pi} \\mathbb{E} \\left[ R_t \\big| \\ s_t=s, a_t =a, \\pi \\right] \n",
    "\\end{align} $$\n",
    "\n",
    "Optimal Action-Value Function은  <span style=\"color:red\">**Bellman Equation**</span>을 따릅니다. <br>\n",
    "만약 optimal value $ Q^{*}(s^{\\prime}, a^{\\prime}) $ (여기서 $ s^{\\prime} $는 next time-step의 state이고, $ a^{\\prime} $는 모든 가능한 actions들을 말함) 의 값을 알고 있다면, <br>\n",
    "expected value $ r + \\gamma Q^{*}(s^{\\prime}, a^{\\prime}) $를 maximize 하는 action $ a^{\\prime} $을 선택하는 것에 기초를 두고 있습니다.\n",
    "\n",
    "$$ Q^{*} (s, a) = \\mathbb{E}_{s^{\\prime} \\sim \\varepsilon} \\left[ r + \\gamma \\max_{a^{\\prime}} Q^{*} \\left(s^{\\prime}, a^{\\prime}\\right) \\  \\big| \\ s, a \\right] $$\n",
    "\n",
    "\n",
    "> <span style=\"color:#999999\">위의 bellman equation은 reinforcement learning에서 기본적인 방향틀이지 정말로 위의 공식을 사용해서 구한다는 뜻은 아닙니다. <br>\n",
    "> 실제 위의 공식을 \"따라서\" 구현한 알고리즘이 value iteration, policy iteration, Q-Learning, DQN 등등이 있습니다. <br>\n",
    "> 즉 일종의 개념이라고 생각하면 됩니다. </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Nonlinear Approximator and Loss Function\n",
    "\n",
    "일반적으로 iterative update를 통한 (value iteration 또는 policy iteration알고리즘등등) action-value function을 \n",
    "구하는 것은 일반화 시키지 못하며, 연산도가 매우 높아 복잡한 문제일수록 풀수가 없습니다.\n",
    "따라서 **function approximator**를 사용하여 action-value function을 측정합니다.\n",
    "\n",
    "$$ Q(s, a; \\theta) \\approx Q^{*} (s, a) $$\n",
    "\n",
    "보통 Reinforcement Learning에서는 Linear function approximator를 사용하지만, <br>\n",
    "Deep Q Learning에서는 Nonlinear function approximator인 뉴럴네트워크를 사용합니다.<br>\n",
    "위의 공식에서 neural network function approximator로서 weights $ \\theta $가 Q-network로 사용되었습니다.<br>\n",
    "즉 Q-network는 parameters $ \\theta $를 조정해가면서 학습이 진행됩니다.\n",
    "\n",
    "Loss function은 mean-squared error를 사용합니다.<br>\n",
    "이때 Bellman Equation의 **optimal target values** $ r + \\gamma \\max_{a^{\\prime}} Q^{*}\\left(s^{\\prime}, a^{\\prime}\\right) $ 이 부분을 <br>\n",
    "approximate target values $ y = r + \\gamma \\max_{a^{\\prime}} Q \\left( s^{\\prime}, a^{\\prime}; \\theta^{(-)}_i \\right) $ 으로 대체시켜줍니다. \n",
    "\n",
    "따라서 최종 <span style=\"color:red\">**Loss Function**</span>은 다음과 같습니다.\n",
    "\n",
    "$$ L_i(\\theta_i) = \\mathbb{E}_{s, a, r} \\left[ \\left(\n",
    "r + \\gamma \\max_{a^{\\prime}} Q\\big( s^{\\prime}, a^{\\prime} ; \\theta^{-}_i \\big)\n",
    "- Q\\big( s, a; \\theta_{i} \\big)\n",
    " \\right)^2  \\right] $$\n",
    " \n",
    "Target network $ Q^{-} $ 는 training중에만 사용되며 Loss를 구할때 사용이 됩니다.<br>\n",
    "즉 2개의 networks를 사용하는 것이며 target network는 주기적 또는 느리게 업데이트가 됩니다.<br>\n",
    "1개의 network사용시 예를들어... $ Q(s, a) $ 그리고 $  Q(s^{\\prime}, a) $ 는 1step밖에 차이가 나지 않으며 이는 서로 매우 가까움을 의미합니다. 마치 고양이가 자기 자신의 꼬리를 잡으려고 계속 돌듯이.. target의 값이 지속적으로 shift되는 상황은 instability, oscillation 또는 divergence로 이어질수 있습니다. \n",
    "\n",
    "Target network의 장점은 stable한 learning을 만들어주지만, 단점으로는 학습속도가 매우 저하되며 <br>\n",
    "이유는 Q network에서 변경된 값은 바로 적용되지 않으며, target network 상당한 시간 이후에 업데이트가 되기 때문 입니다.<br>\n",
    "target network가 업데이트 되는 시점은 보통 10000번 정도의 step이후에 업데이가 되게 됩니다. \n",
    "\n",
    "> 복잡한 문제일수록 target network의 update시점을 늦게 잡습니다. (예를 들어 50000) <br>\n",
    "> 문제가 복잡하지 않다면 update시점을 좀 빠르게 잡아도 됩니다.\n",
    "\n",
    "<span style=\"color:#999999\">\n",
    "구글 딥마인드에서 내놓은 DQN Nature 논문에서는 target network를 바로 이전 step의 network로 지칭하고 있습니다.<br>\n",
    "추후 딥마인드팀에서는 이 방법이 문제가 있다는것을 알았고, 추가적인 논문에서 target network의 update를 늦은 시점에서 하는 것을 발표합니다.\n",
    "</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Frame-skipping Technique \n",
    "\n",
    "간단히 말해 Agent는 모든 frame마다 action을 선택하는 것이 아니라, $ k $ 번째마다 action을 선택하는 형태입니다. <br>\n",
    "그리고 마지막에 했었던 action을 skipped frames에서 선택을 합니다. <br>\n",
    "이것을 통해서 많은 computation을 줄일수 있습니다.<br>\n",
    "구글 딥마인드 Nature논문에서는 $ k=4 $ 값으로 실행했습니다.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Error Clipping\n",
    "\n",
    "Mean Squared Error (MSE) loss function을 위해서 사용하였습니다. \n",
    "\n",
    "$$ \\text{MSE} = \\frac{1}{N} \\sum^n_{i=1} \\left( \\hat{y}_i - y_i \\right)^2 $$\n",
    "\n",
    "문제는 제곱을 하게 되는 부분에서 MSE는 너무나 큰 값의 loss를 만들어내게 됩니다. <br>\n",
    "이는 DQN에서는 unstable한 learning이 될 수 있습니다. <br>\n",
    "따라서 다른 대안으로 Mean Absolute Error(MAE) 를 사용할 수 있습니다.\n",
    "\n",
    "$$ \\text{MAE} = \\frac{1}{N} \\sum^n_{i=1} \\big|\\ \\hat{y}_i - y_i \\ \\big| $$\n",
    "\n",
    "MSE가 값이 커질때 너무 커지는 단점이 있었다면, MAE의 경우 loss값이 작을때 너무 작아져서 학습이 느려지는 단점이 있습니다.<br>\n",
    "MSE와 MAE의 2가지의 단점을 극복한 loss function 이 있습니다.\n",
    "\n",
    "**Huber loss** function을 사용하면 MSE와 MAE의 장점을 섞어서 사용할수 있습니다.<br>\n",
    "즉 loss값이 작을때는 MSE를 사용한것과 같으며, loss값이 크면 MAE를 사용한것과 같은 효과를 낼 수 있습니다.\n",
    "\n",
    "$$ \\text{HL} = \\frac{1}{N} \\sum^N_{i=1} L_\\delta(a) $$\n",
    "\n",
    "여기서 $ L_\\delta(a) $ 는 다음과 같습니다.\n",
    "\n",
    "$$  L_\\delta(a)=\\begin{cases}\n",
    "\\frac{1}{2}a^2                & |a|\\leq \\delta,\\\\\n",
    "\\delta(|a|-\\frac{1}{2}\\delta) &\\text{otherwise}\n",
    "\\end{cases} $$\n",
    "\n",
    "delta $ \\delta $ 는 값이 커지면 커질수록 y값의 증가량이 점점 커짐. \n",
    "\n",
    "> 실제 딥마인드팀의 DQN 논문에서는 error clipping을 1과 -1사이로 제한을 하였습니다. <br>\n",
    "> 이렇게 제한을 걸면 여러게임에서 동일한 learning rate를 사용할수 있도록 만들어 줍니다. <br>\n",
    "> 즉 delta값이 제한됨으로서 다양한 게임에서 완만하게 학습될 수 있도록 해줍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7faa154f1b38>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd4VFX6wPHvTSEN0ivpPSEhoYQmIKC0taArawERWERU\nVHTVFUVx4Scqu5ZFbIiKCCIWFGyrFAWR3oQQQjppJCG9kT5zfn9MSEIJCckkd5Kcz/PwaG5ueSeE\nd86c8h5FCIEkSZLU9RmpHYAkSZKkHzKhS5IkdRMyoUuSJHUTMqFLkiR1EzKhS5IkdRMyoUuSJHUT\nMqFLkiR1EzKhS5IkdRMyoUuSJHUTJp35MEdHR+Hj49OZj5QkSeryjh49mi+EcGrpvE5N6D4+Phw5\ncqQzHylJktTlKYqS1przZJeLJElSNyETuiRJUjchE7okSVI30al96JJhqK2tJTMzk6qqKrVDkQyU\nubk5Hh4emJqaqh2KdA1kQu+BMjMz6dOnDz4+PiiKonY4koERQlBQUEBmZia+vr5qhyNdgxa7XBRF\nWaMoSq6iKDGXHH9MUZQ4RVFOKYryn44LUdK3qqoqHBwcZDKXrkhRFBwcHOQnuC6oNX3oa4HJTQ8o\nijIOuA2IFEKEAa/rPzSpI8lkLl2N/P3omlpM6EKI3UDhJYcfBpYLIarrz8ltzcNKqkuuOUBJkqSe\n7O0/3271uW2d5RIEjFYU5aCiKL8rijKkuRMVRZmnKMoRRVGOZJdno9Fq2vhIqTtJTU0lPDy81eev\nXbuWRx99tAMjkiTDE18Yz+ro1a0+v60J3QSwB4YD/wS+Upr5jCaEWC2EiBJCRGnQsDtzdxsfKUlt\np9HIhoTU9Xwe9znmxuatPr+tCT0T+FboHAK0gGNLF5kambIhbkMbHyl1NxqNhgceeICwsDAmTpxI\nZWUlY8eObSgPkZ+fT9PaPxkZGYwdO5bAwECWLl3acPyzzz5j6NChDBgwgAcffLAheffu3ZunnnqK\nyMhI9u/f36mvTZLaq7iqmJ9SfuIW/1tafU1bpy1uAcYBOxVFCQJ6AfktXWRvbs/B7IMkFyfjb+vf\nxkdL+rT0h1PEZpXq9Z79+lrzr1vDWjwvMTGRjRs38uGHH3LXXXfxzTffXPX8Q4cOERMTg6WlJUOG\nDOHmm2/GysqKL7/8kr1792Jqasr8+fPZsGEDM2fO5Pz58wwbNow33nhDXy9NkjrNN4nfUK2pZnrI\ndJawpFXXtJjQFUXZCIwFHBVFyQT+BawB1tRPZawBZgkhREv3sjO3w8jIiM9Pf87iEYtbFaDUffn6\n+jJgwAAABg8eTGpq6lXPnzBhAg4ODgDccccd7NmzBxMTE44ePcqQIbphnMrKSpydnQEwNjZm6tSp\nHfcCJKmD1Gnr+CL+C4a5DiPQLrDV17WY0IUQ05r51oxWP6WesWLMTX438UPKDywYtAAbM5trvYWk\nZ61pSXcUMzOzhv83NjamsrISExMTtFotwGXzoC8dplEUBSEEs2bN4tVXX73s/ubm5hgbG3dA5JLU\nsXZl7CLnfA7PDX3umq7r9Fou00OnU1lXyZakLZ39aKkL8PHx4ejRowBs2rTpou9t376dwsJCKisr\n2bJlCyNHjuTGG29k06ZN5ObqZs4WFhaSltaqSqOSZLA2nN6Ae293xniMuabrOj2hh9iHMMh5EBvj\nNsopjNJlnn76ad5//30GDhxIfv7FwzJDhw5l6tSpREREMHXqVKKioujXrx/Lli1j4sSJREREMGHC\nBLKzs1WKXpLaL74wniPnjnBP8D0YG13bJ0ylFV3fehMVFSWOHDnCttRtPPX7U6wct5JxXuM67fmS\nzunTpwkNDVU7DMnAyd8Tdfxr37/4+czPbP/b9oZuaUVRjgoholq6VpXyuTd43YCLpQsbTsspjJIk\nSRdcmKp4s9/NbRpjVCWhmxiZcE/IPRzMOUhCUYIaIUiSJBmcrxO+bpiq2BaqbXBxZ9CdmBub81ns\nZ2qFIEmSZDBqNbV8EfcFI9xGXNNUxaZUS+g2ZjZM8Z/CTyk/UVBZoFYYkiRJBmFr2lZyK3O5r999\nbb6HqlvQ3dvvXmq0NXyV8JWaYUiSJKlKCMH62PX4WPsw0n1km++jakL3s/FjlPsovoz7khpNjZqh\nSJIkqebP3D+JLYjlvn73YaS0PS2rvkn0ff3uo6CqgJ/P/Kx2KFIPN3v27MsWM7VF7969WzxnxYoV\nVFRUtPtZUvewPnY91r2sudX/1nbdR/WEPsJtBAG2AayPXU9nzomXJDV1VEKvq6u76tetvU7qPJll\nmfyW8Rt3Bt2JhYlFu+6lekJXFIX7+t1HfJFudZTU/aWmphISEsLs2bMJCgri3nvvZceOHYwcOZLA\nwEAOHToEwPnz55kzZw5Dhw5l4MCBfPfddw3Xjx49mkGDBjFo0CD27dsHwK5duxg7dix/+9vfCAkJ\n4d57771iI+HDDz9kyJAhREZGMnXq1IsS644dO4iKiiIoKIgff/wRgFOnTjWU542IiCAxMRGAN998\nk/DwcMLDw1mxYsVlz9m1axe33NJY+vTRRx9l7dq1rFy5kqysLMaNG8e4cbqFddu2bWPEiBEMGjSI\nO++8k/Ly8svul5yczOTJkxk8eDCjR48mLi4O0H2yeOihhxg2bBjPPPMMS5Ys4b777mPkyJHcd999\nVFVV8fe//53+/fszcOBAdu7cCeg2DZkyZQo33HADN9544zX+LUr68nnc5xhhxLSQ5spmtV5by+fq\n1U2+N7Hi6ArWxa5jiGuzmx9JHeHnZyHnpH7v6dof/rL8qqckJSXx9ddfs2bNGoYMGcLnn3/Onj17\n+P7773nllVfYsmULL7/8MjfccANr1qyhuLiYoUOHMn78eJydndm+fTvm5uYkJiYybdq0hhrqf/75\nJ6dOnaJv376MHDmSvXv3MmrUqIuefccdd/DAAw8A8MILL/Dxxx/z2GOPAbo3i0OHDpGcnMy4ceNI\nSkpi1apVPP7449x7773U1NSg0Wg4evQon3zyCQcPHkQIwbBhwxgzZgwDBw5s8cezYMEC3nzzTXbu\n3ImjoyP5+fksW7aMHTt2YGVlxb///W/efPNNXnzxxYuumzdvHqtWrSIwMJCDBw8yf/58fvvtNwAy\nMzPZt28fxsbGLFmyhNjYWPbs2YOFhQVvvPEGiqJw8uRJ4uLimDhxIgkJuvUfx44dIzo6Gnt7+1b8\nxUr6Vl5TzreJ3zLRZyIuVi7tvp9BJHRzE3PuCr6L1dGrSStNw9vaW+2QpA7m6+tL//79AQgLC+PG\nG29EURT69+/fUEZ327ZtfP/997z+um4P8qqqKtLT0+nbty+PPvoox48fx9jYuCE5ga7ei4eHBwAD\nBgwgNTX1soQeExPDCy+8QHFxMeXl5UyaNKnhe3fddRdGRkYEBgbi5+dHXFwcI0aM4OWXXyYzM5M7\n7riDwMBA9uzZw1//+lesrKwA3ZvEH3/80aqEfqkDBw4QGxvLyJG62Q01NTWMGDHionPKy8vZt28f\nd955Z8Ox6urqhv+/8847L6osOWXKFCwsdB/f9+zZ0/CGFRISgre3d8PPbMKECTKZq2hz0mbO155v\n11TFpgwioQPcE3IPa2LWsD52PS8Mf0HtcHqOFlrSHaVp6VwjI6OGr42MjBr6c4UQfPPNNwQHB190\n7ZIlS3BxceHEiRNotVrMzc2veF9jY+Mr9g3Pnj2bLVu2EBkZydq1a9m1a1fD965Uonf69OkMGzaM\nn376iZtuuokPPvigVa+xaSlguLwc8AVCCCZMmMDGjRubvZdWq8XW1pbjx49f8fsX3lia+7o5rT1P\n0r86bR2fxX7GIOdBhDu2fn/dq2mxD11RlDWKouTWb2Zx6feeUhRFKIrS4vZzLXG0cORW/1v5Luk7\niqqK2ns7qRuYNGkSb7/9dkM/+J9//glASUkJbm5uGBkZsX79+mveL7SsrAw3Nzdqa2vZsOHiekJf\nf/01Wq2W5ORkUlJSCA4OJiUlBT8/PxYsWMBtt91GdHQ0o0ePZsuWLVRUVHD+/Hk2b97M6NGjL7qX\nt7c3sbGxVFdXU1xczK+//trwvT59+lBWVgbA8OHD2bt3L0lJSYBu7KDppw4Aa2trfH19+frrrwHd\nm8CJEyda9XpHjx7d8DoTEhJIT0+/7E1S6nw70naQdT6L2WGz9XbP1gyKrgUmX3pQURRPYCKQrq9g\nZvabSZWmii/jv9TXLaUubPHixdTW1hIREUFYWBiLF+t2uZo/fz6ffvopkZGRxMXFXXMr86WXXmLY\nsGGMHDmSkJCQi77n5eXF0KFD+ctf/sKqVaswNzfnq6++Ijw8nAEDBhATE8PMmTMZNGgQs2fPZujQ\noQwbNoy5c+de1t3i6enJXXfdRXh4OHfddddF3583bx6TJ09m3LhxODk5sXbtWqZNm0ZERAQjRoxo\nGPBsasOGDXz88cdERkYSFhbWMEjckvnz56PVaunfvz933303a9euveiTjNT5hBB8cuoTfKx9GON5\nbTXPr6ZV5XMVRfEBfhRChDc5tgl4CfgOiBJCtLin6IXyuVfzyK+PEJMfw7a/bcPMWP7SdQRZFlVq\nDfl70nEO5xxmztY5LB6+mLuC72rx/A4tn6soym3AWSFE6z7zXYNZ/WZRWFXID8k/6PvWkiRJBuHT\nU59iZ2bHFP8per3vNSd0RVEsgUXAiy2dW3/+PEVRjiiKciQvL6/F84e4DiHUPpR1sevQCm2L50uS\nJHUlKSUp/J75O9NCpmFuYt7yBdegLS10f8AXOKEoSirgARxTFMX1SicLIVYLIaKEEFFOTk4t3lxR\nFGaFzeJMyRn+yPyjDeFJkiQZrnWn1mFmbMbdIXfr/d7XnNCFECeFEM5CCB8hhA+QCQwSQuToK6iJ\nPhNxtXJl7am1+rqlJEmS6vIr8/kh+Qem+E/B3lz/8/9bM21xI7AfCFYUJVNRlPv1HsUlTI1MmRE6\ngyPnjhCTf9lsSUmSpC7pi7gvqNXW6m0h0aVaTOhCiGlCCDchhKkQwkMI8fEl3/dpzQyXazU1cCp9\nTPvwScwn+r61JElSp6uorWBj3EbGeo7F18a3Q56henGu5vTu1Zu7gu9iR/oO0kv1NtVdkgzW2LFj\naWlar9R1fZv4LaU1pcwJn9NhzzDYhA4wo98MTBQT2ZcudVmdVZZWls01bLXaWtbFrmOQ8yAGOA/o\nsOcYdEJ3tHBkSsAUvkv6jvxKvffqSCpRu3zuypUr6devHxEREdxzzz0AFBQUMHHiRMLCwpg7dy7e\n3t7k5+t+526//XYGDx5MWFgYq1evbrhP040sNm3axOzZs4HLy9k29zoqKyu55557CA0N5a9//SuV\nlZVX/HkdPXqUMWPGMHjwYCZNmkR2djaga9E/8cQTREVF8dZbb1323MLCQm6//XYiIiIYPnw40dHR\nAJeV15U63i9nfiH7fDb39+/YIUiDKc7VnFn9ZvFNwjd8fvpzFgxaoHY43c6/D/2buMLLl5m3R4h9\nCAuHLrzqOWqWz12+fDlnzpzBzMyM4uJiAJYuXcqoUaN48cUX+emnn/j448ahojVr1mBvb09lZSVD\nhgxh6tSpODg4XPX1NS1nu2jRoiu+jg8++ABLS0tOnz5NdHQ0gwYNuuw+tbW1PPbYY3z33Xc4OTnx\n5Zdf8vzzz7NmzRpAV5nxwmufPXv2Rc997LHHGDhwIFu2bOG3335j5syZDcW9mpbXlTqWEII1MWsI\nsA1glPuoli9oB4NP6D42Poz3Hs8X8V9wf//7sTKV1eG6AzXL50ZERHDvvfdy++23c/vttwOwe/du\nvv32WwBuvvlm7OzsGs5fuXIlmzdvBiAjI4PExMQWE3rTcrbNvY7du3ezYMGChpgiIiIuu098fDwx\nMTFMmDABAI1Gg5ubW8P377774rnMTZ+7Z88evvnmGwBuuOEGCgoKKC0tBS4uryt1rD/O/kFScRIv\nj3q5XfuFtobBJ3SAOeFz2J62nU0Jm5gVNkvtcLqVllrSHUXN8rk//fQTu3fv5ocffuDll1/m5Mnm\nN/jYtWsXO3bsYP/+/VhaWjJ27NiGMrhNS+1eWhq3acGw5l5HawghCAsLY//+/Vf8viyba/jWxKzB\n1cqVv/j+pcOfZdB96BeEO4Yz1HUo62LXUaupVTscqZN0RPlcrVZLRkYG48aN49///jclJSWUl5dz\n/fXX8/nnnwPw888/U1RU1PAsOzs7LC0tiYuL48CBAw33cnFx4fTp02i12oYW/LW8jqbPjImJaejj\nbio4OJi8vLyGhF5bW8upU6da9Vqbls3dtWsXjo6OWFtbt+paST9O5J3g6LmjzOw3E1Mj0w5/XpdI\n6AB/D/87uRW5/Jjyo9qhSJ2kI8rnajQaZsyY0bC/5oIFC7C1teVf//oXu3fvJiwsjG+//RYvLy8A\nJk+eTF1dHaGhoTz77LMMHz684V7Lly/nlltu4brrrruoG6S1r+Phhx+mvLyc0NBQXnzxRQYPHnzZ\ntb169WLTpk0sXLiQyMhIBgwY0DAI3JIlS5Zw9OhRIiIiePbZZ/n0009b/XOS9OOTmE+w7mXN1MCp\nnfK8VpXP1ZfWlM9tjhCCu368i6q6KrbctgVjI+OWL5KuSJZFbZmPjw9HjhzB0bHde7d0WfL3pH2S\ni5O5/bvbeTDiQR4d+Gi77tWh5XPVoCgKc/vPJbU0lV/Tf235AkmSJBV9fPJjLEwsmBE6o9Oe2WUS\nOsB4r/H4WPvw0cmPrji/WJL0JTU1tUe3zqX2ySjL4H9n/sedQXdia27bac/tUgnd2MiYOeFzOF14\nmj1n96gdjiRJ0hWtjVmLkWLEzH4zO/W5XSqhA9zidwuuVq58dPIjtUORJEm6TF5FHpuTNnNbwG24\nWLl06rO7XEI3NTZldthsjuUe4+i5o2qHI0mSdJF1sevQCA1zwjquCFdzulxCB11pXXtzez48+aHa\noUiSJDUoqS7hy/gv+YvvX/C09uz053fJhG5uYs59/e5j79m9xBbEqh2OpKIlS5Y0LKlvLVmmVuoo\nn5/+nMq6Su4P7/B9gK6oNTsWrVEUJVdRlJgmx15TFCVOUZRoRVE2K4rSecO49e4Ovps+pn34MFq2\n0qXOI8vNSs05X3uez05/xljPsQTaBaoSQ2ta6GuByZcc2w6ECyEigATgOT3H1aI+vfowPXQ6O9J3\nkFiU2NmPl9qhNeVztVotgYGB5OXlAbol+wEBAQ1fNxUbG8vYsWPx8/Nj5cqVDc8IDw9vOOf1119n\nyZIlDV+vX7+eAQMGEB4e3mK53rVr1zJlyhRuuOEGbrzxxo76sUhd3Ma4jZTWlPJgxIOqxdBicS4h\nxG5FUXwuObatyZcHgL/pN6zWua/ffayPXc/q6NW8NuY1NULo8nJeeYXq0/otn2sWGoLrokVXPac1\n5XNnzJjBhg0beOKJJ9ixYweRkZE4OTlddq+4uDh27txJWVkZwcHBPPzwwy3GWFFRwfHjx9m9ezdz\n5swhJiam2XK9AMeOHSM6Ohp7e/1v7Ct1fRW1Faw7tY6R7iMJdwxv+YIOoo8+9DnAz3q4zzWzMbNh\nWsg0tqZuJaU4RY0QpDa6UD7XyMio2fK5c+bMYd26dYCuJvnf//73K97r5ptvxszMDEdHR5ydnTl3\n7lyLz582bRqgK5BVWlpKcXEx27ZtY/ny5QwYMKChqmJ6um77wwkTJshkLjXrq/ivKKou4qGIh1SN\no13lcxVFeR6oAzZc5Zx5wDygoeCRPs0Mm8nncZ/z4ckPeXX0q3q/f3fXUku6o7SmfK6npycuLi78\n9ttvHDp0qKFy4NXudaFkromJCVqttuH4peVtm5a+vfB1c2VuDx48KMvNSs2qrKtk7am1DHcb3qHb\ny7VGm1voiqLMBm4B7hVXWYcvhFgthIgSQkRd6eNye9mb23N38N3878z/SCtN0/v9JXXNnTuXGTNm\nXLRxQ2u4uLiQm5tLQUEB1dXV/PjjxVU6v/zyS0C3CYSNjQ02NjbNlrmVpKv5JuEbCqoKVO07v6BN\nCV1RlMnAM8AUIUSFfkO6drPCZmFqZCpXj3ZDU6ZMoby8vNnuluaYmpry4osvMnToUCZMmEBISMhF\n3zc3N2fgwIE89NBDDdvNNVfmVpKaU62p5pOYT4hyiSLKtcViiB1PCHHVP8BGIBuoBTKB+4EkIAM4\nXv9nVUv3EUIwePBg0VGWH1wuIj+NFBmlGR32jO4iNjZW7RBa7fDhw2LUqFFqh9EjdaXfE7VsPL1R\nhK8NF/uz9nfoc4AjohU5tsUWuhBimhDCTQhhKoTwEEJ8LIQIEEJ4CiEG1P9RdyQA3QYYxoqxbKV3\nI8uXL2fq1Km8+qocG5EMT62mlo9jPmaA0wCGuQ5TOxygi64UvRJnS2fuCLyD75K+42z5WbXDkfTg\n2WefJS0t7bJNniXJEGxO2kzO+RweinzoskF2tXSbhA4wt/9cjBQjVkevVjsUgydkPXnpKuTvx9XV\naGpYHb2aSKdIrut7ndrhNOhWCd3FyoU7g+/ku6TvyCjNUDscg2Vubk5BQYH8RytdkRCCgoICzM3N\n1Q7FYH2T+A3nKs7xyIBHDKZ1Du2ch26I7g+/n00Jm/gg+gOWjVqmdjgGycPDg8zMzCsuo5ck0L3p\ne3h4qB2GQarWVPNR9EcMch7EcLfhLV/QibpdQneydOLOoDvZGLeRByIewNvaW+2QDI6pqSm+vr5q\nhyFJXdKmhE3kVuby6uhXDap1Dt2sy+WC+/vfj6mRKR+c+EDtUCRJ6kaq6qr46ORHRLlEMdRtqNrh\nXKZbJnRHC0fuDr6bn878xJmSM2qHI0lSN/FV/FfkV+Yzf8B8tUO5om6Z0EE3L93M2IxVJ1apHYok\nSd1AZV0la2LWMMx1GENch6gdzhV124TuYOHAPSH38POZn2W9dEmS2u2LuC8oqCow2NY5dOOEDjAn\nbA6Wppa8e/xdtUORJKkLK6sp4+OYjxnpPpJBLoPUDqdZ3Tqh25rbMqvfLH5N/5VT+afUDkeSpC5q\nfex6SqpLeGzgY2qHclXdOqGDblcjWzNb3v7zbbVDkSSpCyqqKuLTU58ywXsCYQ5haodzVd0+offu\n1Zs54XPYm7WXIzlyp3dJkq7Nmpg1VNZV8siAR9QOpUXdPqED3BNyD04WTrz959tyubskSa2WW5HL\nxriN3Op/K/62/mqH06IekdAtTCyYFzGPY7nH2Ju1V+1wJEnqIlZHr0aj1fBQpOoVwlulRyR0gKmB\nU3Hv7c7KYytlK12SpBZllGXwTcI33BF4B559PNUOp1VaTOiKoqxRFCVXUZSYJsfsFUXZrihKYv1/\n7To2zPYzNTbl4ciHOV14mq1pW9UOR5IkA/fe8fcwNjJmXsQ8tUNptda00NcCky859izwqxAiEPi1\n/muDd4vfLQTYBvD2sbep1daqHY4kSQYqvjCen1J+4t7Qe3GxclE7nFZrzRZ0u4HCSw7fBnxa//+f\nArfrOa4OYWxkzOODHie9LJ3NiZvVDkeSJAO14tiKhhlyXUlb+9BdhBDZ9f+fAzT7FqYoyjxFUY4o\ninLEEOpvj/EYwyDnQbx3/D0qaivUDkeSJANzKPsQe87u4YH+D2BjZqN2ONek3YOi9TtSNzvKKIRY\nLYSIEkJEOTk5tfdx7aYoCv8Y/A8KqgpYH7te7XAkSTIgQghWHFuBi6UL00KmqR3ONWtrQj+nKIob\nQP1/c/UXUscb4DyAcZ7j+OTUJxRVFakdjiRJBmJH+g5O5p/kkQGPYG7S9bbga2tC/x6YVf//s4Dv\n9BNO53l80ONU1lXKDaUlSQKgVlvLymMr8bfx51b/W9UOp01aM21xI7AfCFYUJVNRlPuB5cAERVES\ngfH1X3cp/rb+3B5wO1/Ef0FmWaba4UiSpLLNiZtJLU3l8UGPY2LUNXfnbM0sl2lCCDchhKkQwkMI\n8bEQokAIcaMQIlAIMV4IceksmC7h4ciHMVaMZeEuSerhKmoreO/4ewx0HshYz7Fqh9NmPWal6JW4\nWrkys99M/nfmf8Tkx7R8gSRJ3dInpz6hoKqAp6KeMriNn69Fj07ooNtQ2t7cntePvC5LAkhSD3Tu\n/DnWxqxlks8kIp0i1Q6nXXp8QrcyteKRAY9w9NxRfsv4Te1wJEnqZO8cfweN0PDEoCfUDqXdenxC\nB7gj8A58bXz579H/ypIAktSDxBfG813Sd0wPmY5HHw+1w2k3mdABEyMTnhr8FGmlaXwd/7Xa4UiS\n1AmEELx+5HX69OrDAxEPqB2OXsiEXu96j+sZ5jqM90+8T1lNmdrhSJLUwfZm7eVA9gEeinyoyy3x\nb45M6PUUReGpqKcoqS7hw+gP1Q5HkqQOVKet440jb+DZx5N7gu9ROxy9kQm9iVCHUKb4T+Gz05+R\nUZqhdjiSJHWQTQmbSCpO4snBT2JqbKp2OHojE/olLqwSe/3I62qHIklSByipLuHd4+8yxHUIN3rd\nqHY4eiUT+iWcLJ14oP8D/JbxGwezD6odjiRJerbqxCpKa0pZOGRhl15EdCUyoV/BzLCZuPd259+H\n/02dtk7tcCRJ0pOUkhS+iPuCOwLvINg+WO1w9E4m9CswMzbjycFPkliUyLeJ36odjiRJevLa4dcw\nNzHn0QGPqh1Kh5AJvRkTvCcw2GUw7/z5DqU1pWqHI0lSO/2R+Qd7zu7hociHcLBwUDucDiETejMU\nRWHhkIUUVxfz/vH31Q5HkqR2qNXW8tqR1/C29mZ6yHS1w+kwMqFfRahDKHcE3sHGuI0kFSWpHY4k\nSW30+enPOVNyhqejnu5W0xQv1a6ErijKPxRFOaUoSoyiKBsVRel6eza14PFBj2NlasXyQ8tlNUZJ\n6oLyKvJ47/h7XO9xfZeudd4abU7oiqK4AwuAKCFEOGAMdJ8lV/XszO14bOBjHMw5yLa0bWqHI0nS\nNXrz6JvUamtZOGSh2qF0uPZ2uZgAFoqimACWQFb7QzI8dwbdSYh9CK8dfo2K2gq1w5EkqZWOnjvK\njyk/MjtsNl7WXmqH0+HanNCFEGeB14F0IBsoEUJ0yyassZExi4Yt4lzFOT48Keu8SFJXUKet45WD\nr+Bq5cqjD3wKAAAgAElEQVTc/nPVDqdTtKfLxQ64DfAF+gJWiqLMuMJ58xRFOaIoypG8vLy2R6qy\ngc4DudXvVtaeWktqSara4UiS1IKv4r8ioSiBf0b9E0tTS7XD6RTt6XIZD5wRQuQJIWqBb4HrLj1J\nCLFaCBElhIhycnJqx+PU92TUk5gZm7H8sBwglSRDVlBZwDvH32G423AmeE9QO5xO056Eng4MVxTF\nUtEVRLgROK2fsAyTo4Uj8yPns/fsXnak71A7HEmSmvHm0TeprKvkuaHPdbt6LVfTnj70g8Am4Bhw\nsv5eq/UUl8GaHjqdYLtglh9azvna82qHI0nSJQ7nHOb75O/5e9jf8bP1UzucTtWuWS5CiH8JIUKE\nEOFCiPuEENX6CsxQmRiZsHjEYvIq8nj3+LtqhyNJUhO1mlqWHViGe2/3brOt3LWQK0XbINIpkr8F\n/Y0NpzdwuqBb9zJJUpfyaeynpJSksGjYIixMLNQOp9PJhN5Gjw96HFszW5YdWIZWaNUOR5J6vIyy\nDFadWMUE7wlc73G92uGoQib0NrIxs+HpqKeJzo9mU8ImtcORpB5NCMGrB1/FWDHmmSHPqB2OamRC\nb4db/G5hiOsQVhxbQX5lvtrhSFKPtSN9B3+c/YNHBjyCq5Wr2uGoRib0dlAUhcXDF1NVV8XyQ8vV\nDkeSeqTSmlJeOfgKIfYhTA/tvqVxW0Mm9HbytfHlwYgH2Zq6lV0Zu9QOR5J6nDePvElRVRFLr1uK\niZGJ2uGoSiZ0PZgTPocA2wCWHVhGeU252uFIUo9xOOcw3yR+w3397qOfQz+1w1GdTOh6YGpsypLr\nlpBbkctbx95SOxxJ6hGqNdUs3b8Uj94ezB8wX+1wDIJM6HoS6RTJ9NDpfBn/Jcdzj6sdjiR1ex+c\n+IC00jReHPFij5xzfiUyoevRYwMfw8XKhX/t+xc1mhq1w5Gkbiu+MJ5PYj5hiv8URvQdoXY4BkMm\ndD2yMrVi8fDFpJSksDq625e1kSRV1GnreHHfi1ibWfPPqH+qHY5BkQldz673uJ5b/W7lo5MfEVsQ\nq3Y4ktTtrD21ltiCWJ4f9jy25rZqh2NQZELvAAuHLsTO3I7FexdTq6lVOxxJ6jaSipJ47/h7TPSe\nyESfiWqHY3BkQu8ANmY2vDj8RRKKEuSWdZKkJ3XaOhbvXUxv094sGrZI7XAMkkzoHWSc1zhu9ruZ\nD6M/JK4wTu1wJKnL+/TUp8QUxLBo+CIcLBzUDscgyYTegZ4b+hw2Zja8sOcFarWy60WS2iq5OJl3\nj7/LBO8JTPKepHY4BqtdCV1RFFtFUTYpihKnKMppRVHk/KEmbMxseHHEi8QXxfNhtOx6kaS2uNDV\nYmVqxaJhi3rUlnLXqr0t9LeAX4QQIUAk3XxP0ba4wesGbvG7hdXRqzmZd1LtcCSpy/nw5IeczD/J\n88Ofx9HCUe1wDFqbE7qiKDbA9cDHAEKIGiFEsb4C606eG/YcjhaOLNqziMq6SrXDkaQu41T+KT44\n8QE3+d7EZJ/Jaodj8NrTQvcF8oBPFEX5U1GUjxRFsdJTXN2KdS9rlo1aRmppKv89+l+1w5GkLqGq\nrorn9jyHg4WDnNXSSu1J6CbAIOB9IcRA4Dzw7KUnKYoyT1GUI4qiHMnLy2vH47q24W7DmRE6g41x\nG9mXtU/tcCTJ4K04toIzJWdYNnIZNmY2aofTJbQnoWcCmUKIg/Vfb0KX4C8ihFgthIgSQkQ5OTm1\n43Fd3+ODHsfXxpfFexdTUl2idjiSZLD2Ze1jw+kNTA+ZLmu1XIM2J3QhRA6QoShKcP2hGwG51v0q\nzE3MeXXUqxRWFrLswDKEEGqHJEkGp6S6hMV7F+Nj7cMTg59QO5wupb2zXB4DNiiKEg0MAF5pf0jd\nW5hjGA8PeJhfUn/h++Tv1Q5HkgyKEIIl+5ZQWFXI8uuXy7K416hd+zUJIY4DUXqKpce4P/x+9mft\n5+WDLzPAeQDe1t5qhyRJBmFT4iZ2pO/gqcFPEeYQpnY4XY5cKaoCYyNjXh39KqZGpizcvVAW8JIk\nIKU4hf8c+g8j3EYwM2ym2uF0STKhq8TVypWl1y3lVMEp3jn+jtrhSJKqajQ1PLP7GSxMLHh51MsY\nKTI1tYX8qalovPd4/hb0Nz6J+YQD2QfUDkeSVPPfo/8lviiel0a+hJNlz54N1x4yoavsmSHP4Gvj\ny3N/PEd+Zb7a4UhSp/s943c+O/0Z00OmM8ZzjNrhdGkyoavMwsSC18a8RllNGc/98RwarUbtkCSp\n02SXZ7NozyJC7UN5MupJtcPp8mRCNwBBdkEsGraIA9kHWH1S7kUq9Qy1mlqe3v00GqHh9TGvY2Zs\npnZIXZ5M6AbirwF/5Va/W3n/+PsczD7Y8gWS1MWtOLaC6Lxoll63FC9rL7XD6RZkQjcQiqLwwvAX\n8LXxZeHuhbI/XerWfkv/jXWx67gn+B4m+cgNK/RFJnQDYmlqyetjXud87XkW7l5InbZO7ZAkSe/O\nlp/lhb0vEGofyj+H/FPtcLoVmdANTKBdIC8Mf4FDOYd4+8+31Q5HkvSqqq6Kf+z8Bwh4Y8wb9DLu\npXZI3Uq7lv5LHeO2gNuIzotmTcwawh3DmeA9Qe2QJKndhBC8dOAlThee5p0b3sHT2lPtkLod2UI3\nUAuHLiTCKYIX9rxASnGK2uFIUrt9Ff8V3yd/z8ORD8v55h1EJnQD1cu4F2+OeRNzE3Me3/k45TXl\naockSW12PPc4yw8vZ7T7aB6KfEjtcLotmdANmIuVC6+PeZ2Msgye3/O8rJ8udUn5lfk8uetJXC1d\neXX0q7JOSweSP1kDN8R1CE9FPcVvGb+xKnqV2uFI0jWp0dTwxM4nKKspY8W4FXIruQ4mB0W7gBmh\nM4grjOO94+8RYBsgB0mlLkEIwf/t/z9O5J3g9TGvE2wf3PJFUru0u4WuKIqxoih/Koryoz4Cki6n\nKAovjniRSKdInt/zPKcLTqsdkiS1aF3sOr5L/o6HIx+Wi4c6iT66XB4HZIbpYGbGZqwYtwLrXtYs\n2LlAriSVDNruzN28ceQNJnhPkIOgnahdCV1RFA/gZuCjVl2gqWnP43o8RwtH3r7hbYqrinli5xPU\nyJ+nZICSi5NZuHshwfbBLBu5TA6CtodWC5lHWn16e3/SK4BnAG1zJyiKMk9RlCOKohzh3Cl4dzhs\n/xek7QONXNp+rUIdQnl51MucyDvB4r2L5cwXyaAUVBbwyK+P0Mu4FyvHrcTS1FLtkLqeqlI4tQU2\nPwxvBMFHN7b60jYPiiqKcguQK4Q4qijK2ObOE0KsBlYDRIV4CXo7wf53YO8KMLeFgPEQNBkCbgRL\n+7aG06NM9JnIgtIFrPxzJR59PHhs4GNqhyRJVNZVsuC3BRRUFrBm0hrcerupHVLXkZ8EiVsh4RdI\n2w/aWjC3gYAJEDQJlt7dqtu0Z5bLSGCKoig3AeaAtaIonwkhZjR7RW9nmPUDVJVA8k5I2AqJ2yBm\nEyhG4DlMF3zgJHAOBUVpR3jd29z+czlbfpbV0avx6O3BXwP/qnZIUg+m0Wp47o/nOJl/kv+O+y/9\nnfqrHZJhq6uB9H2QsE2XxAuTdcedQmHEfF0j12MoGF9I0a1L6Io+PrLXt9CfFkLccrXzoqKixJEj\nl/QHabWQdUyX3BN+gZxo3XEbLwiaqHthPqPB1LzdcXY3tdpaHv31UQ5mH+S9G9/jOvfr1A5J6qH+\nc/g/rI9dz8IhC5nRr/k2XY9WnqdrwCb8omvQ1pSBsRn4jtblucCJYOd9xUsVRTkqhIhq6RHqJ/RL\nlWbVv+htkLITaivA1BJ8x+ha70GTwLpvu2PuLsprypn5y0yyyrP4dPKncq6v1Ok2nN7A8kPLuTf0\nXp4d+qza4RgOIXQN1AuN1bPHAAF93HTJO2gy+I2BXlYt3qpTE3prtSqhN1VbBWl7Gn8gxem64679\ndd0yQZPBfRAYGXdMwF1Ezvkc7v3pXgSC9Tetx723u9ohST3EL6m/8MzvzzDGcwwrxq7AuIf/W6Tm\nPKTsauxOLssGFHAf3NggdY245u7k7pHQmxIC8uIaf1DpB0BowNJB924XOFE3sGreM5cWJxYlMuuX\nWdib2/Pp5E9xsHBQOySpm9uftZ/5v84nwjGCVRNWYWFioXZI6ihKbewLT90Dmmro1Qf8xzV2pfR2\natcjul9Cv1RlEST9qkvwSdt1XxuZgNeI+nfCyeAQ0KMGVv/M/ZN52+bha+PLmklr6N2rt9ohSd1U\nTH4Mc7bOwaOPB59M+qRn1WjR1EHGQV0CT9yma2iCLt9cSOBeI8BEf5t3dP+E3pSmDs4e0f2AE7ZC\nbqzuuJ2v7gccNBG8R4JJ999VfHfmbhb8toAolyjeHf+u3Eld0ruUkhRm/zwbS1NL1v1lHc6WzmqH\n1PEqCiFphy7HJO3QzdQzMgXv65o0IP077PE9K6Ffqji9vt99K5zZXf8RqDf4jW18B+3j0vFxqOSH\n5B9YtGcR473G89qY1zAxkjXYJP3ILs9m1i+zqNZUs+4v6/C2vvKsjC5PCF3D8EIeyTwEQgtWTvUD\nmpPAbxyYW3dKOD07oTdVU6FL6hda72VZuuN9B9a33ieBayQYda/lyZ/Ffsa/D/+bm3xv4pVRr8jB\nKqndcitymf3LbIqrivl40seEOoSqHZJ+1VbCmT8au1JKMnTH3SLrG4KTdHlDhVzR2oTe/ZtuvSwh\neLLujxBwLqY+uW+DXcth16vQ2xUCJ9RPIxoLZl2/73lGvxlUaap469hb9DLuxdLrlsqaGlKb5Vfm\nM3fbXAoqC1g9cXX3SeYlmY0TLVJ+h7pKMLXSDWhe/09da9y666x47f4JvSlF0U15dO2v+8s6n9/Y\nLxb7Pfy5Hox7gc+o+mmRk8DeV+2o22xu/7nUaGp4/8T7mBmb8fyw51F60CCxpB/FVcXM2z6PnPM5\nvD/+fSKdItUOqe20Gl2xq8T6rpRzMbrjtt4waKbu37zPqC473tb9u1xaS1ML6fsb363zE3THHYMb\nV6x6DgNjU3XjvEZCCFYcW8GamDXMCJ3BM0OekUldarWS6hIe2PYAycXJvDv+XYa7DVc7pGtXWQzJ\nv+o+lSdug8pCUIybzIibBI5BBj0jTna5XCtjU/C9Xvdn0stQmNI4IHJgFex7G8xsdHPdgybpiuZY\nGf5cb0VReGKQrtTuZ6c/Qyu0PDv0WZnUpRYVVRXx4PYHSSpOYsW4FV0nmQsB+YmNfeFp+3RrVizs\nLl6zYmGndqR6JxN6c+z9YPjDuj/VZfWrv36BxO1w6ltAAY8hjVOWXMIM9h1eURSeGfIMRooR62LX\nUaOtYfHwxbJPXWpWQWUBD2x/gLSSNFbesJJR7qPUDunq6qp1i3ou1EopStUddwmHkY/XF7uK6var\nymWXy7XSaiH7eOMvTtafuuPW7o31GXyv1w3GGhghBCv/XMlHJz/iNv/bWHrdUjn7RbpMXkUec7fN\nJas8i7dvfNtwW+ZlOfX/Drfqil3VngcT88a6T4ETwdZT7Sj1Qna5dBQjI139GPdBMPbZ+l+q7brk\nHv0VHP1E90vlM7qxf87WS+2oAV1LfcHABfQy6sV7J96jVlvLslHLMDXqWuMCUsfJOZ/D3G1zya3I\n5b3x7zHEdYjaITXSaiH7z8Zl9tnHdcetPSDynvoBzdEG2ZjqLLKFrk/Nfexz7tfYNeMxxCA+9n10\n8iPeOvYWYzzG8NqY13puHQ6pQUpJCg9uf5DymnLeH/8+A5wHqB2SrrszeWf9rJRtcD4XUMBzaOPe\nCQbc3akvcmGR2q42MHNhFxKVB2a+iPuCVw6+wkDngbx949tY9+qcVW+S4TmZd5L5v87HSDFi1fhV\n6s4zL0hu7EpJ3aPbvcfMBgJugKC/6HY56wITEvRJJnRDU1kMyb/pflETt0FFQf3UqeGNfe9OwZ3e\n0vgl9Ree++M5fG18+WD8BzhZtq8qnNT17MvaxxM7n8De3J7VE1bjZd3JXYRNpwwnbIWCRN1xx+DG\nbssuOGVYn2RCN2Raja7Y/YVyBOdO6o7bejUuMfYZ1Wm7NO3P2s/jOx/H3tyeVeNX4WPj0ynPldT3\n85mfWbRnEX42fqwav6rz3tDP5zeOPSX/BtWl9Yv6RjcOaHbhRX361uEJXVEUT2Ad4AIIYLUQ4q2r\nXSMTejNKMpvs0rSrfvmxpa74z4Vf7g5efnwq/xTzf51PnbaOFeNWGNZgmKR3Qgg+PPkhb//5NoNd\nBrPyhpUd2+V2UdmNrbrVmgjo7dLYF+43tluU3egInZHQ3QA3IcQxRVH6AEeB24UQsc1dIxN6K1wo\nEHRhaXLTAkEXdmnqoAJBGWUZPPLrI2SUZbD0uqVM8Z+i92dI6qvV1LJk/xK+T/6eW/xuYel1S+ll\nrL/a3Q1qKuDM742rr0vP6o73HdRk957uVxivI3R6l4uiKN8B7wghtjd3jkzo10iFEp6lNaU8ufNJ\nDuYc5MGIB3lkwCNyVWk3UlJdwj92/YPDOYeZHzmfhyIf0u/fb9PS1al/QF2VrnT1hd17AiZ069LV\nHaWzN4n2AXYD4UKI0ku+Nw+YB+Dl5TU4LS2t3c/rsRqK7Nfv0tRBRfZrNbW8dOAlNidtZoL3BJaN\nXIalac+d29tdJBcn8/jOx8kqz2LpdUu51f/W9t9UUweZhxtnc13YXMber3HvAe+Ret29p6fR1Gkw\nMTXpnISuKEpv4HfgZSHEt1c7N9TSSqwYMpZqV3eMPL2wCvDDuV8g3hHB9LaVU+auyYVtsC50zVzY\nBsvev3GXJq/r2vwPSQjB2lNrWXFsBX42frw17q3On/0g6c2OtB08v+d5zE3M+e/Y/zLIZVDbb1ZR\nqNv+MXGrbmCzqli3/aP3dY3dgo4B+gu+B9BqteRnniMjOo7C+GQqU86gZGZgmXcWh+JcBp6O6fiE\nriiKKfAjsFUI8WZL5wc4uoiVg0Zik5+NfUXxRd8rsrCh2MGNWjcPTL296RPgh2tYEF7hAZhZdM5s\njy7too1q/wBNDZhZX/xRtw0b1e7L2sczu59BK7QsH72c6z2u13/sUofRaDW8c/wdPjr5ERGOEbw5\n9k1crK6xy6Nhg/b6fQQyDui6/iwdG7v+/Mf12A3ar0VJfhHp0QnknU7gfPIZREYa5ueysC/Kwaq2\nquG8WsWYAhsnyp36ou3rwe2r/9Phg6IK8ClQKIR4ojXXNO1DLy8uJSMmkXOxiZQnpaBJT8Ms5yx2\nRTlYV59vuEaDQkEfR8ocXdH09cTM1wfbIH/6hgfjHuSDsYn6qy4NTnX5xYNRZdmAAu6DmwxGRbR6\nzntmWSb/2PUP4gvjeXjAw8zrP0/WgOkCCqsKWfTHIvZm7WVq4FQWDVvU+sHP2irdop6EX3Qt8eJ0\n3XHX/rpWePBfdIObckDzMhXlFWTEJJBzKpHy5BTq0tLolXMW24JsbKvKGs7TolBoZUepoxt1fT3o\n5eODTZA/bv2C8AjxxdSs8e+qM2a5jAL+AE4C2vrDi4QQ/2vumtYOihZm55EeHU9+XCKVKamIzHQs\nzp3FofgcFnU1DefVGJlQYOPMeee+aN09sPD1xSEkAI/+ITh5uWIkf9l0rauc6MaBqrNHAQF93JoM\nrI6FXlZXvU1lXSX/t///+DHlR4a5DuPV0a/KRUgG7HDOYZ7d/SxF1UU8O/RZ7gq+q+WLSrOa7N6z\nC2or6qfPjm0sO2vj3sGRdw211TVkxp0hOzaBkoRkalJTMcnKxDo/G/vzRRjRmFdLzPtQbO9Ktas7\nxt4+9PH3wTU8GM/wICx7t25sqlsuLNJqteSmniXjZDyFcUlUn0lFycrAKjcLx5I8TIWm4dwKEzMK\n7VypdHFH8fTC0s8Xp9AAvCJDsHWy18fL6ZrK83QDqgm/6GpkVJeCsRn4jm7cpcnuyhv/CiHYkrSF\nVw+9ioWJBS+Petnwy6r2MBqthg+iP+CD6A/w6uPFa2NeI8Q+5Mona7WQdWGB2y+QU7/AzcarcZC9\nExe4GZor5puzGfTOy8Lh0nxjak6BnQtVzu4oXt5Y+fviFBKAZ0SwXvJNt0zoV1NbXcPZ+FSyYxMp\nTkhqeMfsk5+NQ3nh1d8xA/xwCQ3EK6L175jdQl2Nbsn1hWJiBUm6406hjbs0eQwF44uLciYXJ/P0\n70+TVJzE7LDZPDbwsY6Zxyxdk+zybBbtWcSRc0e41e9WXhj+wuWzk6pKdSszL7TEK/JBMQLP4Y1/\n504h3b7YVVNFOQWknTh9hR6BXCzqqhvOa+wRcEe4e2Dh54t9sD+ekaE4erh0aI9Aj0voV1NRXkHm\nqURyTiVSmpiMJr0VfVpuHpj6eGMT5E/fsODL+rS6pYLkxpV8aXtBWwfmtrpiSEGTdP+11LU2quqq\n+M/h//B1wtcE2Abw8qiX6efQT+UX0DNd+OT0n8P/QSM0PD/seW4LuK3xhPykxr7wtH2X/L1O1hWJ\ns+zen1rLi0tJP5lA7umk1o3ZObnpxux8vLELDsQtLFDVMTuZ0FupOK+QjJOJl406OxTlYHnpqLO1\nI+ed+qJx98Tc1xu7IF1/vau/R/frr68qhZSdjS2583m6lpzH0MaBVed+7D77B0v2LaGoqoh5kfOY\n23+urK/eifIq8li6fym/Z/7OYJfBvDTyJTwtXHRvyBc+eRWm6E52Cm38u7vCJ6+urqaqmvRTSeSc\nSqA0MYXatFRMc862MKvOHdP6T+mGPKtOJvR2amleqJm2ruHcSpNeFNo4U+HsDvX99Q6hAXhHhGDv\n1g0GDrVa3c5MF1p52Sd0x208IXAiJb7X80reHv6XtpVQ+1CWXLdEttY7mBCCH1J+4D+H/0NVXRWP\nh93PvaI3RolbIXkX1JQ1jo1cWODTzNhIV6Kp05CVlEbWyQSKEpKoTk3D+GwGffKzcCgrwFhoG84t\n72VJob0rVa7uGHt4YeXvi3NYEF79g+lj17XWvciE3oEu+6U6k4pxVuYVf6nKzKwotHOl2rUvxp7e\n9K5fTOXVP6jrLqYqzW6sV52yq37rLwu2+wzkZQop0lQxPXQ6jw58FCvTq8+eka5dSkkKy/a/xOFz\nR4g0c+Slsjp8z0brvtmnb2NfuO/1Lc5eMkRarZaCs+fIiI6noKExlY5lbhYOxecuakxVGfeiwM6F\nCqe+4OGBpb8/DqEBeIUH4+DurOKr0C+Z0FVSXVlFRmxSfX99CrXpaZhmZWBTkHPZx75CS1tKHFzr\nF1P5YB3gh2t4IJ79DPNj3xXVVkHanvppkb9QWprBSjtbvurTBycTC54Nnc34AfNQutnHezVUnc/n\nw33/x5qsXVhotTxZWMgdZRUYeUQ1Vix07d9lBjRLC0tIPxFH7ulEXXdnehpmuVnYF+bQu7ay4bw6\nxYh8ayfOO7qh8fDC3NcXuyA/3MODcAvw6n7dnVcgE7oBujAwc+6Urr9ek5GGec5Z7Apz6FNT0XCe\nRjGioI8DZY5uDYup7IICVB+YaZEQkBcPiVuJTvie/6vLJL5XL4bWaPin7SBCQu8A/xvAwlbtSLuO\nwjOIhK38nLCJFbVZZJsYc0tFDU/bD8Yh+FYInABWjmpH2ayqikrST+oW2ZQlJVOXnk6v7ExsC3Ow\nrbyo7BMFVnaUOPbVTUjw9sYm0B+38CA8Q/26/4SEFsiE3sUUnM0lPSaegtNJVJw5A+npWOZl4VB0\nDnNN42KqaiMTCmydqXBqMnUqJADPiJAOnzp1rerO5/HVgdd4/+x2SrS13FZ+nsdKynF2H9bYonQM\n7DItyk6hqdXV6KlfZv9n2Rles7fjpLkZoSY2/LPfbIZEzDKo3XvqauvIPJ1C1ql4ShJTLlpkY1de\niPGlU4btXOqnDOtKfLj0C8arfyCWfbpe91BnkQm9m9BqtZw7k0nGiTiKElKoOnMGo6xMeueexaE0\n/6LFDedNzSm0c6XKpX5xg58PTqFBeEUEYeOo3t6lpTWlfHjiAzac/hwTBPdWGzMrOwVbrRbsfBuL\niXmPBBMz1eJUzfmC+iqav0Dyr1BVQryZBe/39eZXKnA2s2dB1JPc6n8rRoo6b9gXFtlknkygMD6J\nqjOpKGfTdYv6SvMw1V66qM+FShcPFE/dYKRj/SIbO+eetReovsiE3gM0XX5cHJ9EbVqarmVUkIP9\nJYupii2sKbZ3pcbVHRNvb/oE+OPSL+Calh+3V0ZZBm8fe5tfUn/Bwtice+3CmVVYhE3avsa62X5j\nG2dldNe62ULAuVONlTIzD9fXuXcmwX8kq0yq2F50it6mvZkZNpNZ/WZ1WvniotwC0k/EkX86kfMp\nqXBhkU3RuWYW2ciyG51BJvQermmBoLKkFDRpqZhlZ2JbdA6bSxdT9bavX0zl3lAgqG9YMB6hfpiY\n6n8wM6koiQ+iP2Br6lYsTS25K+AOppt74Zp24JKdbQY2liNwG9C1C0HVVsKZ3Y0VC0szdcfdBkDQ\nJI47+7Mu9wDb03fQ27Q3M/rNYEboDGzM9F/B8HxJOekn48k9naT73ahfZGNbmINNdXnDeRoUCuvH\nci4Uj7ILDsAtLAj3IJ8O+d2QrkwmdKlZxXmFZETHkxubSEXKGd1S55yz2Befu3wxVX0JT+HuiZmP\nD/bB/nhEBOPi2/7FVIlFiayOXs22tG0YYcQEnwnMDL2PcA1Ndmk6TMPek4ETdAnefxyY9WnfD6Ez\nFGfUt8K36apf1lWBqVV9SeNJ1PnfyI7Ck6yPXU90fjR9evVhWsg0Zvab2e5EXlNVTcbpFHJOxVOS\nkExtejqmWZlYF2TjcL7oonOLLKwpcXCjxs1D9+nN3xe3sCA8wwMxt7RoVxySfsiELl0zrVZLXnoO\nmTG6YkQVKSkYnc3EMi8Lx+Jcel2ymKrA1oVKF3fw8NQtpgpp22Kqs+Vn+fz053yb+C3lteVEOEVw\nR8AdTPadjFVNVX0xsa26TRWq63dp8hlZ3/c+Sbc7jiHQanSbH1/YvedcjO64nU9jN5LPKDIr8/gu\n+Tu65/cAAAnRSURBVDs2J27mXMU5vK29mRE6gyn+U66pa0VTpyEnJYOzJ+MpjE+mOvUMxmcz6J2X\njUNZPiZNF9mYWlBo70a1qzuKpye9/f1wCg3EKyIYa3tZx9zQyYQu6ZWmTsPZhFSyTyU2LqY6m06f\n/BwcyvIvmsnQuJjKHWMv3WIql36BePUPxsqm+V3dy2vK2ZK0ha8TvialJAULEwsmek/ktoDbGOQ8\nSLdgq8kMEPLjdRc6BDZWB/Qa3rkzQCqLdQOZCfW791QWgmIMXiMaY3IMpKKukp0ZO9mcuJmDOQdR\nUBjRdwTTQqZxvcf1zQ52arVaCrPyyIiOIz8uicqUM3A2A8vcLByLz2GmqW04t9rYlHxbFyqc+4KH\nJxZ+frrByP7B2Pd1kv3aXZhM6FKnaVhMFaMrflabnoZpdmbzi6kc63em8vLGOtAP17BAvMIC6WWu\nm+EihCA6P5rNiZv5+czPVNRV4GDuwHjv8Yz3Hk+USxQmRia6GiUJ23TdGql76ndpsoGAGxp3abLS\n86wKISA/obFLKH0/CA1Y2NfXl58I/jeChS2lNaX8nvE729O2sy9rH9Waatx7u3N7wO3cHnA7rlau\nDbctKyol7UQcead1myKI9HTM6ney6d1kjUKdYkSBtSPljm5oGrrBAnDvH4yrn6fhrlGQ2qVTErqi\nKJOBtwBj4CMhxPKrnS8Tes9TVlSqG4CLTeR8UgqazHTMc87qVgNecTGVrviZbjGVPw4hXiSYn2FH\n5q/sObuHyrpKbMxsGOY6jOF9hzPcbTiefTx1uzSl7Grs7ig/ByjgMaRxKbxLeNvmvNdV1+/es1X3\n5lGUqjvuEt44n94jCg0QWxDLgewDHMg+wLHcY9Rp63C2dGa813jGOl+Pfa4V52KTKEs+o9vJJjsT\nm4Js7K6wyKb0QvEoH19sAn1xDQvGM9Sv4Y1P6jk6Y8ciYyABmABkAoeBaUKI2OaukQldaqrgbC7p\nJ06TH59M5ZlUyEjHMldX/OyixVTGpvVT5Fwpde1FlmMZ8b2zSLIqoagP9O3jQaRTJOGO4YQ7hhNi\nG4RF01Z01jHdjazd61vRF+qcXKW/uiynsdJk8s6GejX4jWnY6amglwWnCk4Rkx9DTH4Mx/OOU1lR\nilMJ9Kt0w7/UCedcc6xyyrDOz8G+rOCirqlSMyuK6vu1jb10i2ycQwNa7JqSep7OSOgjgCVCiEn1\nXz8HIIR4tblrZEKXWkOr1ZKTnEnmyTiKEpKoOqOrqGeVp1vE0nSwr9bIiHwbUwqstRT0qaOoNxT1\nMUJrbYeZnQvWjp64OLrgqZTgUhSH49kj2FeW0svEHHxGN5aTtfbQVZRM1NWkEdknqFAU8m36kuc2\nhHN9/MioNiEvN4PzBZnUFudiVnYe+zKBfTnYl/XCoQTsy6tp2lPduHNWX/DwwsrfD8cQf7z6h2Dn\nKhfZSK3TGQn9b8BkIcTc+q/vA4YJIR5t7hqZ0KX2urCY6lx8MiUpqVSlZ6BkZ9GrqACr8v9v795C\n46jiOI5/f9mkMdXGSttgL4mtUopFbfVBKn0Rb9QLLQqCooIo+KKgIIi1oIgIQkF9UJCi4oNFEVQU\nrWjFQl+8a6vVWhAftN6iFXshtWmyPx9mCrHdzW6bzZ7J+P9ASGYzy/zYZP6cOefMnD2cOrSPrmq1\n/vsrcKgLhjvBHVAVuMNYQoaOKqiafe8ege5hqIxzihzs7GLfyTMZ6p3F4Vlz6Ji3gJ4z+pm1+Ezm\nnn0WfWfMi8HIMGHNFvRJvzNA0h3AHQADAwOTfbhQcl3d01i0bAmLli2p+fvRkVH2/DLI378Osn9w\nDwf++JO9g7sZ2vs7h4f2MnpwH/xzAA3/g6qjdI4eYtroQSrVEQ5Xuhmu9FCtdOGOTtw9HfWcQqWn\nl2nTZzJj1nx658xn+pzT6O2bzez+05M+UiGEo02koP8M9I/ZXpC/9h+2NwAbIGuhT+B4ITRU6azQ\nNzCXvoG5qaOE0HYTuRb8FFgsaZGkacANwJutiRVCCOF4nXAL3faIpLuAd8mmLT5v+5uWJQshhHBc\nJtSHbnsTsKlFWUIIIUxADL+HEEJJREEPIYSSiIIeQgglEQU9hBBKIgp6CCGURFsfnytpP7CrbQc8\ncbOBP1OHaMJUyDkVMkLkbLXI2VpLbDdcpqvdiwLuauZ5BKlJ+ixytsZUyAiRs9UiZ2tJauohWNHl\nEkIIJREFPYQQSqLdBX1Dm493oiJn60yFjBA5Wy1ytlZTOds6KBpCCGHyRJdLCCGURLKCLuleSZY0\nO1WG8Uh6RNJXkrZJek/SvNSZjiZpvaTv8pyvS5qZOlMtkq6X9I2kqqTCzSiQtErSLknfS7o/dZ5a\nJD0vaVDSjtRZxiOpX9IWSd/mf/O7U2eqRdJJkj6RtD3P+XDqTPVIqkj6UtJbjfZNUtAl9QNXAD+m\nOH6T1ts+z/Zy4C3gwdSBatgMnGP7PLIFu9cmzlPPDuA6YGvqIEfLFzt/GrgSWArcKGlp2lQ1vQCs\nSh2iCSPAvbaXAiuAOwv6eR4CLrG9DFgOrJK0InGmeu4GdjazY6oW+hPAfUBhO/Bt7xuzeTIFzGr7\nPdsj+eZHZKtGFY7tnbaLekPZhcD3tn+wPQy8DKxJnOkYtrcCf6XO0YjtX21/kf+8n6wQzU+b6ljO\nHMg3u/Kvwp3jkhYAVwPPNrN/2wu6pDXAz7a3t/vYx0vSo5J+Am6imC30sW4D3kkdYgqaD/w0Zns3\nBSxAU5GkhcD5wMdpk9SWd2VsAwaBzbaLmPNJssZv/ZXPx5iUO0UlvQ+cXuNX64AHyLpbkhsvp+03\nbK8D1klaC9wFPNTWgDTOmO+zjuxSd2M7s43VTM7w/yHpFOBV4J6jrnYLw/YosDwfe3pd0jm2CzNG\nIekaYND255IubuY9k1LQbV9W63VJ5wKLgO2SIOsi+ELShbZ/m4ws46mXs4aNZCsztb2gN8oo6Vbg\nGuBSJ5yDehyfZdE0tdh5aJ6kLrJivtH2a6nzNGL7b0lbyMYoClPQgZXAaklXAScBvZJetH1zvTe0\ntcvF9te2+2wvtL2Q7PL2ghTFvBFJi8dsrgG+S5WlHkmryC7HVtseSp1niorFzltIWUvtOWCn7cdT\n56lH0pwjs8Ik9QCXU7Bz3PZa2wvyWnkD8MF4xRxiHvp4HpO0Q9JXZF1ERZx+9RQwA9icT698JnWg\nWiRdK2k3cBHwtqR3U2c6Ih9UPrLY+U7glSIudi7pJeBDYImk3ZJuT52pjpXALcAl+f/ktryFWTRz\ngS35+f0pWR96w2mBRRd3ioYQQklECz2EEEoiCnoIIZREFPQQQiiJKOghhFASUdBDCKEkoqCHEEJJ\nREEPIYSSiIIeQggl8S/J+nHbLDu67AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7faa17c407b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "DELTA = 0.5\n",
    "\n",
    "\n",
    "\n",
    "data = {'mean sqaured error': [], \n",
    "        'mean absolute error': [], \n",
    "        'huber': [], \n",
    "        'pseudo huber': [],\n",
    "        'my huber': []}\n",
    "x = np.arange(-4, 4.1, 0.1)\n",
    "\n",
    "def anderson_huber(delta, value):\n",
    "    \"\"\"\n",
    "    scipy.special.huber의 결과값과 동일하게 나옴.\n",
    "    \"\"\"\n",
    "    \n",
    "    if abs(value) <= delta:\n",
    "        return 1/2 * value**2\n",
    "    return delta * (abs(value) - 1/2 * delta)\n",
    "\n",
    "for y in x:\n",
    "    mse = mean_squared_error([0], [y])\n",
    "    mae = mean_absolute_error([0], [y])\n",
    "    he = huber(DELTA, 0-y)\n",
    "    my_he = anderson_huber(DELTA, 0-y)\n",
    "    phe = pseudo_huber(DELTA, 0-y)\n",
    "    \n",
    "    data['mean sqaured error'].append(mse)\n",
    "    data['mean absolute error'].append(mae)\n",
    "    data['huber'].append(he)\n",
    "    data['pseudo huber'].append(phe)\n",
    "    data['my huber'].append(my_he)\n",
    "\n",
    "\n",
    "del data['pseudo huber']\n",
    "data = pd.DataFrame(data, index=x)\n",
    "data.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Experience Replay and Loss Function\n",
    "\n",
    "게임을 진행하면서 학습을 할 경우 observation sequence같의 연관성(correlation)때문에 학습이 제대로 안 될수 있습니다.<br>\n",
    "연관성을 끊어주는 방법으로 experience replay를 사용합니다.\n",
    "\n",
    "먼저 Agent의 experiences $ e_t = (s_t, a_t, r_t, s_{t+1} )$를 각각의 time-step마다 data set $ D_t = \\{ e_1, e_2, ..., e_t \\} $에 저장합니다.<br>\n",
    "학습시 Q-Learning updates를 uniformly random으로 꺼내진 experiences $ (s, a, r, s^{\\prime}) \\sim U(D) $ 통해 실행하게 됩니다.<br>\n",
    "궁극적으로 <span style=\"color:red\">**Loss function**</span> 은 다음과 같게 됩니다.\n",
    "\n",
    "$$ L_i(\\theta_i) = \\mathbb{E}_{s, a, r, s^{\\prime}} \\sim U(D) \\left[ \\left( r + \\gamma \\max_{a^{\\prime}} Q\\left(s^{\\prime}, a^{\\prime}; \\theta^{-}_i\\right) - Q\\left( s, a; \\theta_i \\right) \\right)^2 \\right] $$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Deep Q-Learning with Experience Replay Algorithm\n",
    "\n",
    "<img src=\"./images/deep-q-learning-algorithm.png\">\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}