notebooks/DQN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Flappy Bird - Deep Q-Network\n",
    "\n",
    "In this notebook, we will implement a **Deep Q-Network (DQN)** for the Flappy Bird game.\n",
    "\n",
    "The **Deep Q-Network (DQN) algorithm** is a variant of the Q-Learning that uses a neural network to approximate the Q-function. In 2013, DeepMind proposed this algorithm as a way to solve Atari games with a high level of efficiency.\n",
    "\n",
    "Two different memory approaches are used:\n",
    "\n",
    "- **Experience Replay (ER)**: The most common approach, where the most recent experiences are stored in a replay memory.\n",
    "- **Prioritized Experience Replay (PER)**: The approach where experiences are stored in a replay memory and the importance of each experience is stored in a priority queue.\n",
    "\n",
    "## Index\n",
    "\n",
    "- [1. Initial Setup](#1.-Initial-Setup)\n",
    "- [2. Check the Gym environment](#2-check-the-gym-environment)\n",
    "- [3. Define custom functions](#3-define-custom-functions)\n",
    "- [4. Training the RL agent](#4-training-the-rl-agent)\n",
    "  - [4.1 Agent (Replay Memory) - Training](#4-1-agent-replay-memory-training)\n",
    "  - [4.1 Agent (Replay Memory) - Scores](#4-1-agent-replay-memory-scores)\n",
    "  - [4.1 Agent (Replay Memory) - Loss](#4-1-agent-replay-memory-loss)\n",
    "  - [4.2 Agent (Prioritized Memory) - Training](#4-2-agent-prioritized-memory-training)\n",
    "  - [4.2 Agent (Prioritized Memory) - Scores](#4-2-agent-prioritized-memory-scores)\n",
    "  - [4.2 Agent (Prioritized Memory) - Loss](#4-2-agent-prioritized-memory-loss)\n",
    "- [5. Save agent](#5-save-agent)\n",
    "- [6. Load agent](#6-load-agent)\n",
    "- [7. Test trained agent](#7-test-trained-agent)\n",
    "- [8. Conclusions](#8-conclusions)\n",
    "\n",
    "## References\n",
    "\n",
    "- [[1312.5602] Playing Atari with Deep Reinforcement Learning](https://arxiv.org/abs/1312.5602)\n",
    "- [[2007.06700] Revisiting Fundamentals of Experience Replay](https://arxiv.org/abs/2007.06700)\n",
    "- [[1511.05952] Prioritized Experience Replay](https://arxiv.org/abs/1511.05952)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Initial Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add project root directory to the path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "from pathlib import Path\n",
    "\n",
    "if str(Path(\".\").absolute().parent) not in sys.path:\n",
    "    sys.path.append(str(Path(\".\").absolute().parent))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Imports modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import flappy_bird_gym\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from src.trainer import Trainer\n",
    "from src.utils import play_env, aggregate_list\n",
    "from src.dqn.agent import Agent\n",
    "from src.dqn.config import AgentConfig\n",
    "\n",
    "seed = 1993"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Check the Gym environment\n",
    "\n",
    "Before we start, we need to check how the environment works. For this, we will check the action space and observation space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of actions: 2\n",
      "Sample state shape: (2,)\n",
      "Sample state: [ 1.65625   -0.0078125]\n"
     ]
    }
   ],
   "source": [
    "env = flappy_bird_gym.make(\"FlappyBird-v0\")\n",
    "\n",
    "env.seed(seed)\n",
    "\n",
    "action_size = env.action_space.n\n",
    "state_size = env.observation_space.shape[0]\n",
    "\n",
    "print(f\"Number of actions: {action_size}\")\n",
    "print(f\"Sample state shape: {env.observation_space.shape}\")\n",
    "print(f\"Sample state: {env.reset()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Define custom functions\n",
    "\n",
    "In the next cell, we define a `custom_prep_state()` function that can be used to preprocess the state of the game.\n",
    "\n",
    "We will replace the original `prep_state()` (that does nothing) with this function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of state: (2,)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([ 1.65625   , -0.01171875])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def custom_prep_state(state: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    Custom preprocessing of the state.\n",
    "\n",
    "    This function replaces the default preprocessing of the state of the agent (prep_state())\n",
    "\n",
    "    Args:\n",
    "    - state: State to preprocess.\n",
    "\n",
    "    Returns:\n",
    "    - Preprocessed state.\n",
    "    \"\"\"\n",
    "    return state\n",
    "\n",
    "sample_state = custom_prep_state(env.reset())\n",
    "print(f\"Shape of state: {sample_state.shape}\")\n",
    "sample_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Training the RL agent\n",
    "\n",
    "First, we will instantiate the AgentConfig that defines the hyperparameters of our DQN Agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'state_size': 2,\n",
       " 'action_size': 2,\n",
       " 'seed': 1993,\n",
       " 'nb_hidden': (64, 64),\n",
       " 'learning_rate': 0.0005,\n",
       " 'memory_size': 100000,\n",
       " 'prioritized_memory': False,\n",
       " 'batch_size': 64,\n",
       " 'gamma': 0.99,\n",
       " 'tau': 0.001,\n",
       " 'small_eps': 0.03,\n",
       " 'update_every': 4,\n",
       " 'epsilon_enabled': True,\n",
       " 'epsilon_start': 1.0,\n",
       " 'epsilon_end': 0.01,\n",
       " 'epsilon_decay': 0.99995,\n",
       " 'model_dir': '../models/DQN.pt'}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = AgentConfig(\n",
    "    state_size=len(sample_state),\n",
    "    action_size=action_size,\n",
    "    seed=seed,\n",
    "    prioritized_memory=False,\n",
    "    model_dir=\"../models/DQN.pt\"\n",
    ")\n",
    "\n",
    "params.dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will define the trainer parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer_args = {\n",
    "    \"n_episodes\": 60000,\n",
    "    \"print_range\": 1000,\n",
    "    \"early_stop\": 120,\n",
    "    \"max_timestep\": None,\n",
    "    \"verbose\": True,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Agent (Replay Memory) - Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 1000\tAvg Score: 99.45\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 2000\tAvg Score: 101.01\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 3000\tAvg Score: 101.06\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 4000\tAvg Score: 101.16\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 5000\tAvg Score: 101.63\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 6000\tAvg Score: 101.55\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 7000\tAvg Score: 102.48\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 8000\tAvg Score: 101.92\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 9000\tAvg Score: 101.87\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 10000\tAvg Score: 102.87\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 11000\tAvg Score: 103.12\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 12000\tAvg Score: 102.84\tEpsilon: 0.01\tAvg. MSE Loss: 1.00\n",
      "Episode 13000\tAvg Score: 103.75\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 14000\tAvg Score: 103.69\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 15000\tAvg Score: 102.30\tEpsilon: 0.01\tAvg. MSE Loss: 1.00\n",
      "Episode 16000\tAvg Score: 102.56\tEpsilon: 0.01\tAvg. MSE Loss: 1.00\n",
      "Episode 17000\tAvg Score: 102.13\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 18000\tAvg Score: 101.59\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 19000\tAvg Score: 103.03\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 20000\tAvg Score: 103.15\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 21000\tAvg Score: 103.48\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 22000\tAvg Score: 102.09\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 23000\tAvg Score: 102.37\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 24000\tAvg Score: 103.56\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 25000\tAvg Score: 103.17\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 26000\tAvg Score: 102.73\tEpsilon: 0.01\tAvg. MSE Loss: 2.00\n",
      "Episode 27000\tAvg Score: 106.12\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 28000\tAvg Score: 104.17\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 29000\tAvg Score: 102.57\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 30000\tAvg Score: 103.52\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 31000\tAvg Score: 101.51\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 32000\tAvg Score: 104.35\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 33000\tAvg Score: 104.98\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 34000\tAvg Score: 107.56\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 35000\tAvg Score: 108.17\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 36000\tAvg Score: 107.45\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 37000\tAvg Score: 106.94\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 38000\tAvg Score: 107.37\tEpsilon: 0.01\tAvg. MSE Loss: 1.00\n",
      "Episode 39000\tAvg Score: 109.25\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 40000\tAvg Score: 107.17\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 41000\tAvg Score: 107.78\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 42000\tAvg Score: 111.08\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 43000\tAvg Score: 116.23\tEpsilon: 0.01\tAvg. MSE Loss: 1.00\n",
      "Episode 44000\tAvg Score: 112.58\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 45000\tAvg Score: 108.48\tEpsilon: 0.01\tAvg. MSE Loss: 1.00\n",
      "Episode 46000\tAvg Score: 109.58\tEpsilon: 0.01\tAvg. MSE Loss: 1.00\n",
      "Episode 47000\tAvg Score: 110.51\tEpsilon: 0.01\tAvg. MSE Loss: 1.00\n",
      "Episode 48000\tAvg Score: 106.07\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 49000\tAvg Score: 101.16\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 50000\tAvg Score: 103.09\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 51000\tAvg Score: 102.19\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 52000\tAvg Score: 101.95\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 53000\tAvg Score: 101.58\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 54000\tAvg Score: 102.74\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 55000\tAvg Score: 102.52\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 56000\tAvg Score: 102.32\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 57000\tAvg Score: 102.47\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 58000\tAvg Score: 103.32\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 59000\tAvg Score: 103.11\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 60000\tAvg Score: 103.39\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Wall time: 1h 10min 12s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "agent = Agent(**params.dict())\n",
    "\n",
    "# Add our custom prep_state function to the agent\n",
    "agent.prep_state = custom_prep_state\n",
    "\n",
    "trainer = Trainer(agent=agent, env=env, **trainer_args)\n",
    "trainer.run(logs_callback=agent.logs, save_best_model=True, output_path=\"../models/DQN_ER_best_score.pt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Agent (Replay Memory) - Scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best score: 116.228 - (episode: 43000)\n",
      "Last score: 103.388 - (episode: 60000)\n"
     ]
    }
   ],
   "source": [
    "print(f\"Best score: {trainer.best_score} - (episode: {trainer.best_episode})\")\n",
    "print(f\"Last score: {trainer.last_score} - (episode: {trainer.last_episode})\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEGCAYAAABCa2PoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA/NElEQVR4nO3dd3gc5bX48e9Z9d4ly5bcG7ZxlR16byH00BwSTGjhBpKbegM3jXDJ/RGSGxKSUAwxmAQIoTv0ErAN2Aa5N2y5qiCry+p1398fMyuvZa20krZopfN5nn00Ozsz+w6s9+zbzivGGJRSSqnBcgS7AEoppYYHDShKKaV8QgOKUkopn9CAopRSyic0oCillPKJ8GAXIBDS09PN+PHjg10MpZQKKevXr680xmR4e/yICCjjx48nPz8/2MVQSqmQIiIH+3O8NnkppZTyCQ0oSimlfEIDilJKKZ/QgKKUUsonNKAopZTyCQ0oSimlfEIDilJKKZ/QgKKUCpqVuys4UNkY7GIoH9GAopQKmu88s4GHPtwT7GIoH9GAopQKipb2TupaOiiuaQ52UZSPaEBRSgVFdWMbACW1GlCGCw0oSqmgqGqwAkppbQtOpy5FPhxoQFFKBUVlYysAbZ1OKhpag1wa5QsaUJRSQVFt11BAm72GCw0oSqmgqGo8Uisp0Y75YUEDilIqKKoa2ghzCKA1lOFiRCywpZQaeiob2shKiKKhtYMvNKAMCxpQlFJBUdXYSlp8FEmxkdrkNUxoQFFKBUV1Yxtp8ZGEO0QnNw4T2oeilAqKqoY2UuMiGZMcozWUYcJvAUVElolIuYhsc9t3lYhsFxGniOS57R8vIs0issl+POLhmqki8q6IFNh/U/xVfqWU/xhjqGxoJT0+ijEpMdS3dlDX0h7sYqlB8mcN5Unggm77tgFXAKt6OH6vMWau/bjNwzXvBN43xkwB3refK6VCTGNbJ60dTtLiIhmTHAvo0OHhwG8BxRizCqjutm+nMWbXIC57KbDc3l4OXDaIaymlgsQ1qTEtPorRydGABpThYCj1oUwQkY0islJETvVwTJYxptTePgRkebqYiNwqIvkikl9RUeHzwiqlBs6VdiUtLpIxKTGAzkUZDoZKQCkFxhpj5gE/AJ4RkcTeTjDGGMBjRjljzFJjTJ4xJi8jI8O3pVVKDUpVVw0lkvS4KCLDHToXZRgYEgHFGNNqjKmyt9cDe4GpPRxaJiLZAPbf8sCVUinlK1V2Msi0+CgcDmF0UjTFGlBC3pAIKCKSISJh9vZEYAqwr4dDVwBL7O0lwKuBKaFSypeq7LVQ0uIiARiTokOHhwN/Dht+FlgDTBORYhG5SUQuF5Fi4ETgdRF52z78NGCLiGwCXgBuM8ZU29d53G2I8X3AuSJSAJxjP1dKhZiqhjbio8KJjggDsOaiaA0l5PltprwxZrGHl17u4dgXgRc9XOdmt+0q4GyfFFApFTRVja2k2rUTgDHJsVTUt9La0UlUeFgQS6YGY0g0eSmlRpaqBivtiotr6HBpbUuwiqR8QAOKUirgKhtaSYuL6nquQ4eHBw0oSqmAq25sI92thpKjs+WHBQ0oSqmAcjoN1Y1tR/WhjEqKRkRrKKFOA4pSKqDqWtrpcBrS4o80eUWGO8hMiNKAEuI0oCilAqrSniXv3uQFaBr7YUADilIqoKq7JjVGHbV/TEosXxzWgBLKNKAopQLKlXbFvQ8FrKHDpbUtOJ0eU/SpIU4DilIqoCobe27yykmOoa3TSYUdcFTo0YCilAooVw0lpVsNxTUXRdeXD10aUJRSAVXV0EZybAQRYUd//bhWbtQ09qFLA4pSKqCqG9u6sgy761q5UQNKyNKAopQKqO5pV1wSoiNIjA7XocMhTAOKUiqgqhqPTgzpbkxKrNZQQpgGFKVUQFU1tHoOKMkx2ocSwjSgKKUCpqPTSW1ze49NXgBjkqO1ySuEaUBRSgVMTVM7xtBLk1cM9a0dHG5uD3DJlC9oQFFKBUxVozUHxXMNRdPYhzINKEqpgKmyE0N6qqG4hg5rP0po0oCilAqYSnuWfPe0Ky66cmNo81tAEZFlIlIuItvc9l0lIttFxCkieW77zxWR9SKy1f57lodr3i0iJSKyyX5c6K/yK6V8z1OmYZf0uCgiwx0aUEKUP2soTwIXdNu3DbgCWNVtfyVwsTHmeGAJ8LdervuAMWau/XjDV4VVSvlfVUMbYQ4hKSaix9cdDtF1UUJYuL8ubIxZJSLju+3bCSAi3Y/d6PZ0OxAjIlHGGE07qtQwUtXYSkpsJA6HeDxmdHK0rosSooZiH8pXgQ29BJM7RGSL3aSW4ukiInKriOSLSH5FRYV/SqqU6pfKhjaP/ScuWYnRlB1uCVCJlC8NqYAiIjOB3wDf8nDIw8AkYC5QCvyfp2sZY5YaY/KMMXkZGRm+LqpSagCqe0m74jIqMZqy+lY6daGtkDNkAoqI5AAvA9cbY/b2dIwxpswY02mMcQKPAYsCWUal1OBUNbSS6qFD3iU7KZpOp+laN0WFjiERUEQkGXgduNMY83Evx2W7Pb0cq5NfKRUiqhp6Tl3vLivRmotyqE6bvUKNP4cNPwusAaaJSLGI3CQil4tIMXAi8LqIvG0ffgcwGfiF25DgTPs6j7sNMb7fHlq8BTgT+L6/yq+U8q3Wjk7qWzv67EPJTrLmopRqP0rI8ecor8UeXnq5h2PvBe71cJ2b3ba/4ZvSKaUCrWsOSnzvTV5ZSdbrZVpDCTlDoslLKTX8udKupPbR5JUeF0W4QzikNZSQowFFKRUQfaVdcXE4hKzEaA0oIUgDilIqILoSQ/YxygsgKzFKO+VDkAYUpVRAdKWu76OGAjAqSWsooUgDilIqIKoa24gMdxAf1fdYoFGJMRyqa8EYndwYSjSgKKUCwjUHpXsuv56MSoqiqc0aZqxChwYUpVRAVDW0etXcBTDKnouizV6hRQOKUiogqhrbvOqQByufF2hACTUaUJRSAVHV0HdiSJdRmn4lJGlAUUr5nTGGqsbWPvN4uWQmWjUZraGEFg0oSim/a2rrpKXd2WemYZfoiDBS4yK1hhJiNKAopfzuC3uN+NHJ0V6fowtthR4NKEopvyusbgJgbGqs1+dkJ0VrxuEQowFFKeV3AwkoWYnRmnE4xGhAUUr5XWF1E3GRYX1mGnaXnRRNVWMbrR2dfiyZ8iWvAoqIjBORc+ztGBFJ8G+xlFLDSVF1E7mpsV7NkndxDR0ur9OlgENFnwFFRG4BXgAetXflAK/4sUxKqWGmsLqpX81dAFlJA5uL0t7p7Nfxyne8qaHcDpwM1AEYYwqATH8WSik1fBhjKLRrKP2RndT/2fKrCyqY+Yu3OVDZ2K/3Ur7hTUBpNca0uZ6ISDigKUCVUl6paGilpd3Z/xpKP9OvGGP4w3sFtHU6KShv6Hc51eB5E1BWish/AzEici7wPPAv/xZLKTVcFA1ghBdAYnQ4MRFhXjd5fbq/mvUHawAor9fRYcHgTUD5CVABbAW+BbwB/Mybi4vIMhEpF5FtbvuuEpHtIuIUkbxux98lIntEZJeInO/hmhNEZJ193HMi4v2wEaVUwLmGDPe3yUtEyO7HQlsPfbjXTo8PZdqRHxS9BhQRCQN2GmMeM8ZcZYy50t72tsnrSeCCbvu2AVcAq7q91wzgWmCmfc5D9vt39xvgAWPMZKAGuMnLsiilgqCwypoln5MS0+9zsxKjvaqhbCs5zMrdFdx4ygTS4iKp0BpKUPQaUIwxncAuERk7kIsbY1YB1d327TTG7Orh8EuBfxhjWo0x+4E9wCL3A8Qac3gW1qgzgOXAZQMpm1IqMAqrmxiVGE10RE+/D3vnbQ3l4Q/3khAVzjdOHEdmQrQONQ6SvtfihBRgu4h8CnQNnTDGXOLjsowB1ro9L7b3uUsDao0xHb0co5QaQooGMGTYJSvJmi3vdBocjp7nsOyraOCNbaXcdvokEqMjyEyMokxrKEHhTUD5ud9L4QcicitwK8DYsQOqYCmlfKCwuomTJ6cP6NxRidF0OA1VjW1kJPScqfiRlXuJDHNw48kTAMhMiGLHF3UDLq8auD475Y0xK4HPgQT7sdPe52slQK7b8xx7n7sqINkeuuzpGACMMUuNMXnGmLyMjAyfF1Yp1beW9k4O1bUMuIYyyp6L4imnV+nhZl7eWMI1C3O7Ak5mQjSVDa10OnV2Q6B5M1P+auBT4CrgamCdiFzph7KsAK4VkSgRmQBMsd+3iz0Y4APA9f5LgFf9UBallA8U11gd8mPT+t8hD0fSr3jKOvzYqv0YA7eeNrFrX1ZiFE4DVY3ajxJo3gwb/imw0BizxBhzPVZHuVfNYCLyLLAGmCYixSJyk4hcLiLFwInA6yLyNoAxZjvwT2AH8BZwuz0oABF5Q0RG25f9CfADEdmD1afyV29vVikVWAOdg+Iyqpf0K9WNbTz7aSGXzB1NTsqR62ckaA6wYPGmD8VhjCl3e16Fl0kljTGLPbz0sofjfw38uof9F7pt76Pb6C+l1NA00DkoLunxUYQ5pMeFtp5ee5Dm9k7+4/RJR+13LR9sTW5MGtD7qoHxJqC8ZdcinrWfXwO86b8iKaWGi8LqJqIjHGTEe7f0b3dhDiEzIeqYJq9Op+EfnxVxyuR0pmQdnfw8S7MUB02fAcUY82MRuQI4xd611BjTYw1DKaXcubIM9ydtfXejko5daGt1QQUltc3894XHHXO8K3iV12tACbQ+A4rdQf6GMeYl+3mMiIw3xhzwd+GUUkNHU1sHNU3tjEn2voN9MHNQXEYlRh+T7PGZdYWkxUVy7oysY46PDHeQEhuhqz0GgTd9Ic8D7gsMdNr7lFIjyO/f2c1XHlxNh5frjQw0bX13WYlHz5Yvq2vh/c/LuTIvh8jwnr/CMhOitYYSBN4ElHD39PX2tiZkVGqE+XhvFbVN7eyp8C41fFVjG01tnYOuoWQnRdPQ2kF9SzsAz+cX0ek0XLvQ84TlzMQoDShB4E1AqRCRrjQrInIpUOm/Iimlhpq6lnZ2HbJmn28uqvXqnMJBDhl2cZ/c6HQanv20iJMmpTEhPc7jOVY+L23yCjRvAsptwH+LSKGIFGHNA/mWf4ullBpKNhbW4pp4vqnosFfnDHYOisuRhbZaWb2nkpLaZhYv6j2dUmZiFBX1rTh1tnxAeTPKay9wgojE2891KTSlRpj8A9WEOYQ5OUne11CqrIDiPulwILLdJje+t6OM1LhIzpt5bGe8u8yEKDqchpqmNtIGOGRZ9Z/HGoqIXCwi49x2/QD4WERW2CO/lFIjxGcHqpmRnchJk9LZVVZPc1tnn+cUVjeRmRBFTGT/09a7c9VQthbX8t7OMq5ckENUeO/X7JqLov0oAdVbk9evsVZqREQuAr4O3IiVc+sR/xdNKTUUtHc62VRUS974FObkJtPpNOwo7bvZq9AHQ4YBoiPCSImN4B+fFdHhNFy7MLfPczLtRJE6dDiwegsoxhjTZG9fAfzVGLPeGPM4oOl7lRohtn9RR0u7k4XjU5mTY6Uy8aYfxRdzUFyyEqNp7XBy4sQ0JmbE93l8ZoLWUIKht4AiIhIvIg7gbOB9t9ei/VsspdRQkX/AWnQ1b1wKmYnRZCdF99mP0trRSWldy6DnoLi4Rnot/pJ3axu58nlVaEAJqN4Cyh+ATUA+1hoo+QAiMg8o9XvJlFJDwmcHqhmbGkum3S8xJyeZzcW1vZ5TUtOMMYMf4eUyOSOejIQozu+jM94lOiKMxOhwHTocYB4DijFmGXA6cBNwodtLh4Bv+rlcSqkhwBjD+oM15I1P6do3JzeZg1VN1DS2eTyvaw5Kmm8Cyo/On8Zb/3lqn53x7jIToynTBJEB1es8FGNMiTFmozHG6bav1BhT6P+iKaWC7UBVE5UNbSwcn9q1b06u1Y/SWy3FV3NQXKIjwvo9/DczIcpOYa8Cxat1TZRSoaWlvZNlH+2ntslzLcIbn7n1n7gcPyYJEdhS7LljvrC6iajwgaet9wUroGgNJZA0oCg1DP3Pazu457Ud3POvHYO6Tv6BapJjI5jkNrIqITqCyRnxvXbMu5JCOhwDT1s/WFmJ0ZTXtWKtHK4CwauAIiKniMg37e0Mndio1ND19vZDPL2ukLGpsby0sYR1+6oGfK38gzXkjUs5JjDMtjvmPX1ZF1Y3+6y5a6AyEqJo63RyuLk9qOUYSfoMKCLyS6z8XXfZuyKAv/uzUEqpgTl0uIWfvLiFWWMS+dd3TmFMcgw/f3Ub7V6mnHdX1dDKvopG8tz6T1zm5iZR2dBGSW3zMa8ZY3w6B2WgMnW2fMB5U0O5HLgEaAQwxnwBJPR6hlIq4Dqdhu8/t4nWdicPXjuPpJgI7r5kJrvLGnji4/39vl7+wRrg6P4Tlzm5yQBs7mGCY01TOw2tHT6bgzJQrtnyuhRw4HgTUNqMVa81ACLiOWe0UipoHl21lzX7qvjVJTO7ZpOfOyOLs6dn8of3Cig9fGxtojf5B6qJDHdwvD073t30UYlEhjnY0sNIr42FViAKdg3Flc9L068EjjcB5Z8i8iiQLCK3AO8Bj/V1kogsE5FyEdnmti9VRN4VkQL7b4q9/8cissl+bBORThE5pp4tIk+KyH63Y+d6fadKDWObimqtFRWPz+aqvJyjXrv7kpl0Og33vrazX9fMP1jDnJykHud+RIY7mDE6kU3dOuYPN7Xzs1e2MSE9jlMmp/f7Pnypq4aiTV4B02dAMcb8DngBeBGYBvzCGPMnL679JHBBt313Au8bY6ZgpXK5036P3xpj5hpj5mL11aw0xlR7uO6PXccaYzZ5UQ6lhrWG1g6+++xGshKj+d/Lj0fk6A703NRYbj9zMq9vLWXV7gqvrtnc1sm2ksM99p+4zM1NZmvJYTrtNUeMMfz0la1U1Lfyh2vmDjrL8GDFRYUTFxmmc1ECyKtRXsaYd40xPzbG/MgY866X56wCugeFS4Hl9vZy4LIeTl0MPOvNeyil4M2tpRRWN/HbK2eTFBvR4zG3njaRCelx/HLFdlo7+k49v7m4lvZO02P/icvsnCSa2jrZU24tkfTyxhJe21LK986Z0tXHEmyZibq2fCB5M8qrXkTquj2KRORlEZnYz/fLMsa48oAdAo5KzCMisVi1mhd7ucavRWSLiDwgIh5nTYnIrSKSLyL5FRXe/SpTKhTtOlRPVLiDL01M83hMdEQYv7pkJvsrG/nNm7v6vKYrIeSCXgLKkY75Woqqm/jFq9tZOD6F/zhjcv9uwI8yE6I0n1cAeVND+QPwY2AMkAP8CHgG+AewbKBv7N7R7+Zi4ONemrvuAqYDC4FUrOHMnq6/1BiTZ4zJy8jQbPtq+NpVVs+UrHjC+phEeNrUDG44aTzLPt7PU2sOeDzu0OEW/plfzPRRCSTHRno8bkJaHAnR4WworOH7z21CgAeumdtnOQJJayiB5U1AucQY86gxpt4YU2eMWQqcb4x5DvD886VnZSKSDWD/Le/2+rX00txl5xEzxphW4AlgUT/fX6lhp6CsgamZ3o3k//lFMzjnuCzuXrGd93aUHfN6SW0z1yxdQ1VDK/9z2axer+VwCHNyknl+fTH5B2v4n8tmDXq5X1+zaig6Wz5QvAkoTSJytYg47MfVgKsO2d//SyuAJfb2EuBV1wsikoSV3fjVHs5zHeMKRoLV/7LN07FKjQSHm9s5VNfClCzvAkqYQ3hw8VxmjUniO89uZKtbPq6i6iaueXQN1Q1t/O3mLx2VENKTOblJdDoNl8wZzWXzxgz4PvwlMyGK5vZO6ls7gl2UEcGbgHId8A2s2kSZvf11EYkB7vB0kog8C6wBpolIsYjcBNwHnCsiBcA59nOXy4F3jDGN3a7zhoiMtp8+LSJbga1AOnCvF+VXatgqKKsHYNqovlcxdImNDOfxJXmkxkVy4/LPKK5p4mBVI9c8uoa65naevuVLzB/rXePDxXNG85XZ2X3WZoKla215ndwYEOF9HWCM2YfVt9GTj3o5b7GHl872cPyTWEONu++/0G37LE/vp9RItLvMGmE1xcsmL5fMhGiW37iQKx76hCXLPqWxtZPWjk6eueUEZo05diKjJ9NHJfKXr83v13sH0pG5KC1MzvQ+6KqB6TOgiEg01iJbM3Fb+tcYc6Mfy6WU8sLusnriIsMYkxzT73MnZybwyDcWsGTZpyRGR/DsrScwfVSiH0oZPLoUcGD1GVCAvwGfA+cD92A1gfVvyq1Syi92l9UzOSthwGniT5qUzsvfPpmUuMgBBaWhLiNB068Ekjd9KJONMT8HGo0xy4GvAF/yb7GUUt7YXVbPtKzBNeXMGpM0LIMJQGJ0ONERDu1DCRBvAoprMYFaEZkFJAGZ/iuSUsob1Y1tVDa0MdXLEV4jkYiQmaBzUQLFmyavpXYSx59hDfuNB37u11Ippfq02x7h5e2Q4ZFK15YPnF4Diog4gDpjTA2wCuhvqhWllJ+4Aso0DSi9ykqMZmdpXbCLMSL02uRljHEC/xWgsiil+mF3WT0J0eFkJXpMaaewlgLWJq/A8KYP5T0R+ZGI5NrrmaT2tFaJUiqwdh9qYFpWwjHp6tXRMhOjaGjtoFFny/udN30o19h/b3fbZ9DmL6WCxhjD7vJ6vjwrO9hFGfIyE46sLT8hypuvPDVQ3syUnxCIgigVKmoa29hYVMNZ07P6PthPKhpaqW1qH/SQ4ZHA1SRYXtfChHRdwdyfvFkPJVZEfiYiS+3nU0TkIv8XTamh6el1B7nxyfx+r9HuS7sPWSlXdMhw39xrKMq/vOlDeQJoA06yn5egSRnVCHagqgmADQdrg1YGHTLsvYwETb8SKN4ElEnGmPuxJzgaY5oA7QVUI1ZRtRVQ1h+sCVoZdpfVkxoXSXq85wWwlCU5JoIwh1DZoAHF37wJKG12qnoDICKTAP0/o0as4hqrqWtDYXADypTMeB3h5QWHQ0iNi6SqoS3YRRn2vAkodwNvAbki8jTwPjo3RY1Q7Z1OSg83ExEmbP/iMC3tnQEvgzGGgrIGpo3S5i5vpcdHaQ0lAPoMKMaYd4ArgBuwlufNM8Z86N9iKTU0fVHbjNPAGdMyae80bCs53PdJPlZ6uIX61g7tP+mH9PhIDSgB4M0or38B5wEfGmNeM8ZU+r9YSg1NRdVWc9clc6xFRIPR7OXqkJ+qC0Z5LSM+ikpt8vI7b5q8fgecCuwQkRdE5Ep70S2lRpyiGqtDft7YZMamxgZlpFdXQNEaitfSE6KoaGjFGBPsogxr3jR5rTTGfBtrZvyjwNVY68srNeIUVTcR7hCyk2KYPzaZ9YU1Af+S2l3WQEZCFClxOsLLW2lxkbR1OGnQ9Ct+5U0NBXuU11eB24CFwHJ/FkqpoaqoppnRyTGEOYQF41KoqG/tGvUVKAVl9ZphuJ/S4625KNrs5V/e9KH8E2vJ37OAP2PNS/mONxcXkWUiUi4i29z2pYrIuyJSYP9NsfefISKHRWST/fiFh2tOEJF1IrJHRJ4TEf2ZpgKmqLqJ3FRrdcN5Y1OAwPajOJ2G3WUNTNGUK/2SnuAKKNox70/e1FD+ihVEbjPGfACcJCJ/8fL6TwIXdNt3J/C+MWYK1hDkO91eW22MmWs/7vFwzd8ADxhjJgM1wE1elkWpQSuuaSI3JRaA6aMSiI0MY0MAJziW1DbT3N6pNZR+ck0ArdTZ8n7lTR/K28BsEblfRA4A/wN87s3FjTGrgOpuuy/lSJPZcuAybwsr1iyus4AXBnK+UoPR1NZBZUMbualWQAkPczAnJ5kNhbUBK8OuQ5pyZSAy4rWGEggeA4qITBWRX4rI58CfgCJAjDFnGmP+NIj3zDLGlNrbhwD3lK0nishmEXlTRGb2cG4aUGuMcfWsFQNjBlEWpbzm6ivJSYnp2jd/XDI7S+tobgvMBMfPDlq/z7TJq39S7QEM2ofiX73VUD7Hqg1cZIw5xQ4iPv1XY6zhMa4hMhuAccaYOVgB7JXBXFtEbhWRfBHJr6ioGFxBleJIDi9XDQVg/tgUOpyGLcW1fn//bSWH+evq/Vw0O5vE6Ai/v99wEh7mICU2QmsoftZbQLkCKAU+EJHHRORsfJMUskxEsgHsv+UAxpg6Y0yDvf0GECEi6d3OrQKSRcS1jksOVvbjYxhjlhpj8owxeRkZGT4othrpugJKypGA4uqYX+/njvmW9k6+99wm0uIjufeyWX59r+FK06/4n8eAYox5xRhzLTAd+AD4HpApIg+LyHmDeM8VwBJ7ewnwKoCIjLL7SBCRRXbZqrqVydhlubL7+Ur5W1FNMzERYUdl+E2Ni2RiepzfJzje9+bn7Clv4HdXzSE5Vgc2DkS6zpb3O2865RuNMc8YYy7GqhFsBH7izcVF5FlgDTBNRIpF5CbgPuBcESkAzrGfgxUktonIZuBB4Fo7gCAib4jIaPu4nwA/EJE9WH0qf/XyXpUalKLqJnJSYo7J8Dt/XAob/TjBcXVBBU9+coAbThrPqVO0tj1Q6QlaQ/G3fi2wbIypAZbaD2+OX+zhpbN7OPbPWPNcerrOhW7b+4BF3ry/Ur5UVNN8VP+Jy/yxKbywvpiDVU2M9/ESs7VNbfzo+c1Myojjzi9P9+m1R5r0eE1h729ezZRXaqQzxlBc3USu2wgvl/njkgHfT3A0xvCzV7ZR1dDGH66ZR3REmE+vP9Kkx0fR0NoRlCUHRgoNKEp54XBzO/WtHT3WUKZkJpAQFe7zgPKvLaW8tqWU750zheNzknx67ZHI1felSwH7jwYUpbzgSlufk3JsQAlzCHPHJrPexx3zL6wvZmJ6HLedPsmn1x2p0nVyo99pQFHKC6609a48Xt3NG5vCrkN1Ps1mu7e8geNzkggP03+mvqAJIv1PP6lKeaGnSY3u8sal4DSQf6B7pqGBaW7rpKS2mYnpOiPeV1wJIqu0huI3GlCU8kJRTRNJMREeZ6gvHJ9KZLiDjwp8s6Dp/spGACZl+nbU2EiW1pV+RQOKv2hAUcoLRdXNHpu7AGIiw1g0PpXVPgoo+yobALSG4kPREWEkRIVrk5cfaUBRygtFbmnrPTl1Sjq7yuopq2sZ9PvtLbdqKBN8PK9lpHMtBaz8QwOKOsamolqWrtrL29sPsbusfsSP23c6DcUeJjW6c81i90UtZV9lA2OSY4iJ1LknvpQeH6lrovhRv2bKq+GvvdPJHc9sOGpZWxHITozmxEnp/O6q2cekHhnuKhpaaetw9jip0d1x2Qmkx0exancFVy7IGdR77q1oYGKG1k58LT0+ioLyhmAXY9jSgKKOsmLTFxTXNPPHa+cyIT2O/ZWNHKxq4rMD1by4oZibT53AcdmJwS7moBVVN5EcG0GCF2ngXSO8cvqooYgIp01J58PdFTidBodjYIHXGMP+ikby8lIHdL7yLD0+ijX7qvo+UA2INnmpLk6n4aEP9zB9VAKXzBnN7JxkLp07hu+ePYXfXz0XEXh7+6FgFxOAtg5n1xd9fzW2dnDhH1dz8Z8+orCq72t0zUHpow8F4NSp6VQ3trGjtK7H151Ow4+e38yq3Z7X6Cmra6WxrZNJWkPxubT4SGqb2mnvdAa7KMOSBhTV5a3th9hb0cjtZ04+plkrIyGKBWNTeHt7WZBKd0RRdRNXPfIJZ/7uQ68CQndvbz9EfWsHZXWtXPHwJ2wrOdzH+x27UqMnJ0+2lvBZVdBzwHhvZxkvrC/mn/lFHq+xt8Ie4ZWhI7x8zTW5cSQkiTTG8OGucpxO/2TB7okGFAVYH76/fLCHielxXHh8do/HnD9zFDtL6wZcM/CFt7Yd4sIHV7OvopEOpxlQjemlDSXkpsaw4o6TiQwTrl26lo/3eO5IL6puIjMhyqvkjJkJ0RyXneixBvLoqn0AbOxlHfp9dkCZpAHF54ZD+pXKhlY+2FVOZy+BoqS2mRuf/IwbnviM17eWejzO1zSgKAA+3FXB9i/quO2MSYR5aPs/f+YoIDjNXm0dTn71r+3c9vf1TEyP443/PJXjshP7XZbSw818vLeSy+flMCUrgZe+fTJjkmO44YlPWbH5ix7PKapp6nOEl7vTpqSz/mANjd3SsOQfqGb9wRomZ8ZTUttMeX3Pw4v3VjQSFxlGVmKU9zemvJKRYCeIDOGA8qPnN/PNJz7j7P/7kOc+K6St40jzXafTsOyj/Zz7+5Ws21/Nzy+a4fEHoj9oQBlB2jqcPS4CZYzhzx/sYUxyDJfPG+Px/LFpsUwflRDQgNLR6ST/QDVXPfIJT3x8gG+ePJ7nbzuJ3NRYzpuRxfrCmn5lj3110xcYA1fY9zkqKZp/futE5uWm8N1nN7L8kwPHnFNU3dznCC93p07JoL3TsG7/0Z2/j6zcR0psBL+8eAYAmzzUUqwRXvEjbjRdIIR6k1f+gWo+3FXBZXNHkxAdwU9e3Mrpv/2AZR/tZ2NhDVc8/An3vLaDRRNSeef7p3HTKRM8/kD0Bx3lNQK0tHfy+Op9PPzhXubkJvO/lx9/1EJQa/dZv5zvuXQmEX0kIjx/5ige/HcBlQ2tXf84fckYw/7KRj7aU8nqgkrW7q2ivrWDhOhwHvn6fC6YdeTX1vkzR/HH9wt4b2cZixeN9eraL20oZv7Y5KPuPyk2gqduWsR3nt3IL1dsxxjDDSdPAKxh1KWHm8lN9Rxou8sbn0J0hINVuys5a3oWAHvK63lvZxnfO2cKC8enEu4QNhXVcp5d63O3r6KRvPEpXr+f8l5fTV7PfVZIfFQEX5kduF/13jLG8Nu3d5EeH8X/XnE8MRFhrC6o5M8f7OGe13YAVnqZP147l0vmjA7KDxINKMOYMYYVm7/g/rd2UVLbzKlT0tlUWMv5f1jF98+dys2nTCA8zMFfPthDenwUV+fl9nnNri/xHWVc68WXeH/dvWI7y9ccBKzMvhfNGc0pk9M5ZXI6SbFHD/E9LjuB3NQY3t5+yKuAsv2LOnaXNXDvZbOOeS06IoyHrpvPHc9s4O5/7cDhEK4/cTyltS04jXcjvNyv9aUJaUd1zC9dtY/oCAfXnzie6IgwjstO7LEfxZUU8ur0vv9fqP6LjQwjOsLR4+RGYwz3vfk5nU7DqVPTPeZtC5aP91Sxbn81v7x4BrGR1lf3aVMzOG1qBvkHqtlQWMNVC3JJsXOWBYMGlGFqY2EN97y2g42FtcwcncjvrprDiZPSOHS4hV+8uo373vycFZu+4BsnjuOjPZXc9eXpXnU6u3+J+zqgtLR38s/8Ys6bkcVPv3Ic49J6HzYrIpw/YxRPrTlIfUt7n3NKXtpQQmSYg4s8/PqMCHPwp8Xzuf2ZDfzi1e0IR0Za5fSSx6snp05J597Xd1Jc00REmIOXN5bwtUVjSbX/sc8bm8yL64vpdJqjmiQ0KaR/iQjp8T2vLb+vspGapnYA/rbmILefOTnQxfPIGMNv39nF6KRovvalY//d5Y1PJW988OctaR/KMLSztI4rH1lDcU0z9185mxV3nMKJk9IAq89g6fV5PPL1+VQ0tHLXS1tJionguhPGeXVt15f4x3uqqG9p92m51+ytorm9k+tOGNdnMHE5f9Yo2jqdfLDL87wOsPpiVmwu4azpmSTHev4FFxnu4C9fm885x2Xx81e38+d/7wH6V0MB65cjwEcFlSz7eD+dTsPNp07sen1ubjKNbZ0UlNcfdZ4mhfQ/K6Ac24ey/qC14ubkzHj++tF+mtp8t7bNYL2/s5zNRbV85+wpRIUP3XQ8GlCGoWUf7ScyzMHb3zuNq/Nye+yUu2BWNu/94HS+ddpE7r1sFvFR3ldWz5tpfYl/2MeXeH+9u7OMuMgwTpjo/S+t+WNTSI+P7HOgwOqCSiob2rhift99IZHhDh66bj7nHJfJmn1VhDmE7KRor8sEMCUznqzEKF7fWsozawv5yuzRR40UmzfW6iPp3uy1t7wREU0K6U+eaigbDtaQFBPBfVccT3VjG8+sKxz0e7W0d/Lbtz9nzd6Bz853Og2/e2cX49JiB53Sx9/8FlBEZJmIlIvINrd9qSLyrogU2H9T7P3XicgWEdkqIp+IyBwP13xSRPaLyCb7Mddf5Q9VlQ2tvLr5C766YExX84onSTER3HXhcVw8Z3S/3mPBuBTS4vr+Eu8Pp9Pw/s4yTp+W0a9fYGEO4dwZWXz4eTmtHZ6TWL60sYSU2AjOmJbp1XUjwx385br5nDcji9kDWDVRRDh1SgarCyqpb+3gW6dNPOr18WmxJMdGsLHbOvT7KhsYnaRJIf0pIyHSYw1l/thk8sanctKkNJau2jfoxKivbSnlLx/sZfFja/n64+vYVFTb72u8vrWUzw/V8/1zpvY5aCbY/Fm6J4ELuu27E3jfGDMFeN9+DrAfON0YczzwP8DSXq77Y2PMXPuxybdFDn3PrLPGpd9w0gS/vUfXl/iuimO+xCsbWnl89T5qGvs3LHPbF4cpq2vlnOOy+l2e82aOorGtk0/29PwrsK6lnXe2H+LiOaOJDPf+Ix8VHsaj31jAi7ed1O8ygdWPAnDK5HRmjUk66jURYW5u8jFfMJoU0v/S4qKobmw9amLg4aZ2CsobWDDOqjnecdZkyutbeb6XjAbeeGF9EePSYvnZV45jR2kdl/3lY25ens9OD6l5uuvodPLAe7uZkhnf7x9+weC3gGKMWQV0Xw/1UmC5vb0cuMw+9hNjjOun2lpgaNfrAuDzQ3W8u6N/aU7aOpz8be1BTp+aweRM/7bBnz9zFA2tHV1f4h2dTp78eD9n/u5D7n19J197fB3V/Qgq7+0sxyFwppc1CHcnTUojPircY43pra2HaO1w9jrHxhMRGXCSxzOmZjJvbDLfP3dqj6/Py02hoLyhqy/KlRRSZ8j7V3p8JE4DNU1HPp8biqyvn/l2QDlxYhoLxqXwyMp9R00c7I/CqibW7qvmqgU53HzqRFb915n88NyprNtfxYUPruaWp/JZu6+qx7lhAK0dnTy2ej/7Khr54XlTAzqfZKACXX/KMsa48gAcAnr6OXoT8GYv1/i13Tz2gIh4nAghIreKSL6I5FdU+Lat39+cTsPtT2/glqfy+ffn3geV17d+QUV9Kzee4r/aictJk498iX92oJqL/vQRd/9rB3Nzk7n/q7PZV9HA1x5b6/X63e/tKCNvXOqAhjxGhYdxxrQM3t1R1mM6ihc3FDMxPY65ucn9vvZgJMVG8PK3T+761dvd3LHJGANbiq1cYpoUMjBca8u796NsOFhDmEO6PiMiwh1nTaaktplXNpYM6H1e2FCMCFwx3/p9HB8VznfOnsLq/zqTO86cTP6Baq5dupaL/vQRL64vprWjk/ZOJyt3V/Dj5zez8N73+M1bn7NwfEpXloqhLmgNcsYKy0f96xeRM7ECyk88nHYXMB1YCKT2chzGmKXGmDxjTF5GRoZvCh0g7+ywkjSmxEbwvX9s8ip3ljGGJz4+wKSMOE6zm1r8yfUl/tKGEq56ZA11ze08fN18nrpxEVcvzGXZDQs5UNXI4sfW9jmTvaS2mR2ldZwzo/+1E5fzZ46iqrGta6QOwOHmdn768lbW7a/mqwtyhtzM87k5yQBd/Sh7NYdXQPQ0W379wRpmZCd2ze8AOGNqBrPGJPLQh3vo6Gd2YqfT8OL6Yk6ZnM7o5KOHnCfHRvLD86bxyZ1n8/+uOJ62Dic/fH4zJ9/3AYt+/R5Lln3Km9sOcc5xWTxxw0KevvmEIffZ9STQAaVMRLIB7L/lrhdEZDbwOHCpMabHxnBjTKmxtAJPAIsCUOaAMsbw0Id7GZcWy0vfPhmA2/6+vs/OwfUHa9hSfJgbTp4QsA/fVXm5OBxw+5mTeO+Hp/Pl47O73vvkyeksu2EhhdVNLH5srce8VQDv77RqYQPpP3E5Y1qGNbJt+yGMMby+pZRzfr+SZz8t5MaTJ3BTAGpt/ZUUG8GkjLiukV77NMtwQHSfLd/R6WRTUe0xNUkR4Y4zp3CgqqnfCRbX7quipLa511FZMZFhLF40lne+fxpP3biIBeOSOWVKBo9+YwH5PzuH318zlzOnZ/ar3y/YAj2xcQWwBLjP/vsqgIiMBV4CvmGM2e3pZBHJNsaUivWtdRmwzdOxoerjPVVsKT7Mry+fxYT0OB64Zi43Lc/n7hXbue+rsz2e98THB0iMDuerXgyL9ZXTp2aw854LPAawkyal8+Q3F/HNJz7j2qVr+cctJ5CZeOzw23d3lDExI25QX6QJ0RGcPDmNN7eWcqCykfc/L2fm6ESWLVnI8TlJfV8gSObmpvDhrnKMMZoUMkAy7IDiqjl/fqieprbOrv4Td+fNyGJaVgL3/GsHDa0dXJ2X69VIq+fXF5MQHe5VU5WIdM14D3X+HDb8LLAGmCYixSJyE1YgOVdECoBz7OcAvwDSgIfs4cD5btd5Q0RcwxueFpGtwFYgHbjXX+UPloc+3ENGQhRftdtdzz4uizvOnMw/Pivin5/1POKkpLaZt+z0I+5V9kDoqzZ0wsQ0lt+4iEOHW7jlb8fWtOpb2lm7r4pzB1E7cTl/5ii+ONzCJ3ur+OmFx/Hq7ScP6WAC1oz5qsY2iqqbNSlkgCTGhBMRJl1DhzfYTY499XU5HMKDi+cxIT2On768jfMeWMXrW0o9dqSD9Zl+c1spF88Z7VX2ieHEb98+xpjFHl46u4djbwZu9nCdC922z/JN6YamTUW1fLK36pg0KN8/dyobi2r4+avbmDE68ZghqE+tOQDA9SeND2BpvbdoQiq/v3oOt/19Az97ZRu/vfLIuvSrCypp7zScM2PwAeWyeWOoaWrnotnZ/Uo3H0yuTuCNRTWaFDJARIS0uCOTG9cfrGFUYjSjPUxenTYqgedvO5H3d5Zz/9ufc/szGzh+TBJ3XTidkyYd21/5+pZSWtqdXDXEJyH6g+byGkIe+mAPidHhx6RBCXMID147j4v+9BHXPb6OmaMTGZUUTXZSNKOSYnh2XSHnz8xiTHL/8k0F0gWzsvnuWZN58N97OH5MEkvs4PfejjJSYiOYP3bwX6TREWH8xxmTBn2dQJo+KoHoCAdr9lpt7tdkaFLIQEhPiOwagbj+YA0LxqX0WjMUEc6ZkcWZ0zN5eWMJD7y7m689to4fnTf1mBVOn19fzOTM+ICPKhwKNKAMEQVl9byzo4zvnDW5xzQoafFR/HXJQh5euZcvaptZt6+aQ3UtXcNkh2Knc3ffO2cqO0rruOe1HUzNSmDh+BT+vaucs6ZnhsQYe38ID3MwOye5q9NXJzUGhiufV1ldC8U1zXzzZO/+/YQ5hCsX5HDR7Gzuemkrv3tnNwXlDfzmq7OJjghjb0UD6w/WcNeXp4/IpksNKEPEwyv3Eh3h4IZemq1mjE7kT4vndT3vdBoqG1ppae/0OpliMDkcwgPXzOWyv3zM7c9s4GdfOY7apnaf9J+Esnm5yXy635oDrEOGAyM9Popdh+q7hpl7mivkSXREGL+/eg6TM+P57du7OFjVxNLrF/Di+mLCHDKgSbTDgQaUAPtgVzmHm9qZnBnPpIx4YiLDKK5pYsWmL/j6CeNI68eiVWEOIauHUVNDWUJ0BEuvz+OyP3/Mj57fTGSYg1OHweiWwZg3NhlAk0IGUFp8JFUNbeQfqCEq3MGM7MR+X0NEuP3MyUzKiOf7z23isj9/TFun4fSpGT2OZhwJNKAE0PJPDvDLFdu7notYadHDw6yq8S3dEggOV5My4vnjYms49Al22pSRbG6u9et4dFLMiBsVFCwZ8VF2xuxy5uQkD2quxwWzRpGTciK3PJVPZUPriOyMdxnZ/5ID6JWNJfxyxXbOnZHFj8+fxp7yBgrKGigor2dPeQO3nT5pSHeq+9pZ07NYdsNCxobIaCx/GpVkjTCakqXNXYHimty4r7Kxx2WY+2vWmCReveNk/r2z3CfXC1UaUALgvR1l/PD5zZw4MY0/LZ5HdEQYU7MS4Phglyy4BpIIcrh6+OsLiI/Wf46Bku7WtNzf/hNPMhOi/bIsdijRT7Cfrd1Xxe3PbGDm6EQeW5KnTRqqR3NG4BDTYEpPOJKEdL7dh6UGL3SSxISgbSWHuXl5PjkpMTz5zUUjvq9AqaHCVUOZkB7Xr4Ewqnf6DedDtU1tbC4+zJaiWjYXH2btviqSYiL4+81f6nP1RKVU4KTERhLmEJ9MqFVHaEDpp06nYUNhDUXVTZTUNFNSaz0OVjVR6JZmfmJGHOfNzOI/z55CdtLI6WxXKhSEOYT/u2oOs4d4rrdQowGln/625gB3/2tH1/P0+CjGpMRwfE4SixeNZU5OErNykkiMjghiKZVSfblshE4+9CcNKP30/uflTEyP4/EleYxO1nkDSinlop3y/dDc1sm6/dWcOT2TiRnxGkyUUsqNBpR+WLu/irYOJ6eP8FQhSinVEw0o/bByVwXREQ4WTUgNdlGUUmrI0YDSD6t2V3DCxDRt6lJKqR5oQPFSYVUT+yobtblLKaU80IDipZUFFQAaUJRSygMNKF5auauC3NQYXa9CKaU88GtAEZFlIlIuItvc9qWKyLsiUmD/TbH3i4g8KCJ7RGSLiMz3cM0FIrLVPu5BCcA6m20dTtbsreS0KRkjcllPpZTyhr9rKE8CF3TbdyfwvjFmCvC+/Rzgy8AU+3Er8LCHaz4M3OJ2bPfr+9z6gzU0tnVqc5dSSvXCrwHFGLMKqO62+1Jgub29HLjMbf9TxrIWSBaRbPcT7eeJxpi1xhgDPOV2vt+s3F1BuEM4aXK6v99KKaVCVjD6ULKMMaX29iEgy94eAxS5HVds73M3xt7f2zEAiMitIpIvIvkVFRWDKvDK3RXkjU/R9PNKKdWLoHbK27UM46drLzXG5Blj8jIyBt5UVVbXws7SOk6fqqsLKqVUb4IRUMpcTVn233J7fwmQ63Zcjr3PXYm9v7djfGrVbh0urJRS3ghGQFkBLLG3lwCvuu2/3h7tdQJw2K1pDAD7eZ2InGCP7rre7Xy/WLm7goyEKI7LTvDn2yilVMjz97DhZ4E1wDQRKRaRm4D7gHNFpAA4x34O8AawD9gDPAZ82+06m9wu+23gcfu4vcCb/ip/p9Pw0R4dLqyUUt7way+zMWaxh5fO7uFYA9zu4Tpz3bbzgVm+KF9fthTXUtvUzunTtLlLKaX6ojPle7FydwUicKoOF1ZKqT5pQOlFdlI0Vy3IISUuMthFUUqpIU8nVvTimoVjuWbh2GAXQymlQoLWUJRSSvmEBhSllFI+oQFFKaWUT2hAUUop5RMaUJRSSvmEBhSllFI+oQFFKaWUT2hAUUop5RNipdAa3kSkAjg4wNPTgUofFifYhtP9DKd7Ab2foWw43Qt4fz/jjDFeJzMcEQFlMEQk3xiTF+xy+Mpwup/hdC+g9zOUDad7Af/djzZ5KaWU8gkNKEoppXxCA0rflga7AD42nO5nON0L6P0MZcPpXsBP96N9KEoppXxCayhKKaV8QgOKUkopn9CA0gsRuUBEdonIHhG5M9jlcRGRZSJSLiLb3Palisi7IlJg/02x94uIPGjfwxYRme92zhL7+AIRWeK2f4GIbLXPeVBExI/3kisiH4jIDhHZLiL/GeL3Ey0in4rIZvt+fmXvnyAi6+wyPCcikfb+KPv5Hvv18W7Xusvev0tEznfbH9DPpYiEichGEXltGNzLAfuzsElE8u19IflZs98vWUReEJHPRWSniJwY1PsxxuijhwcQBuwFJgKRwGZgRrDLZZftNGA+sM1t3/3Anfb2ncBv7O0LgTcBAU4A1tn7U4F99t8UezvFfu1T+1ixz/2yH+8lG5hvbycAu4EZIXw/AsTb2xHAOvu9/wlca+9/BPgPe/vbwCP29rXAc/b2DPszFwVMsD+LYcH4XAI/AJ4BXrOfh/K9HADSu+0Lyc+a/X7LgZvt7UggOZj347cbDfUHcCLwttvzu4C7gl0ut/KM5+iAsgvItrezgV329qPA4u7HAYuBR932P2rvywY+d9t/1HEBuK9XgXOHw/0AscAG4EtYs5LDu3+2gLeBE+3tcPs46f55cx0X6M8lkAO8D5wFvGaXLSTvxX6PAxwbUELyswYkAfuxB1cNhfvRJi/PxgBFbs+L7X1DVZYxptTePgRk2due7qO3/cU97Pc7u4lkHtav+pC9H7uJaBNQDryL9Su81hjT0UMZusptv34YSKP/9+kvfwD+C3Daz9MI3XsBMMA7IrJeRG6194XqZ20CUAE8YTdJPi4icQTxfjSgDEPG+jkRUuPBRSQeeBH4njGmzv21ULsfY0ynMWYu1q/7RcD04JZoYETkIqDcGLM+2GXxoVOMMfOBLwO3i8hp7i+G2GctHKvp+2FjzDygEauJq0ug70cDimclQK7b8xx731BVJiLZAPbfcnu/p/vobX9OD/v9RkQisILJ08aYl+zdIXs/LsaYWuADrKadZBEJ76EMXeW2X08Cquj/ffrDycAlInIA+AdWs9cfCc17AcAYU2L/LQdexgr4ofpZKwaKjTHr7OcvYAWY4N2PP9srQ/mBFf33YVUrXR2GM4NdLrfyjefoPpTfcnRH3P329lc4uiPuU3t/Klb7a4r92A+k2q9174i70I/3IcBTwB+67Q/V+8kAku3tGGA1cBHwPEd3ZH/b3r6dozuy/2lvz+Tojux9WJ3YQflcAmdwpFM+JO8FiAMS3LY/AS4I1c+a/X6rgWn29t32vQTtfvz6IQz1B9aoiN1YbeA/DXZ53Mr1LFAKtGP9SrkJq636faAAeM/tAyHAX+x72ArkuV3nRmCP/fim2/48YJt9zp/p1unn43s5BatKvgXYZD8uDOH7mQ1stO9nG/ALe/9E+x/nHqwv5Ch7f7T9fI/9+kS3a/3ULvMu3EbXBONzydEBJSTvxS73Zvux3fV+ofpZs99vLpBvf95ewQoIQbsfTb2ilFLKJ7QPRSmllE9oQFFKKeUTGlCUUkr5hAYUpZRSPqEBRSmllE9oQFGqFyLSaWemdT16zYgrIreJyPU+eN8DIpI+gPPOF5Ff2Rln3xxsOZTqj/C+D1FqRGs2VhoVrxhjHvFjWbxxKtbs/FOBj4JcFjXCaA1FqQGwaxD322tFfCoik+39d4vIj+zt74q1zssWEfmHvS9VRF6x960Vkdn2/jQReUesNVQex5qE5nqvr9vvsUlEHhWRsB7Kc42dkPK7WAkdHwO+KSIr/PyfQqkuGlCU6l1Mtyava9xeO2yMOR5rBvEfejj3TmCeMWY2cJu971fARnvff2OlnQH4JfCRMWYmVo6psQAichxwDXCyXVPqBK7r/kbGmOewMjVvs8u01X7vSwZ+60r1jzZ5KdW73pq8nnX7+0APr28BnhaRV7DSYoCVauarAMaYf9s1k0SsRdOusPe/LiI19vFnAwuAz+zF8mI4kuyvu6lYubEA4owx9X3dnFK+pAFFqYEzHrZdvoIVKC4Gfioixw/gPQRYboy5q9eDrOVs04FwEdkBZNtNYN8xxqwewPsq1W/a5KXUwF3j9neN+wsi4gByjTEfAD/BSuUej5Ud9jr7mDOASmOt/7IK+Jq9/8tYSf7ASvJ3pYhk2q+lisi47gUxxuQBrwOXYi0B+1NjzFwNJiqQtIaiVO9i7F/6Lm8ZY1xDh1NEZAvQirU8qrsw4O8ikoRVy3jQGFMrIncDy+zzmoAl9vG/Ap4Vke1YadULAYwxO0TkZ1irDDqwMkzfDhzsoazzsTrlvw38fhD3rNSAaLZhpQbAXnQqzxhTGeyyKDVUaJOXUkopn9AailJKKZ/QGopSSimf0ICilFLKJzSgKKWU8gkNKEoppXxCA4pSSimf+P9lH5loU1NEfAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "avg_scores = aggregate_list(trainer.scores, trainer_args[\"print_range\"])\n",
    "x = [i*trainer_args[\"print_range\"] for i in range(len(avg_scores))]\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(x, avg_scores)\n",
    "plt.ylabel(\"Average Score\")\n",
    "plt.xlabel(\"Episode #\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 Agent (Replay Memory) - Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEVCAYAAAAb/KWvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAApWklEQVR4nO3deXxcVf3/8debthS0yNaCyFZUZNeqFfGLIovIpha/oIKCgCh+Ffy6CwhfRRFFREB+IlgW2WQTASurlC5sbSEsLaUFmm40bWnTlu5tmuXz++PetNN0kpkkk9nyfj4e88idc8/c+WSS+cyZc849VxGBmZlVl81KHYCZmRWek7uZWRVycjczq0JO7mZmVcjJ3cysCjm5m5lVISd3swKRdJGk20sdhxk4ufcKkmZJWidpYJvylySFpMHp/V0k/VPSIknLJE2WdHq6b3Bad2Wb21faec4xkr7Z079bZ0k6VFJLGvsKSa9LOqPUcXVE0s3paz+sTfmVafnpJQrNypiTe+8xEzi59Y6kA4B3tKlzGzAH2B3YHjgVWNCmzjYRMSDjdncPxtxT5kXEAOBdwA+B6yXtVeKYcnkD+HrrHUl9gS8D00sW0caxWJlxcu89biMjOQCnAbe2qfMx4OaIWBURTRHxUkQ8UsggJG0m6UJJsyUtlHSrpK3TfVtIul3SYklLJT0vacd03+mSZqSt7ZmSvtbdWCLxMLAE+GBGfOdJmp7GcY+k7dJ9rd9ezpI0T9J8ST/p4Hf9h6S30m9BT0raLy3/mKQFkvpk1P1vSRM7CPffwCclbZvePxqYBLzV5jm/IWmqpLclPSZp94x9Iem7kqalr+PFkt4n6VlJy9PfdfOM+t+SVCtpiaQRkt7T5lhnS5oGTJN0jaQ/tollhKQfdvA7WQ9ycu89xgPvkrRPmlROAtr2D48HrpF0kqTdeiiO09PbYcB7gQHAn9N9pwFbA7uSfHP4H2CNpHcCVwPHRMRWwH8BL3c3kDSRfwEYCNSmxd8Djgc+DbwHeBu4ps1DDwP2BD4LnCvpM+08xSNpvR2AF4G/A0TE88Di9PGtTmXTD9tMa4F/kfzdIPmg3qh+2m3zc+C/gUHAU8CdbY5zFPBR4CDgZ8Bw4BSS13x/0m93kg4Hfkfy7WAnYDZwV5tjHQ98HNgXuAU4WdJm6eMHAp8B7ujgd7KeFBG+VfkNmEXyRruQ5A17NPA40BcIYHBab1vgUuBVoJkkgX4s3Tc4rbu0zW2fdp5zDPDNLOVPAN/NuL8X0JjG8g3gWeCDbR7zzvS5TgC27OZrcSjQkh6vIf09f5CxfypwRMb9nTLia30N9s7YfxlwY7p9EXB7O8+7TfrYrdP75wJ/T7e3A1YDO7Xz2JuB3wCfBMalx1oAbAk8DZye1nsEODPjcZulx909vR/AwRn7XwDOzbj/R+CqdPtG4LKMfQPS12FwxrEObxPnVODIdPsc4OFS/+/35ptb7r3LbcBXSVrOm7QSI+LtiDgvIvYDdiRJ7g9IUka1gRGxTcZtaidjeA9JK7DVbJLEuWMa32PAXWm3x2WS+kXEKuArJC35+ZIekrR3toO3Gext79vHvIjYhqTP/Wrg8Ix9uwP3p91CS0kSVnMaX6s5beJ/D21I6iPp0rR7ZznJBywk3xIg+db0+fRbyZeBpyJifjvxAhART5O0yC8AHoyINW2q7A78KSP2JYCAnTPqZI6hrMlyf0C6vdHfKSJWknzbyDxW5usASev9lHT7FJK/p5WIk3svEhGzSQZWjwXuy1F3EXA5yZt8uwKGMY8kCbXaDWgCFkREY0T8KiL2Jel6+RzpOEFEPBYRR5K0pF8Drm8n7szB3jc7CiQiGkha0AdIOj4tnkPS/ZP5AbZFRMzNeOiubeKfl+XwXwWGkXxj2pqk1Q9JsiU93jiSLpRTyT8R3g78mOxdOHOAb7eJfcuIeDbPY2fa6O+UfghtD2S+Dm2XlL0dGCbpQ8A+wANdeF4rECf33udMkq/Tq9rukPR7SftL6itpK+A7QG1ELO7ic/VNB0lbb/1I+oB/KGkPSQOA3wJ3R0STpMMkHZCOCSwn6QZokbSjpGFpgmkAVpJ0rXRbRKwj6Y74RVp0HXBJ60CkpEFqMwUR+D9J70gHSM8Ass0Y2iqNdTHJrKTfZqlzK0m/9wHk+LDNcDVwJPBkln3XAednDNxuLelLeR63rTuBMyQNkdSfJP4JETGrvQdERB3wPMkH1T+zfLOwInJy72UiYnpE1LSz+x3A/ST90TNIWm5faFNnaZuujx918HTXknzVb739DbiJ5M3/JMm3iLUkg5gA7wbuJUnsU4Gxad3NgB+RtCaXkAx2fifPXzkfNwG7Sfo88CdgBPAfSStIBpk/3qb+WJIB2CeAyyPiP1mOeStJt8ZcYEp6nLbuJ+0GiojV+QQaEUsi4omI2ORCDBFxP/B7km6t5cBk4Jh8jpvlWCOB/wP+CcwH3seGwdyO3ELyYeUumRJTlv8RM8tCycleM4F+EdFUoGNOJ+lKGVmI45WapENIumd2z/YBZMXjlrtZiUg6gaTfelSpYymEtNvt+8ANTuyl5zPLzEpA0hiS+eGnRkRBxg9KSdI+QA0wkWQcwkos726ZdJCrBpgbEZ+TtAfJSQ3bk8yXPTUi1qWDL7eSnCixGPhKR4MwZmZWeJ3plvk+ySBXq98DV0bE+0nO4jszLT8TeDstvzKtZ2ZmRZRXy13SLiSj4JeQzFr4PFAPvDudwvYJ4KKIOErSY+n2OCULCr0FDOqoD27gwIExePDg7v82Zma9yAsvvLAoIgZl25dvn/tVJPNxt0rvbw8szZgxUMeGM9d2Jj1zLU38y9L6izIPKOks4CyA3XbbjZqa9mbnmZlZNpJmt7cvZ7eMpM8BCyPihUIGFRHDI2JoRAwdNCjrB4+ZmXVRPi33g4EvSDoW2IJkPY4/AdtI6pu23ndhw2nJc0lOz65Lu2W2JhlYNTOzIsnZco+I8yNil4gYTHKG2qiI+BowGjgxrXYayXKkkJzdd1q6fWJa33NezcyKqDsnMZ0L/EhSLUmf+o1p+Y3A9mn5j4DzuheimZl1VqdOYoqIMSTrdBMRM4ADs9RZC3R1sSIzMysALz9gZlaFnNzNzKqQk7uZWTseeGkuKxsKsgBo0Tm5m5llMXHOUn5w98tceP8rpQ6lS5zczcyyWJW22BcsbyhxJF3j5G5mVoWc3M3MqpCTu5lZFXJyNzOrQk7uZmZVyMndzKwKObmbmWVR6UvZOrmbmXVAKnUEXePkbmZWhZzczcyqkJO7mVkVcnI3M6tCOZO7pC0kPSdpoqRXJf0qLb9Z0kxJL6e3IWm5JF0tqVbSJEkf6eHfwcys4Cr9ys/5XGavATg8IlZK6gc8LemRdN9PI+LeNvWPAfZMbx8Hrk1/mplVnOVrG1nX1MLmfSuroyNntJFYmd7tl946+kwbBtyaPm48sI2knbofqplZ8USa5ibPXc6Ztzxf4mg6L6+PIkl9JL0MLAQej4gJ6a5L0q6XKyX1T8t2BuZkPLwuLWt7zLMk1Uiqqa+v7/pvYGbWw56atqjUIXRaXsk9IpojYgiwC3CgpP2B84G9gY8B2wHnduaJI2J4RAyNiKGDBg3qXNRmZj2sobGl1CF0S6c6kSJiKTAaODoi5qddLw3A34AD02pzgV0zHrZLWmZmVjGWrWksdQjdks9smUGStkm3twSOBF5r7UeXJOB4YHL6kBHA19NZMwcByyJifg/EbmZm7chntsxOwC2S+pB8GNwTEQ9KGiVpECDgZeB/0voPA8cCtcBq4IyCR21mZh3KmdwjYhLw4Szlh7dTP4Czux+amZl1VWVN3DQzs7w4uZuZZVHhJ6g6uZuZVSMndzOzKuTkbmZWhZzczcyyiApfFtLJ3cysCjm5m5lVISd3M7Mq5ORuZlaFnNzNzKqQk7uZWRVycjezTTS3BP9750tMmbe81KFYFzm5m9kmZtSvZMTEefzvXS+VOhTrIid3M7Mq5ORuZpZFZZ+f6uRuZlaV8rmG6haSnpM0UdKrkn6Vlu8haYKkWkl3S9o8Le+f3q9N9w/u4d/BzHpIpV8kujfLp+XeABweER8ChgBHpxe+/j1wZUS8H3gbODOtfybwdlp+ZVrPzCpQ/YqGUodQOhXeL5MzuUdiZXq3X3oL4HDg3rT8FuD4dHtYep90/xGSVKiAzcyKISo8u+fV5y6pj6SXgYXA48B0YGlENKVV6oCd0+2dgTkA6f5lwPZZjnmWpBpJNfX19d36JczMbGN5JfeIaI6IIcAuwIHA3t194ogYHhFDI2LooEGDuns4MzPL0KnZMhGxFBgNfALYRlLfdNcuwNx0ey6wK0C6f2tgcSGCNTOz/OQzW2aQpG3S7S2BI4GpJEn+xLTaacC/0u0R6X3S/aOi0i9pYmZWYfrmrsJOwC2S+pB8GNwTEQ9KmgLcJek3wEvAjWn9G4HbJNUCS4CTeiBuM7MeJSp7HkjO5B4Rk4APZymfQdL/3rZ8LfClgkRnZmZd4jNUzcyy6BVTIc3MrLI4uZuZVSEndzPbRGV3SBg4uZuZVSUndzOzKuTkbmZWhZzczcyyqPTz6p3czcyyqPDc7uRuZpaNW+5mZlZ2nNzNbBMr1vraqW0tW11Zr4mTu5lt4qxbXyh1CCXX9uKgFz80pTSBdJGTu5ltYvGqdaUOoew0NreUOoROcXI3M6tCTu5mZlXIyd3MrArlcw3VXSWNljRF0quSvp+WXyRprqSX09uxGY85X1KtpNclHdWTv4CZWU+o9Hnu+VxDtQn4cUS8KGkr4AVJj6f7royIyzMrS9qX5Lqp+wHvAUZK+kBENBcycDMza1/OlntEzI+IF9PtFcBUYOcOHjIMuCsiGiJiJlBLlmutmplZz+lUn7ukwSQXy56QFp0jaZKkmyRtm5btDMzJeFgdWT4MJJ0lqUZSTX19fecjNzOzduWd3CUNAP4J/CAilgPXAu8DhgDzgT925okjYnhEDI2IoYMGDerMQ83MelyvuEC2pH4kif3vEXEfQEQsiIjmiGgBrmdD18tcYNeMh++SlpmZWZHkM1tGwI3A1Ii4IqN8p4xqXwQmp9sjgJMk9Ze0B7An8FzhQjaznvTQpPmlDsEKIJ/ZMgcDpwKvSHo5Lfs5cLKkISTLHs8Cvg0QEa9KugeYQjLT5mzPlDGrHGff8WKpQ7ACyJncI+JpQFl2PdzBYy4BLulGXGZm1g0+Q9XMrAo5uZuZVSEndzOzKuTkbmaWRaWvLePkbmZWhZzczczyUGkteSd3M7Mq5ORuZpZFhTXUN+HkbmZWhZzczcyyqbRO9jac3M3M8lBpqd7J3cysCjm5m5lVISd3M+vVaheu5K9jp2+6Q9kWw60c+aznbmZWtU649lmWrWnk9IMH079vn1KHUzBuuZtZr7ZsTWP2HdU+W0bSrpJGS5oi6VVJ30/Lt5P0uKRp6c9t03JJulpSraRJkj7S07+EmZltLJ+WexPw44jYFzgIOFvSvsB5wBMRsSfwRHof4BiS66buCZwFXFvwqM3MrEM5k3tEzI+IF9PtFcBUYGdgGHBLWu0W4Ph0exhwayTGA9u0uZi2mVnF+ffEeaUOoVM61ecuaTDwYWACsGNEtF4m/S1gx3R7Z2BOxsPq0jIzMyuSvJO7pAHAP4EfRMTyzH0REXTyBC5JZ0mqkVRTX1/fmYeamRXcwuUNpQ6hoPJK7pL6kST2v0fEfWnxgtbulvTnwrR8LrBrxsN3Scs2EhHDI2JoRAwdNGhQV+M3MyuIz//56Y3uT5m/vJ2alSGf2TICbgSmRsQVGbtGAKel26cB/8oo/3o6a+YgYFlG942ZWVlaunrjKZF3PjennZqVIZ+TmA4GTgVekfRyWvZz4FLgHklnArOBL6f7HgaOBWqB1cAZhQzYzMxyy5ncI+JpoL3zcI/IUj+As7sZl5mZdYPPUDUzq0JO7mZmVcjJ3cw6NLXCZ430Vk7uZtahV+c5uVciJ3czsyrk5G5mVoWc3M3MqpCTu5lZFXJyN7MONTa3lDoE6wIndzPr0HXZLh5tZc/J3cw6tGTVulKHUBHWNbVw13Nv0tJSHtdezWfhMDPrzcojVxVFRJAshNt5146ZzpUj36Bfn8044aO7FDiyznPL3cysAN5enXzDWb62MUfN4nByNzOrQk7uZmYFFGXSjeXkbmZWhZzczaxDZdIQLXs3PzsLgHVlcl5APtdQvUnSQkmTM8oukjRX0svp7diMfedLqpX0uqSjeipwM7NytLqhqdQhAPm13G8Gjs5SfmVEDElvDwNI2hc4CdgvfcxfJPUpVLBmZpafnMk9Ip4EluR5vGHAXRHREBEzSS6SfWA34jMzsy7oTp/7OZImpd0226ZlOwNzMurUpWWbkHSWpBpJNfX19d0Iw8x6UpTL9I8KUS6vVleT+7XA+4AhwHzgj509QEQMj4ihETF00KBBXQzDzHpaV8/Y7K1ayuTDsEvJPSIWRERzRLQA17Oh62UusGtG1V3SMjOrUKvXlccAYaVoaKyQ2TLZSNop4+4XgdaZNCOAkyT1l7QHsCfwXPdCNLNSKpN1sIqiEL/rDU/P7P5BCiDnwmGS7gQOBQZKqgN+CRwqaQhJ99Is4NsAEfGqpHuAKUATcHZENPdI5GZm3XDF429w9RPTNiq74akZfPvT7ytRRIWVM7lHxMlZim/soP4lwCXdCcrMrKe1TewAI6cuqJrk7jNUzcxSr81fUeoQCsbJ3cwsVU3DC07uZmapXHP6y2Wt9nw4uZuZpXK13MvlEnr5cHI3M6tCTu5mZlXIyd3MrAo5uZuZpVava2bh8rWlDqMgnNzNzDL8eXRtqUMoCCd3M7MMt46b3e6+MlnwMS9O7mZmVcjJ3cysjblL12Qtv/ihKUWOpOuc3M3M2jj40lFZy+97sXIuT+HkbmZWhZzczcyqkJO7mVkVcnI3M6tCOZO7pJskLZQ0OaNsO0mPS5qW/tw2LZekqyXVSpok6SM9GbyZmWWXT8v9ZuDoNmXnAU9ExJ7AE+l9gGNILoq9J3AWcG1hwjQzs87Imdwj4klgSZviYcAt6fYtwPEZ5bdGYjywjaSdChSrmVnJrWtqKXUIeelqn/uOETE/3X4L2DHd3hmYk1GvLi3bhKSzJNVIqqmvr+9iGGZmxTWpbmnOOk3Npf8A6PaAaiTXper0igsRMTwihkbE0EGDBnU3DDOzsnHHc2+WOoQuJ/cFrd0t6c+FaflcYNeMerukZWZmVWHk1IU566xY21SESDrW1eQ+Ajgt3T4N+FdG+dfTWTMHAcsyum/MzCrehJmLSx1CXvrmqiDpTuBQYKCkOuCXwKXAPZLOBGYDX06rPwwcC9QCq4EzeiBmM7OyJpU6gjySe0Sc3M6uI7LUDeDs7gZlZlauKmVNd5+hamZWhZzczcwKTJS+X8bJvRdZunodMxetKnUYVoGaWyqkL6IIaheuzFknOj87vOCc3HuRo656ksMuH1PqMKwCPTez7UnqvdfKhtzTHC979PUiRNIxJ/deZMHyhlKHYGZF4uRu1kvd/1Idw5+cnlfdQkzta2hq5rJHX2P1utKf4NMb5JwKaWbV6Yd3TwTg0clv8aeTPsyu272j3bqFGB68Y8Kb/GXMdDaT+MlRexXgiNYRt9ytYk1bsIKLH5xCVMrE4zL14ptL+fOo2g7rqJNN93VNLdz/Uh0tLcGNT89kzbrm9aspNjQ1dznWcvH0tEWlDiEnJ3erWCdeN44bn57JW8vXljqUipcrd3e2W+aqkW/ww7sn8qN7XubiB6ewzy8epamKZtw89upbpQ4hJyf3XuLFN99ev33Mn54qYSTd9/pbK7jwgVdYtqYRSOYUX/Gf1xl83kMljqxy5UzunTxe6+D9/GUbPnjrVyRl1fBFqxymOubiPvcKsnxtIzPqVzFk12069biFK9by3395dv39qfOXFziy4jrqqic3uj/1reVcnaNbwXLpOH0XYkC1HNZbKZRK+IByy72CnHLDBI6/5plO91mubqj8Ps6OnPG350sdQsXrqcSbLQdWQF6sCk7uFWRS3TKgMloNVlly5falqxs7dbzWbosqaqxvZMzr5X/1OCf3XqC9z4JP/2E0t4+fXdRYrDzlarm/MndZ5w6Y5Z+udb2VamiczF26ptQh5OTk3gvMX7bpP+Ko1xYwe/FqLnxgcgkisnJTjIWuWj9AKmEwsho4ufewcdMXM2fJ6oIeMyIZJP3nC3U5606cs5SvXj9hk/J783is9R7d6XOvXbiSZ2o3nvft9F16ni3Tw06+fjwAsy49rqDH/eYtNUyqW8anPjCQHbbaImudCTMW85Xh47Pue3H20oLGY5UtV25/pW4Z65pa2Lzvpu3Bz1wxFtj4f7z1xLJqmiFTabrVcpc0S9Irkl6WVJOWbSfpcUnT0p/bFibU3mnWolVZ+/cWpvOIm5rbbyO1l9jBbzrbWK4zUJ94bSG/fXjqRmXLVjdudG7Br/796vqzUK30CtEtc1hEDImIoen984AnImJP4In0vnXRoZeP4eBLR7FwxcZnYW7ovzTrvnw+7F+dt2FQdXr9So5IW+yt/vbMLEZMnJfzONUwoAr5reteSj3R5z4MuCXdvgU4vgeeo9d5pW7DGyuI9V+ju7quihvunTNz0SqmzKvsk7860pkB1en1Kznij2NZtHLTJaRrF65kzOsLszY6uvI/N3LKAu5/qTzHh56aVt7TIbub3AP4j6QXJJ2Vlu0YEfPT7beAHbM9UNJZkmok1dTXl/eLVI5av0Z3tRW0Yq2XXe2Mwy4fw7FXV/ayDa0uf+x1bnhqxkZlnemmW9DBWj7XjZ3O6X97Puv/ZVe6Ar95a8361SvLzeNTFpQ6hA51d0D1kxExV9IOwOOSXsvcGREhKWv6iYjhwHCAoUOHVskXteLpbp/5ijyuJmPFc+69k/iv92/PsCE751U/Iqh7e02Hy/S258+jN12qoae+yWX7RlAtq3iW+6/RrZZ7RMxNfy4E7gcOBBZI2gkg/bmwu0Haptb3uZf5P5jl5+6aOXz/rpfzrn/X83P41GWjeWF2siBc/YqGDq+P29wS/O6RqesX72orn8ZCa6LOpwunte8925z21m+Ny1Y30tScDMC+MHsJkzt7olSJtZT5m6/LyV3SOyVt1boNfBaYDIwATkurnQb8q7tBVqI5S1bzlb+O67Hjrz/br50h1WppHVl2z89Krmk6oz4Z1Pv4b0dy2OVjaGhq5or/vM7axo3XE3q6dhF/HTuD8+97Jevxrn9qZo/0IY+fseHaq9c/NROA+16ayw1PzeBDv/4PP7t3EgAnXDuOz/2/p2lszj7bpnUWTlNzS9n8b5dJGO3qTst9R+BpSROB54CHIuJR4FLgSEnTgM+k93udK0e+wYQuXlT4wUnzGPP6QkZm9OllTjGL2NDSam+J7L9PeLNLz23lp6UlaG4JXqlbxsLla3ntreXc9+JcAG4bP5vmllj/f/C5q5/m6lG1/HXsDB5+ZT5fThsYLWmFppb2pyqeeuNzrFnX/iJzz80q3EWyf/NQMq3yvpfmMm3BivXlfxo5jYamZlas3Xgtm2kLV9DQ1Mz7L3iEyx57nZfnLO0wyT9bu4hz7ngxa523lhVm/f/mMs/uXe5zj4gZwIeylC8GjuhOUL3ZnCWrOeeOlzYp/90jGw1nbPTFuKGpmc0k+vXZjL0ufIQGzzXutvterOP9OwygsTkYOGDz9eVzlqxm6epG+vfbjM9e+SR//uqH+dwH39OpY0cEQ379+Pr16Dty+/jZHS4RMaluGd++7YX196el0/OuHPnG+rLMuei5Frza5xePdrh/1GsL+MbNNR3W6awjr9ywhPOfR9euHxP46O4bTpE57uqn129fO2Y6146Zzu9POICvfGy3rMf86g3JWdnnHr03r7+1gm/eWsNxB+zE4XvvwI//MZE7vvXxbsfd3reMVm+vWse279y8wzo9yWeoFtC6phb+MqaW44fsvL5l1Vmfumx01vI3M5Yw2O+Xj63fvvCBV3imdnGXnqtVzawlDB28HWsbmznnjhf51qfey97vfhdbv6Nfu49Z29jMlPnL+chuyRtw2ZpGtuzXJ+sZjMXU3BL02az9PuGFK9Zy3ZgZ/OzovTjkstFccNw+bP/O/pxyY5IMLvni/hy21w786J7sMzTa/n3OueMlzrnjJab++mi23LxPXjE2NkdeiR3Ia+2fkVOLN2uj0Im9I63jCe2ZtiD3PPPMv9dDr8znoVeSiXzZluTorEl1yxh83kP86gv7sSrLRb9LfeUpJ/c8TJyzlL132or+fTt+817x+BtcN3Y6V42cVqTI6HZih+RydXeddRAnpWe0jpy6YQz8wuP24Zufeu8mj7nwgcnc+0IdT/3sMPr324wDL3kCKPwyC531vp8/zGM/OIS93r1V1v0//cckxr5Rz03PJP2/bQcxL7h/MmN/eminn3fxqgZ22Ty/mSteOKswyuVV/OWIV7OWl/oscC8clsPMRasYds0z/PrfU3LW7WgZ0N89MrVsBoKyOamdpQp+89DUrJeva53ZsHxtIw9Pmr++fPB5D+VscfW0U26cwLz0b/HUtHoeyohvaZ4t5p5Uxv8GFaXcX8d8v531FCf3HJasWgfA2DdyzyTo6IP6r2NnsMf5D29y8kilaLtmSOtJVC0tcFGbD74Trn2W8++bVLTY2qpf0cCZt9TQ1NzCqTc+x9l3vEjd26tpam5Zn/QLLdfaLJnKPSlVinL/BnTEH8duMmupmJzcc2htbde9vSbnAEo+WmcJVJpbx83a6H7r/ORfjsjeJ3znc3NKeq3WNxev4s7n56y//8nfj+b9FzzS7jzvYuooKbXOHFm2upFlnbz6UW9TCR+S37q1eGMUbbnPPYfMMZE9L3gEoN0+3VL3sfWk68ZO36jvfV2a3F98c2m7j3ljwQr22eldPR1aVs0RzH27a630nh4I6ygpZc4cscr31LRFuSv1ELfcc7jr+U3nix911ZPrW65d0fbCBpVg0cp1G/W959Nq6swZl201twTn3juJxVkWp8rH2sYWrhs7vUuP/e7tL3bpcfkq9zMbK0U5j2GVAyf3HNqb0phtLnm+Dfev3dD9aVil0rrMaU/3d97x3JvcXTOHj/5mZI8+TzavZ5xUk6/OfGlzSioMv44dc3Lvoi/34NIC5az1qjs93Whatnrd+u1CjHWUEzc4C6NSvgGVauzEyb2LptdvegJFZ2ZMVLqefl9lHv+g3z7Rs09WAJ3507s7oTAq5WW87smudQ92l5N7AfWe1F54KxuamLt0DWsbm3l08nzWNm2YQrZ41boOHll+Dr50FD/5R/trkFdKUip3lfIyXjumNMnds2W6aG1jdXUVdFahW58f/vV/aOzgerAnDR/HHd88iM06WFqg1B6d/BZH7rsjc5eu4d4X6rj8S5ssvQRUTlIqd/6Q7JiTeze1tCRDi5uJTjXd31y8msaWFh6aNJ9bx81i0cp1/HrYfj0VZsEV+n3VUWKHZOnY9/784QI/a+F84nejctZZuHwtr8xdxnYlXEyquji7d8TJvR2rGpryuozWgb8dyaKVne82OOQPmy4Q9ot/ZV+johzNz3PZ1MHnPcSA/n2Z/Kujejii8hMRXPzgVKYtXMFvjt+fT/9hTKlDqiqtLfcp85YzcKvN2ap/P35670S+8ck9ShtYFisbmhjQv7jptmqTe0tLcNljr/PtQ96bddnNZWsaOf1vz3HVV4aw+/bvBODwy8cwY9EqLj5+f0a/tpBRr3V8Eakf3zOxS4m9t1nZ0MRt42Zx7AE7sf2A/qUOp2hOvG7c+nV2nNgL767n57CmsZl/vTxvo/IHM9YSKhcHXzqK8ecfQUsEi1Y2sK6phaaW4P07DKBfn54Z+lQ5jNwPHTo0amoKe5ruwZeOWr+QV+ZKhReNeJX6FQ3rl/4EOOPgwcxZsqaoS6f2Vt8+5L388MgPsEW/DSts1sxawonX9c6ppWZ/O+NjHLbXDl16rKQXImJo1n3VltwXr2zY5MSXWZcex7qmFo6/5hmmlHC9E9vgos/vy3EffA8SDC3BiUpm5aSrS2WXJLlLOhr4E9AHuCEi2r3cXqGS+98nzOaC+3Nf3MDMrJz0RHLvkT53SX2Aa4AjgTrgeUkjIiL3ouidMGfJaka9tpAP7LgVJ1+ffT1yM7PeqKcGVA8EatPrrCLpLmAYUNDkPm764navgmJm1pv11BmqOwNzMu7XpWXrSTpLUo2kmvr63BfCyObwfbo2CGFmVi5+fuzePXLckk2FjIjhwHBI+ty7coyBA/qX/JqdZmblqKda7nOBXTPu75KWmZlZEfRUcn8e2FPSHpI2B04CRvTQc5mZWRs90i0TEU2SzgEeI5kKeVNEeOTTzKxIeqzPPSIeBsp3pSczsyrm9dzNzKqQk7uZWRVycjczq0JO7mZmVagsVoWUVA/M7uLDBwKLChhOITm2rnFsXePYuqaSY9s9IgZl21EWyb07JNW0typaqTm2rnFsXePYuqZaY3O3jJlZFXJyNzOrQtWQ3IeXOoAOOLaucWxd49i6pipjq/g+dzMz21Q1tNzNzKwNJ3czsypUMcld0tGSXpdUK+m8LPv7S7o73T9B0uAyiu1HkqZImiTpCUm7l0tsGfVOkBSSijYlLJ/YJH05fe1elXRHucQmaTdJoyW9lP5djy1SXDdJWigp65Xglbg6jXuSpI8UI648Y/taGtMrkp6V9KFyiS2j3sckNUk6sZxik3SopJfT98HYvA4cEWV/I1k2eDrwXmBzYCKwb5s63wWuS7dPAu4uo9gOA96Rbn+nnGJL620FPAmMB4aWS2zAnsBLwLbp/R3KKLbhwHfS7X2BWUWK7RDgI8DkdvYfCzwCCDgImFCMuPKM7b8y/pbHlFNsGX/3USSr2Z5YLrEB25Bcf3q39H5e74NKabmvv+B2RKwDWi+4nWkYcEu6fS9whCSVQ2wRMToiVqd3x5NcmaoY8nndAC4Gfg+sLVJc+cb2LeCaiHgbICIWllFsAbwr3d4amFeMwCLiSWBJB1WGAbdGYjywjaSdyiG2iHi29W9Jcd8H+bxuAN8D/gkU6/8MyCu2rwL3RcSbaf284quU5J7zgtuZdSKiCVgGbF8msWU6k6RlVQz5XKj8I8CuEfFQkWJqlc/r9gHgA5KekTRe0tFlFNtFwCmS6khaet8rTmg5dfb/sVSK+T7ISdLOwBeBa0sdSxYfALaVNEbSC5K+ns+DSnaB7N5I0inAUODTpY4FQNJmwBXA6SUOpT19SbpmDiVp5T0p6YCIWFrKoFInAzdHxB8lfQK4TdL+EdFS6sDKnaTDSJL7J0sdS4argHMjoqU4X/g7pS/wUeAIYEtgnKTxEfFGrgdVgnwuuN1ap05SX5KvyovLJDYkfQa4APh0RDQUIa58YtsK2B8Yk/5DvxsYIekLEVFT4tggaXVOiIhGYKakN0iS/fNlENuZwNEAETFO0hYkizwV9St9FmV9cXpJHwRuAI6JiGK8P/M1FLgrfR8MBI6V1BQRD5Q0qkQdsDgiVgGrJD0JfAjoMLkXZcCgAAMOfYEZwB5sGODar02ds9l4QPWeMortwyQDdHuW2+vWpv4Yijegms/rdjRwS7o9kKS7Yfsyie0R4PR0ex+SPncV6bUbTPuDb8ex8YDqc0X+n+sott2AWuC/ihlTPrG1qXczRRxQzeN12wd4Iv2/fAcwGdg/1zErouUe7VxwW9KvgZqIGAHcSPLVuJZkcOKkMortD8AA4B9py+DNiPhCmcRWEnnG9hjwWUlTgGbgp1GE1l6esf0YuF7SD0kGV0+P9J3YkyTdSdJNNTDt7/8l0C+N+zqS/v9jSZLoauCMno6pE7H9gmQc7C/p+6ApirQaYx6xlUyu2CJiqqRHgUlAC3BDRHQ4pRO8/ICZWVWqlNkyZmbWCU7uZmZVyMndzKwKObmbmVUhJ3czsyLLdyGzjPqdXkDPs2XMzIpM0iHASpJ1gPbPUXdP4B7g8Ih4W9IOkcf6Mm65m5kVWWRZLEzS+yQ9mq4f85SkvdNdXVpAz8ndzKw8DAe+FxEfBX4C/CUt79ICehVxhqqZWTWTNIBkvfvWs9gB+qc/u7SAnpO7mVnpbQYsjYghWfZ1aQE9d8uYmZVYRCwnSdxfgvWXS2y9DOEDJK12JA0k6aaZkeuYTu5mZkWWLhY2DthLUp2kM4GvAWdKmgi8yoarfz0GLE4X0BtNngvoeSqkmVkVcsvdzKwKObmbmVUhJ3czsyrk5G5mVoWc3M3MqpCTu5lZFXJyNzOrQv8f5h5Q7T/cOi0AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(agent.losses)\n",
    "plt.title(\"MSE Loss - Replay Memory\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Agent (Prioritized Memory) - Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\torch\\nn\\_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.\n",
      "  warnings.warn(warning.format(ret))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 1000\tAvg Score: 100.63\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 2000\tAvg Score: 100.86\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 3000\tAvg Score: 101.00\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 4000\tAvg Score: 101.00\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 5000\tAvg Score: 101.01\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 6000\tAvg Score: 101.22\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 7000\tAvg Score: 101.50\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 8000\tAvg Score: 92.26\tEpsilon: 0.01\tAvg. MSE Loss: 1.00\n",
      "Episode 9000\tAvg Score: 101.46\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 10000\tAvg Score: 101.66\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 11000\tAvg Score: 101.14\tEpsilon: 0.01\tAvg. MSE Loss: 0.00\n",
      "Episode 11877\tAvg Score: 101.55\tEpsilon: 0.01\tAvg. MSE Loss: 0.00"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "params.prioritized_memory = True\n",
    "params.memory_size = 10000\n",
    "\n",
    "agent = Agent(**params.dict())\n",
    "\n",
    "# Add our custom prep_state function to the agent\n",
    "agent.prep_state = custom_prep_state\n",
    "\n",
    "trainer = Trainer(agent=agent, env=env, **trainer_args)\n",
    "trainer.run(logs_callback=agent.logs, save_best_model=True, output_path=\"../models/DQN_PER_best_score.pt\")\n",
    "\n",
    "print(f\"Episode: {trainer.last_episode} - Score: {trainer.best_score}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Agent (Prioritized Memory) - Scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "avg_scores = aggregate_list(trainer.scores, trainer_args[\"print_range\"])\n",
    "x = [i*trainer_args[\"print_range\"] for i in range(len(avg_scores))]\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(x, avg_scores)\n",
    "plt.ylabel(\"Average Score\")\n",
    "plt.xlabel(\"Episode #\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Agent (Prioritized Memory) - Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(agent.losses)\n",
    "plt.title(\"MSE Loss - Prioritized Memory\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Save agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent.save_model(params.model_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Load agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "del agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = Agent(**params.dict())\n",
    "agent.prep_state = custom_prep_state\n",
    "agent.epsilon_enabled = False\n",
    "\n",
    "agent.load_model(params.model_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Test trained agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 150.00\n",
      "Score: 114.00\n",
      "Score: 137.00\n",
      "Score: 137.00\n",
      "Score: 119.00\n",
      "Score: 121.00\n",
      "Score: 120.00\n",
      "Score: 109.00\n",
      "Score: 151.00\n",
      "Score: 114.00\n"
     ]
    }
   ],
   "source": [
    "env = flappy_bird_gym.make(\"FlappyBird-v0\")\n",
    "env.seed(seed)\n",
    "\n",
    "for i in range(10):\n",
    "    play_env(agent, env, fps=30, render=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Conclusions\n",
    "\n",
    "The Deep Q-Network (DQN) algorithm is a powerful algorithm that can learn playing the Flappy Bird game.\n",
    "\n",
    "We achieved reasonable results with the DQM algorithm after training the agent for 60,000 episodes.\n",
    "\n",
    "Unfortunately, the prioritized replay memory (PER) takes a lot of time to train and we are not able to evaluate the DQN Agent with it."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "b3ba2566441a7c06988d0923437866b63cedc61552a5af99d1f4fb67d367b25f"
  },
  "kernelspec": {
   "display_name": "Python 3.8.8 64-bit ('base': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}