assignment2/Dropout.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Dropout\n",
    "Dropout [1] is a technique for regularizing neural networks by randomly setting some features to zero during the forward pass. In this exercise you will implement a dropout layer and modify your fully-connected network to optionally use dropout.\n",
    "\n",
    "[1] Geoffrey E. Hinton et al, \"Improving neural networks by preventing co-adaptation of feature detectors\", arXiv 2012"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "# As usual, a bit of setup\n",
    "from __future__ import print_function\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from cs231n.classifiers.fc_net import *\n",
    "from cs231n.data_utils import get_CIFAR10_data\n",
    "from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array\n",
    "from cs231n.solver import Solver\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for auto-reloading external modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "def rel_error(x, y):\n",
    "  \"\"\" returns relative error \"\"\"\n",
    "  return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train:  (49000, 3, 32, 32)\n",
      "y_train:  (49000,)\n",
      "X_val:  (1000, 3, 32, 32)\n",
      "y_val:  (1000,)\n",
      "X_test:  (1000, 3, 32, 32)\n",
      "y_test:  (1000,)\n"
     ]
    }
   ],
   "source": [
    "# Load the (preprocessed) CIFAR10 data.\n",
    "\n",
    "data = get_CIFAR10_data()\n",
    "for k, v in data.items():\n",
    "  print('%s: ' % k, v.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Dropout forward pass\n",
    "In the file `cs231n/layers.py`, implement the forward pass for dropout. Since dropout behaves differently during training and testing, make sure to implement the operation for both modes.\n",
    "\n",
    "Once you have done so, run the cell below to test your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running tests with p =  0.3\n",
      "Mean of input:  10.0002078785\n",
      "Mean of train-time output:  10.0350727971\n",
      "Mean of test-time output:  10.0002078785\n",
      "Fraction of train-time output set to zero:  0.699124\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.6\n",
      "Mean of input:  10.0002078785\n",
      "Mean of train-time output:  9.97691075877\n",
      "Mean of test-time output:  10.0002078785\n",
      "Fraction of train-time output set to zero:  0.401368\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.75\n",
      "Mean of input:  10.0002078785\n",
      "Mean of train-time output:  9.99306858826\n",
      "Mean of test-time output:  10.0002078785\n",
      "Fraction of train-time output set to zero:  0.250496\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(500, 500) + 10\n",
    "\n",
    "for p in [0.3, 0.6, 0.75]:\n",
    "  out, _ = dropout_forward(x, {'mode': 'train', 'p': p})\n",
    "  out_test, _ = dropout_forward(x, {'mode': 'test', 'p': p})\n",
    "\n",
    "  print('Running tests with p = ', p)\n",
    "  print('Mean of input: ', x.mean())\n",
    "  print('Mean of train-time output: ', out.mean())\n",
    "  print('Mean of test-time output: ', out_test.mean())\n",
    "  print('Fraction of train-time output set to zero: ', (out == 0).mean())\n",
    "  print('Fraction of test-time output set to zero: ', (out_test == 0).mean())\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Dropout backward pass\n",
    "In the file `cs231n/layers.py`, implement the backward pass for dropout. After doing so, run the following cell to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dx relative error:  5.44561271827e-11\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(10, 10) + 10\n",
    "dout = np.random.randn(*x.shape)\n",
    "\n",
    "dropout_param = {'mode': 'train', 'p': 0.8, 'seed': 123}\n",
    "out, cache = dropout_forward(x, dropout_param)\n",
    "dx = dropout_backward(dout, cache)\n",
    "dx_num = eval_numerical_gradient_array(lambda xx: dropout_forward(xx, dropout_param)[0], x, dout)\n",
    "\n",
    "print('dx relative error: ', rel_error(dx, dx_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Fully-connected nets with Dropout\n",
    "In the file `cs231n/classifiers/fc_net.py`, modify your implementation to use dropout. Specificially, if the constructor the the net receives a nonzero value for the `dropout` parameter, then the net should add dropout immediately after every ReLU nonlinearity. After doing so, run the following to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running check with dropout =  0\n",
      "Initial loss:  2.30047908977\n",
      "W1 relative error: 1.48e-07\n",
      "W2 relative error: 2.21e-05\n",
      "W3 relative error: 3.53e-07\n",
      "b1 relative error: 5.38e-09\n",
      "b2 relative error: 2.09e-09\n",
      "b3 relative error: 5.80e-11\n",
      "\n",
      "Running check with dropout =  0.25\n",
      "Initial loss:  2.29243250883\n",
      "W1 relative error: 2.74e-08\n",
      "W2 relative error: 2.98e-09\n",
      "W3 relative error: 4.29e-09\n",
      "b1 relative error: 7.78e-10\n",
      "b2 relative error: 3.36e-10\n",
      "b3 relative error: 1.65e-10\n",
      "\n",
      "Running check with dropout =  0.5\n",
      "Initial loss:  2.30427592208\n",
      "W1 relative error: 3.11e-07\n",
      "W2 relative error: 1.84e-08\n",
      "W3 relative error: 5.35e-08\n",
      "b1 relative error: 2.58e-08\n",
      "b2 relative error: 2.99e-09\n",
      "b3 relative error: 1.13e-10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "N, D, H1, H2, C = 2, 15, 20, 30, 10\n",
    "X = np.random.randn(N, D)\n",
    "y = np.random.randint(C, size=(N,))\n",
    "\n",
    "for dropout in [0, 0.25, 0.5]:\n",
    "  print('Running check with dropout = ', dropout)\n",
    "  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,\n",
    "                            weight_scale=5e-2, dtype=np.float64,\n",
    "                            dropout=dropout, seed=123)\n",
    "\n",
    "  loss, grads = model.loss(X, y)\n",
    "  print('Initial loss: ', loss)\n",
    "\n",
    "  for name in sorted(grads):\n",
    "    f = lambda _: model.loss(X, y)[0]\n",
    "    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)\n",
    "    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Regularization experiment\n",
    "As an experiment, we will train a pair of two-layer networks on 500 training examples: one will use no dropout, and one will use a dropout probability of 0.75. We will then visualize the training and validation accuracies of the two networks over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "(Iteration 1 / 125) loss: 7.856644\n",
      "(Epoch 0 / 25) train acc: 0.274000; val_acc: 0.192000\n",
      "(Epoch 1 / 25) train acc: 0.410000; val_acc: 0.263000\n",
      "(Epoch 2 / 25) train acc: 0.518000; val_acc: 0.269000\n",
      "(Epoch 3 / 25) train acc: 0.550000; val_acc: 0.248000\n",
      "(Epoch 4 / 25) train acc: 0.684000; val_acc: 0.297000\n",
      "(Epoch 5 / 25) train acc: 0.758000; val_acc: 0.292000\n",
      "(Epoch 6 / 25) train acc: 0.782000; val_acc: 0.266000\n",
      "(Epoch 7 / 25) train acc: 0.860000; val_acc: 0.240000\n",
      "(Epoch 8 / 25) train acc: 0.864000; val_acc: 0.285000\n",
      "(Epoch 9 / 25) train acc: 0.898000; val_acc: 0.279000\n",
      "(Epoch 10 / 25) train acc: 0.910000; val_acc: 0.269000\n",
      "(Epoch 11 / 25) train acc: 0.948000; val_acc: 0.292000\n",
      "(Epoch 12 / 25) train acc: 0.960000; val_acc: 0.287000\n",
      "(Epoch 13 / 25) train acc: 0.952000; val_acc: 0.282000\n",
      "(Epoch 14 / 25) train acc: 0.952000; val_acc: 0.267000\n",
      "(Epoch 15 / 25) train acc: 0.944000; val_acc: 0.287000\n",
      "(Epoch 16 / 25) train acc: 0.936000; val_acc: 0.265000\n",
      "(Epoch 17 / 25) train acc: 0.950000; val_acc: 0.278000\n",
      "(Epoch 18 / 25) train acc: 0.968000; val_acc: 0.294000\n",
      "(Epoch 19 / 25) train acc: 0.970000; val_acc: 0.286000\n",
      "(Epoch 20 / 25) train acc: 0.980000; val_acc: 0.304000\n",
      "(Iteration 101 / 125) loss: 0.204318\n",
      "(Epoch 21 / 25) train acc: 0.952000; val_acc: 0.289000\n",
      "(Epoch 22 / 25) train acc: 0.982000; val_acc: 0.289000\n",
      "(Epoch 23 / 25) train acc: 0.970000; val_acc: 0.284000\n",
      "(Epoch 24 / 25) train acc: 0.986000; val_acc: 0.303000\n",
      "(Epoch 25 / 25) train acc: 0.984000; val_acc: 0.284000\n",
      "0.75\n",
      "(Iteration 1 / 125) loss: 11.299055\n",
      "(Epoch 0 / 25) train acc: 0.246000; val_acc: 0.181000\n",
      "(Epoch 1 / 25) train acc: 0.400000; val_acc: 0.231000\n",
      "(Epoch 2 / 25) train acc: 0.544000; val_acc: 0.270000\n",
      "(Epoch 3 / 25) train acc: 0.622000; val_acc: 0.263000\n",
      "(Epoch 4 / 25) train acc: 0.688000; val_acc: 0.299000\n",
      "(Epoch 5 / 25) train acc: 0.774000; val_acc: 0.289000\n",
      "(Epoch 6 / 25) train acc: 0.776000; val_acc: 0.283000\n",
      "(Epoch 7 / 25) train acc: 0.836000; val_acc: 0.280000\n",
      "(Epoch 8 / 25) train acc: 0.838000; val_acc: 0.284000\n",
      "(Epoch 9 / 25) train acc: 0.888000; val_acc: 0.284000\n",
      "(Epoch 10 / 25) train acc: 0.858000; val_acc: 0.309000\n",
      "(Epoch 11 / 25) train acc: 0.908000; val_acc: 0.290000\n",
      "(Epoch 12 / 25) train acc: 0.904000; val_acc: 0.273000\n",
      "(Epoch 13 / 25) train acc: 0.934000; val_acc: 0.305000\n",
      "(Epoch 14 / 25) train acc: 0.944000; val_acc: 0.309000\n",
      "(Epoch 15 / 25) train acc: 0.934000; val_acc: 0.298000\n",
      "(Epoch 16 / 25) train acc: 0.950000; val_acc: 0.292000\n",
      "(Epoch 17 / 25) train acc: 0.958000; val_acc: 0.304000\n",
      "(Epoch 18 / 25) train acc: 0.954000; val_acc: 0.325000\n",
      "(Epoch 19 / 25) train acc: 0.966000; val_acc: 0.323000\n",
      "(Epoch 20 / 25) train acc: 0.956000; val_acc: 0.306000\n",
      "(Iteration 101 / 125) loss: 0.412024\n",
      "(Epoch 21 / 25) train acc: 0.948000; val_acc: 0.285000\n",
      "(Epoch 22 / 25) train acc: 0.942000; val_acc: 0.290000\n",
      "(Epoch 23 / 25) train acc: 0.948000; val_acc: 0.314000\n",
      "(Epoch 24 / 25) train acc: 0.958000; val_acc: 0.300000\n",
      "(Epoch 25 / 25) train acc: 0.976000; val_acc: 0.288000\n"
     ]
    }
   ],
   "source": [
    "# Train two identical nets, one with dropout and one without\n",
    "np.random.seed(231)\n",
    "num_train = 500\n",
    "small_data = {\n",
    "  'X_train': data['X_train'][:num_train],\n",
    "  'y_train': data['y_train'][:num_train],\n",
    "  'X_val': data['X_val'],\n",
    "  'y_val': data['y_val'],\n",
    "}\n",
    "\n",
    "solvers = {}\n",
    "dropout_choices = [0, 0.75]\n",
    "for dropout in dropout_choices:\n",
    "  model = FullyConnectedNet([500], dropout=dropout)\n",
    "  print(dropout)\n",
    "\n",
    "  solver = Solver(model, small_data,\n",
    "                  num_epochs=25, batch_size=100,\n",
    "                  update_rule='adam',\n",
    "                  optim_config={\n",
    "                    'learning_rate': 5e-4,\n",
    "                  },\n",
    "                  verbose=True, print_every=100)\n",
    "  solver.train()\n",
    "  solvers[dropout] = solver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4IAAAJNCAYAAABkwMu0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3X2c1XWd9/HXx2GQUQsy3IoBF0wjXSCpiXZX27qywjIV\nrSVtu19ztVLb67ow3ccuueyNruyu5WZ5kVvZLdKmhJtGLd1iNzoIC6mxklrMYAXa4A1jDPi5/jhn\ncAYHmHHmnN+cc17Px2MeZ37f393nzDlzmDff7+/7i8xEkiRJktQ4Diq6AEmSJElSdRkEJUmSJKnB\nGAQlSZIkqcEYBCVJkiSpwRgEJUmSJKnBGAQlSZIkqcEYBCVJDSMimiLisYg4suhaJEkqUngfQUnS\naBURj/VZPAT4HbC7vPwXmfnF6lclSVLtMwhKkmpCRDwAnJOZ/7WfbcZk5q7qVVU99fzcJEnV59BQ\nSVLNioi/j4gbIuLLEfEo8PaI+KOI+HFEdEXEgxFxdUQ0l7cfExEZEVPLy18or781Ih6NiB9FxLR9\nnOugiPiPiPhV+djfjYhj+6w/JCKuiohfRsT2iPh+RBxcXvcn5Zq2R8TmiHhHuX11RLy7zzHOiYjv\n7lXr+yNiE/CzcvvHI6IjIh6JiDsi4o/77D8mIv4mIn5eXt8eEZMi4v9FxD/t9XxuiYgLhv8qSJJq\nkUFQklTrzgC+BIwHbgB2ARcBE4ETgJOBv9jP/m8D/gY4HPgl8Hf72fY/gWOA5wM/BT7fZ91VwCzg\nFeVj/RXwZDlY3gL8K/BcYDawYQjP7zTg5cDM8vJPyuc5HPgP4Cu9gRNYALyF0nOeAJwDPAFcD5wd\nEQEQEc8DXg18eQh1SJLqiEFQklTrVmfmzZn5ZGZ2Z+YdmfmTzNyVmfcBS4BX7Wf//8jM9szsAb4I\nHD/QRuXjfzYzH83MJ4DLgJdFxKER0QS8G7gwMx/MzN2Zubp8zLcDt2bmsnJN2zJz3RCe3z9m5m8z\ns7tcx+cz8+HyMNErgWcDR5e3PQf4q8y8t1zvuvK2P6QUCF9d3u5s4L8yc9sQ6pAk1RGDoCSp1m3u\nuxARL46Ir5eHcD4CLKLUO7gvv+rz/Q7gsIE2Ks84emVE3Fc+7qbyqonA84CxwM8H2HXKPtoHa+/n\nd3FE/CwitgO/BQ7lqee3v3N9jlIopfz4+X1sJ0lqAAZBSVKt23vWs/9Hadjm0Zn5bGAhECNwnncC\nbwReQ2kYam8vXAC/BnYCLxxgv837aAd4nNJsqL2eP8A2e55fRPwv4H8Db6Y09PM5wGM89fz2d67P\nA2dExOzyNjfvYztJUgMwCEqS6s2zgO3A4+XJXPZ3feBQj/s74CFK4e0feldk5m7gs8BHI+L55d7D\nE8qT1HwBODki3lyezGViRLykvOs64M0R0RIRLwLeO4gadgHbgGZKw1MP7bP+OuDvI+KFUXJ8RBxe\nrvEXwH9Tul7wK+XhrZKkBmUQlCTVm/8DvAt4lFLv4A0jdNzPAFvKX3cBP9xr/V8C9wBrgIeBf6R0\nm6b7gVOBD5fb7+SpiV/+mVKP32+AT1MKjftzC/BfwL3AA8AjwIN91i8GlgOryuuWAOP6rL++fG6H\nhUpSg/M+gpIkNYiIeA3w78BR6R8AktTQ7BGUJKkBRMRYSrfV+JQhUJJkEJQkqc5FxExKM4weDlxd\ncDmSpFHAoaGSJEmS1GDsEZQkSZKkBmMQlCRJkqQGM6boAkbSxIkTc+rUqUWXIUmSJEmFWLNmzbbM\nPOJA29VVEJw6dSrt7e1FlyFJkiRJhYiIXwxmO4eGSpIkSVKDKSQIRsSnI+I3EfHTfayPiLg6IjZF\nxPqIeGm1a5QkSZKkelVUj+BngZP3s/4NwDHlr3OBT1ahJkmSJElqCIUEwcz8PvDwfjY5HfhclvwY\nmBARL6hOdZIkSZJU30brNYKtwOY+yx3lNkmSJEnSMI3WIDhoEXFuRLRHRPvWrVuLLkeSJEmSRr3R\nevuITmBKn+XJ5banycwlwBKAtra2rHxpkiRJkoq0fG0ni1duZEtXN5MmtLBg7nTmzXYA4VCM1iC4\nAvhgRCwFXgFsz8wHC65JkiRJGnUaLRQtX9vJpTduoLtnNwCdXd1ceuMGgKo873r5eRcSBCPiy8Cr\ngYkR0QF8BGgGyMxrgVuANwKbgB3Ae4qoU5IkSbWlXv5IH6yiQ1ERFq/cuOf59uru2c3ilRsr/pzr\n6eddSBDMzLMPsD6BD1SpHEmSJNWBevojfbCKDEVF2dLVPaT2kbR45UZet/t7XDx2GZNiG1tyIlfu\nms/ilWNr7udd85PFSJIkSbD/UFSvigxFRZk0oWVI7SOp7ZFvcUXzdUw+aBsHBUw+aBtXNF9H2yPf\nqvi5R9povUZQkiRJw1TkMMkizt2ooahzgOdXjVAExbzOC+ZO79fzC9DS3MSCudMrel6AS8d+hUPY\n2a/tkNjJpWO/Alxe8fOPJHsEJUmS6lDvMMnOrm6Sp4ZJLl874ETsI37u1Td9ght2vI+fH/w2btjx\nPlbf9ImKn7vInqKiLJg7nZbmpn5t1QpFRb3H5s1u5fIzZ9I6oYUAWie0cPmZM6vynxzPY9uQ2kcz\ng6AkSVIdKnKY5LqvL2FRLOk3fG5RLGHd15dU9LxFhqKiFBmKinyPzZvdym2XvIb7rziF2y55TdV6\numP85CG1j2YODZUkicabaVBVtn4ZrFoE2ztg/GQ4aSHMml/RUxY5TPKcnV/gkIOePnzunJ1fAP62\nYuft/Z1ttN/lebNbC3mOhQ7FLeB3Ciid5+YLoafPc2xuKbXXGIOgJKnhNeJMg6qi9cv6/+G4fXNp\nGSr6h2uR145NOuihIbWPpHlNtzHv4EUwrgMOngxNC4EqBIQGVNh7rKDfqX7HLyKEjjCDoCSp4RV9\nT6pGmsyjIa1a1L/3AErLqxZV9I/HBXOns/qmT/Ahlu6Z5v6jnMWJc99fsXP2eqLl+RzS/eDA7ZU8\ncZEBgcb7nSps0paCfqf2mDW/JoPf3gyCkqRRo6g/oooa3lRkT2TD9oIWMZxse8fQ2kfIvKbbeFPz\ndYzZ/QQAk2MbVzRdx5iml1DpHrJD3rCIXV+7YM+5AXY1jeOQNyyq6HmLDAh7JshhKZMO3saWHRP5\n6E1nAe+v29+pebNbad38n0y5czG/l1v5TRzB5pcu4OWzT67siQv6nao3ThYjSRoVipzhsKiZBouc\naKH3psirx17IfQe/jdVjL+R1u79XlXMvX9vJCVd8m2mXfJ0Trvh2VV5j4Kneou2bgXyqt2j9soqe\ndkfL84fUPmJWLeoXxIDS8qoKhzGAWfMZc/q/wfgpQMD4KaXlOg3dUNwEOYVav4yXb/gIz2crBwU8\nn628fMNHKv47xb4mZqnBCVuKZBCUJI0KRYaiomYaLHKihaJuilxk4N9vb1EFXdnzVnbk2H5tO3Is\nV/a8taLnLbzXZNZ8+MufwmVdpcdqDKUrMCCcs/MLHBL7miCnwtYvg6tmwGUTSo+VDmK9Cvqd4qSF\npQla+qrRCVuKZBCUJI0KRYaioqZfL/KeZ5eO/cqAf7SWbopcOUUG/qKC0fWPzeGSnnPoeHIiT2bQ\n8eRELuk5h+sfm1PR8zZkr0mBAaGwCXIK6ukGivvPhlnz4dSr+/U4c+rVdXHdXjV5jaAkaVQocoZD\nKGb69SIn8yjqpshburo57aDVXDxm2Z7nfOWu+dzcdWJFzwuUAtD2zQO3V9CkCS2s6DqRFTv7P8fW\nSr+362ia+0ErcEbHwibIKXLilIJ+p4C6mbClSPYISpJGhYa8EXTTbQMOz5zXdFvFz13UTZHfddjt\nAz7ndx12e0XPC3DHCy+ge68hmt05ljteeEFFz1vYe7tRe02KGJJKeYKcpnH92qoyQU6RQ4AdolnT\nDIKSpFGhqOGZhSpyMo+C/oC7uPmGAYekXtx8Q0XPC/Chu4/hw3sN0fxwzzl86O5jKnreQt/bBYWi\nhlTUBDlFDgFu1P9sqBORmUXXMGLa2tqyvb296DIkSRqcyyYAA/07HKU/3CutiFspFPicp13y9X2d\nmfuvOKWi55YqZu97J0LpP3UMZA0rItZkZtuBtvMaQUmSilLk9TVQzDU2BT7noq9DlSqiwOsiVdsc\nGipJUlEa8fqaAp9zI16HqgbhEGA9A/YISpJUlEb8n/wCn3PvNXmLV25kS1c3kya0sGDu9Pq+DlWS\n9sFrBCVJT7N8bad/LEuSVIO8RlCS9IwsX9vJpTdu2HPT786ubi69cQNA5cNgEZOXSJLUgAq5RjAi\nTo6IjRGxKSIuGWD9cyLipohYHxG3R8SMIuqUpEa0eOXGPSGwV3fPbhav3FjZE/fOfLd9M5Clx5sv\nLLVXw/plcNWM0qyWV82o3nklSSpA1YNgRDQB1wBvAI4Dzo6I4/ba7K+AdZk5C3gn8LHqVilJjWvL\nALMq7q99xKxa1H/6cygtV+OeekWHUEmSqqyIHsE5wKbMvC8zdwJLgdP32uY44NsAmfkzYGpEPK+6\nZUpS8Zav7eSEK77NtEu+zglXfJvlazsrfs59TaVf8Sn2t3cMrX0kFRlCJUkqQBFBsBXoewOhjnJb\nX/8NnAkQEXOA3weqdFMlSRodeq/V6+zqJnnqWr1Kh8HCptjf133kqnFPvSJDqCRJBRit9xG8ApgQ\nEeuAC4C1wO6BNoyIcyOiPSLat27dWs0aJamiirpWb97sVi4/cyatE1oIoHVCC5efObPyE8UUeU+9\nIkOoJEkFKGLW0E5gSp/lyeW2PTLzEeA9ABERwP3AfQMdLDOXAEugdPuICtQrSYXY0tXNaQet5uIx\ny5gU29iSE7ly13xu7jqx4ueeN7u1+reLKPKeeictLF0T2Hd4aL3f2F2S1NCKCIJ3AMdExDRKAfAs\n4G19N4iICcCO8jWE5wDfL4dDSWoY7zrsdi7uuY5DYicAk2MbVzRfx+HNY4FTii2uUmbNL+Z2EY14\nY3dJUkOrehDMzF0R8UFgJdAEfDoz74qI88rrrwWOBa6PiATuAv682nVKUtEubr6BQ3bt7Nd2SOzk\n4uYbgL8tpqh6VlQIlSSpAIXcUD4zbwFu2avt2j7f/wh4UbXrkqTR5JDuXw2pXZIkabBG62QxkiQn\nMJEkSRViEJSk0arIWTQlSVJdMwhK0mg1az6cejWMnwJE6fHUq72OTZIkDVsh1whKUi1ZvraTxSs3\nsqWrm0kTWlgwd3r1bq3gBCaSJKkCDIKSakYRgWz52k4uvXHDnhu7d3Z1c+mNGwCqf589SZKkEeLQ\nUEk1oTeQdXZ1kzwVyJav7azoeRev3LgnBPbq7tnN4pUbK3rewq1fBlfNgMsmlB7XLyu6IkmSNIIM\ngpJqQlGBbEtX95Da68L6ZXDzhbB9M5Clx5svNAxKklRHDIKSakJRgWzShJYhtdeFVYugZ6+fa093\nqV2SJNUFg6CkmlBUIFswdzotzU392lqam1gwd3pFz1uo7R1Da5ckSTXHICipJhQVyObNbuXyM2fS\nOqGFAFontHD5mTPre6IYb2QvSVLdc9ZQSTWhN3gVcRuHebNb6zv47e2khaVrAvsOD/VG9pIk1RWD\noKSaMa/pNuYdvAjGdcDBk6FpIVCFe+ytX1a6Pm57R6lX7KSF9X1vv97n1kjPWZKkBmMQlFQbemey\n7O2l6p3JEiobUIo6b9G8kb0kSXXNawQl1YaiZrJ0Bk1JklSHDIKSakNRM1k6g6YkSapDBkFJtaGo\nmSydQVOSJNUhg6Ck2nDSwtLMlX1VYybLos4rSZJUQQZBSbVh1nw49WoYPwWI0uOpV1d+QpOizitJ\nklRBkZlF1zBi2trasr29vegyJEmSJKkQEbEmM9sOtJ09gpIkSZLUYAyCkiRJktRgCgmCEXFyRGyM\niE0RcckA68dHxM0R8d8RcVdEvKeIOiVJkiSpHlU9CEZEE3AN8AbgOODsiDhur80+ANydmS8BXg38\nS0SMrWqhkiRJklSniugRnANsysz7MnMnsBQ4fa9tEnhWRARwGPAwsKu6ZUqSJElSfSoiCLYCm/ss\nd5Tb+vo4cCywBdgAXJSZT1anPEmSJEmqb6N1spi5wDpgEnA88PGIePZAG0bEuRHRHhHtW7durWaN\nkiRJklSTigiCncCUPsuTy219vQe4MUs2AfcDLx7oYJm5JDPbMrPtiCOOqEjBkiRJklRPxhRwzjuA\nYyJiGqUAeBbwtr22+SVwEvCDiHgeMB24r6pVShrQ8rWdLF65kS1d3Uya0MKCudOZN3vv0d2SJEka\nzaoeBDNzV0R8EFgJNAGfzsy7IuK88vprgb8DPhsRG4AAPpyZ26pdq6T+lq/t5NIbN9DdsxuAzq5u\nLr1xA4BhUJIkqYYU0SNIZt4C3LJX27V9vt8CvL7adUnav8UrN/K63d/j4rHLmBTb2JITuXLXfBav\nHGsQlCRJqiGjdbIYSQeyfhlcNQMum1B6XL+s4qdse+RbXNF8HZMP2sZBAZMP2sYVzdfR9si3Kn5u\nSZIkjRyDoFSL1i+Dmy+E7ZuBLD3efGHFw+ClY7/CIbGzX9shsZNLx36loueVJEnSyDIISrVo1SLo\n6e7f1tNdaq+g5zHwpbr7apckSdLoZBCUatH2jqG1j5AYP3lI7ZIkSRqdDIJSLdpX8Kp0IDtpITS3\n9G9rbim1S5IkqWYYBKUadMcLL6A7x/Zr686x3PHCCyp74lnz4dSrYfwUIEqPp15dapckSVLNKOT2\nEZKG50N3H8PLes7h4jHLmBQPsSWfy5W75rPm7mO47bQKn3zWfIOfJElSjTMISjVoS1c3nZzIip0n\n9muPru597CFJkiQ9xaGhUg2aNKFlSO2SJElSXwZBqQYtmDudluamfm0tzU0smDu9oIokSZJUSxwa\nKtWgebNbAVi8ciNburqZNKGFBXOn72mXJEmS9scgKNWoebNbDX6SJEl6RhwaKkmSJEkNxiAoSZIk\nSQ3GIChJkiRJDcYgKEmSJEkNxiAoSZIkSQ3GIChJkiRJDcYgKEmSJEkNxiAoSZIkSQ2mkCAYESdH\nxMaI2BQRlwywfkFErCt//TQidkfE4UXUKkmSJEn1pupBMCKagGuANwDHAWdHxHF9t8nMxZl5fGYe\nD1wKfC8zH652rZIkSZJUj4roEZwDbMrM+zJzJ7AUOH0/258NfLkqlUmSJElSAygiCLYCm/ssd5Tb\nniYiDgFOBr5ahbokSZIkqSGM9sliTgVu29+w0Ig4NyLaI6J969atVSxNkiRJkmpTEUGwE5jSZ3ly\nuW0gZ3GAYaGZuSQz2zKz7YgjjhihEqVBWr8MrpoBl00oPa5fVnRFkiRJ0gEVEQTvAI6JiGkRMZZS\n2Fux90YRMR54FfC1KtcnDc76Zez62gWwfTOQsH1zadkwKEmSpFGu6kEwM3cBHwRWAvcAyzLzrog4\nLyLO67PpGcA3M/PxatcoDcaOWxcyZvcT/drG7H6CHbcuLKgiSZIkaXDGDGfniLgA+EJm/nYo+2Xm\nLcAte7Vdu9fyZ4HPDqc+qZLGdf9qSO2SJEnSaDHcHsHnAXdExLLyTeJjJIqSasGWJ587pHZJkiRp\ntBhWEMzMvwaOAf4deDdwb0T8Y0S8cARqk0a168a+nR05tl/bjhzLdWPfXlBFkiRJ0uAM+xrBzEzg\nV+WvXcBzgP+IiCuHe2xpNDv+lHNZmOfS8eREnsyg48mJLMxzOf6Uc4suTZIkSdqv4V4jeBHwTmAb\ncB2wIDN7IuIg4F7g4uGXKI1O82a3Au/nrStPYktXN5MmtLBg7vRyuyRJkjR6DSsIAocDZ2bmL/o2\nZuaTEfGmYR5bGvXmzW41+EmSJKnmDHdo6K3Aw70LEfHsiHgFQGbeM8xjS5IkSZIqYLhB8JPAY32W\nHyu3SZIkSZJGqeEGwShPFgOUhoQy/OGmkiRJkqQKGm4QvC8iLoyI5vLXRcB9I1GYJEmSJKkyhhsE\nzwP+GOgEOoBXAM6dL0mSJEmj2LCGcWbmb4CzRqgWSZIkSVIVDPc+guOAPwf+ABjX256Z7x1mXZIk\nSZKkChnu0NDPA88H5gLfAyYDjw63KEmSJElS5Qw3CB6dmX8DPJ6Z1wOnULpOUJIkSZI0Sg03CPaU\nH7siYgYwHvi9YR5TkiRJklRBw73n35KIeA7w18AK4DDgb4ZdlTRU65fBqkWwvQPGT4aTFsKs+UVX\nJUmSJI1KzzgIRsRBwCOZ+Vvg+8BRI1aVNBTrl8HNF0JPd2l5++bSMhgGJUmSpAE846GhmfkkcPEI\n1iI9M6sWPRUCe/V0l9olSZIkPc1wrxH8r4j4vxExJSIO7/0akcqkwdreMbR2SZIkqcEN9xrBt5Yf\nP9CnLXGYqKpp/OTScNCB2iVJkiQ9zbB6BDNz2gBfhkBV10kLobmlf1tzS6ldkiRJ0tMMq0cwIt45\nUHtmfu4A+50MfAxoAq7LzCsG2ObVwEeBZmBbZr5qOLWqjvVOCOOsoZIkSdKgDHdo6Mv7fD8OOAm4\nE9hnEIyIJuAa4HVAB3BHRKzIzLv7bDMB+ARwcmb+MiK8N6H2b9Z8g58kSZI0SMMKgpl5Qd/lcoBb\neoDd5gCbMvO+8j5LgdOBu/ts8zbgxsz8Zfk8vxlOnZIkSZKkpwx31tC9PQ5MO8A2rUDfmT06ym19\nvQh4TkR8NyLW7GsIqiRJkiRp6IZ7jeDNlGYJhVKoPA5YNtyiKNX1MkpDTVuAH0XEjzPzfwao4Vzg\nXIAjjzxyBE4tSZIkSfVtuNcI/nOf73cBv8jMA928rROY0md5crmtrw7gocx8HHg8Ir4PvAR4WhDM\nzCXAEoC2trbce70kSZIkqb/hDg39JfCTzPxeZt4GPBQRUw+wzx3AMRExLSLGAmcBK/ba5mvAiREx\nJiIOAV4B3DPMWiVJkiRJDD8IfgV4ss/y7nLbPmXmLuCDwEpK4W5ZZt4VEedFxHnlbe4BvgGsB26n\ndIuJnw6zVkmSJEkSwx8aOiYzd/YuZObOci/ffmXmLcAte7Vdu9fyYmDxMOuTJEmSJO1luD2CWyPi\ntN6FiDgd2DbMY0qSJEmSKmi4PYLnAV+MiI+XlzsAb/UgSZIkSaPYcG8o/3PgDyPisPLyYyNSlSRJ\nkiSpYoY1NDQi/jEiJmTmY5n5WEQ8JyL+fqSKkyRJkiSNvOFeI/iGzOzqXcjM3wJvHOYxJUmSJEkV\nNNwg2BQRB/cuREQLcPB+tpckSZIkFWy4k8V8EVgVEZ8BAng3cP1wi5IkSZIkVc5wJ4v5p4j4b+C1\nQFK6Sfzvj0Rh0lAsX9vJ4pUb2dLVzaQJLSyYO515s1uLLkuSJEkalYbbIwjwa0oh8E+B+4GvjsAx\nVYvWL4NVi2B7B4yfDCcthFnzK37a5Ws7ufTGDXT37Aags6ubS2/cAGAYlCRJkgbwjIJgRLwIOLv8\ntQ24AYjM/F8jWJtqyfplcPOF0NNdWt6+ubQMFQ+Di1du3BMCe3X37Gbxyo0GQUmSJGkAz3SymJ8B\nrwHelJknZua/AbsPsI/q2apFT4XAXj3dpfYK29LVPaR2SZIkqdE90yB4JvAg8J2I+FREnERpshg1\nqu0dQ2sfQZMmtAypXZIkSWp0zygIZubyzDwLeDHwHeBDwO9FxCcj4vUjWaBqxPjJQ2sfQQvmTqel\nualfW0tzEwvmTq/4uSVJkqRaNKz7CGbm45n5pcw8FZgMrAU+PCKVqbactBCa9+qBa24ptVfYvNmt\nXH7mTFontBBA64QWLj9zptcHSpIkSfsQmVl0DSOmra0t29vbiy6jcRU0a6gkSZKkkohYk5ltB9pu\nJG4fIZXMmm/wkyRJkmqAQVAjxpu6S5IkSbXBIKgR4U3dJUmSpNoxrMlipF77u6m7JEmSpNHFIKgR\n4U3dJUmSpNpRSBCMiJMjYmNEbIqISwZY/+qI2B4R68pflb8HgYbFm7pLkiRJtaPqQTAimoBrgDcA\nxwFnR8RxA2z6g8w8vvy1qKpFasi8qbskSZJUO4qYLGYOsCkz7wOIiKXA6cDdBdSiEdI7IYyzhkqS\nJEmjXxFBsBXY3Ge5A3jFANv9cUSsBzqB/5uZd1WjOD1z82a3GvwkSZKkGjBabx9xJ3BkZj4WEW8E\nlgPHDLRhRJwLnAtw5JFHVq9CSZIkSapRRUwW0wlM6bM8udy2R2Y+kpmPlb+/BWiOiIkDHSwzl2Rm\nW2a2HXHEEZWqWZIkSZLqRhFB8A7gmIiYFhFjgbOAFX03iIjnR0SUv59Dqc6Hql6pJEmSJNWhqg8N\nzcxdEfFBYCXQBHw6M++KiPPK668F3gKcHxG7gG7grMzMatcqSZIkSfUo6ilftbW1ZXt7e9FlSJIk\nSVIhImJNZrYdaLtCbigvSZIkSSqOQVCSJEmSGoxBUJIkSZIajEFQkiRJkhqMQVCSJEmSGoxBUJIk\nSZIajEFQkiRJkhqMQVCSJEmSGoxBUJIkSZIajEFQkiRJkhqMQVCSJEmSGoxBUJIkSZIajEFQkiRJ\nkhqMQVCSJEmSGoxBUJIkSZIajEFQkiRJkhqMQVCSJEmSGoxBUJIkSZIajEFQkiRJkhrMmKIL0Mhb\nvraTxSs3sqWrm0kTWlgwdzrzZrcWXZYkSZKkUcIgWGeWr+3k0hs30N2zG4DOrm4uvXEDgGFQkiRJ\nElDQ0NCIODkiNkbEpoi4ZD/bvTwidkXEW6pZXy1bvHLjnhDYq7tnN4tXbiyoIkmSJEmjTdWDYEQ0\nAdcAbwCOA86OiOP2sd0/Ad+sboW1bUtXN6cdtJrVYy/kvoPfxuqxF3LaQavZ0tVddGmSJEmSRoki\nhobOATZl5n0AEbEUOB24e6/tLgC+Cry8uuXVtncddjsX91zHIbETgMmxjSuar+Pw5rHAKcUWJ0mS\nJGlUKGJoaCuwuc9yR7ltj4hoBc4APlnFuurCxc037AmBvQ6JnVzcfENBFUmSJEkabUbr7SM+Cnw4\nM5880IYRcW5EtEdE+9atW6tQ2uh2SPevhtQuSZIkqfEUEQQ7gSl9lieX2/pqA5ZGxAPAW4BPRMS8\ngQ6WmUsysy0z24444ohK1Ftbxk8eWrskSZKkhlNEELwDOCYipkXEWOAsYEXfDTJzWmZOzcypwH8A\n78/M5dW62a6XAAAgAElEQVQvtQadtBCaW/q3NbeU2iVJkiSJAiaLycxdEfFBYCXQBHw6M++KiPPK\n66+tdk11Zdb80uOqRbC9o9QTeNLCp9olSZIkNbzIzKJrGDFtbW3Z3t5edBmSJEmSVIiIWJOZbQfa\nbrROFiNJkiRJqhCDoCRJkiQ1GIOgJEmSJDUYg6AkSZIkNRiDoCRJkiQ1GIOgJEmSJDUYg6AkSZIk\nNRiDoCRJkiQ1GIOgJEmSJDUYg6AkSZIkNRiDoCRJkiQ1GIOgJEmSJDUYg6AkSZIkNRiDoCRJkiQ1\nGIOgJEmSJDUYg6AkSZIkNRiDoCRJkiQ1GIOgJEmSJDUYg6AkSZIkNRiDoCRJkiQ1mEKCYEScHBEb\nI2JTRFwywPrTI2J9RKyLiPaIOLGIOiVJkiSpHo2p9gkjogm4Bngd0AHcERErMvPuPputAlZkZkbE\nLGAZ8OJq1ypJkiRJ9aiIHsE5wKbMvC8zdwJLgdP7bpCZj2VmlhcPBRJJkiRJ0ogoIgi2Apv7LHeU\n2/qJiDMi4mfA14H3Vqk2SZIkSap7o3aymMy8KTNfDMwD/m5f20XEueXrCNu3bt1avQIlSZIkqUYV\nEQQ7gSl9lieX2waUmd8HjoqIiftYvyQz2zKz7YgjjhjZSiVJkiSpDhURBO8AjomIaRExFjgLWNF3\ng4g4OiKi/P1LgYOBh6peqSRJkiTVoarPGpqZuyLig8BKoAn4dGbeFRHnlddfC7wZeGdE9ADdwFv7\nTB4jSZIkSRqGqKd81dbWlu3t7UWXIUmSpAH09PTQ0dHBE088UXQpUs0bN24ckydPprm5uV97RKzJ\nzLYD7V/1HsFGsnxtJ4tXbmRLVzeTJrSwYO505s1+2gSpkiRJDaGjo4NnPetZTJ06lfJVQJKegczk\noYceoqOjg2nTpj2jY4zaWUNr3fK1nVx64wY6u7pJoLOrm0tv3MDytfucF0eSJKmuPfHEEzz3uc81\nBErDFBE897nPHVbvukGwQhav3Eh3z+5+bd09u1m8cmNBFUmSJBXPECiNjOH+LhkEK2RLV/eQ2iVJ\nklR53/jGN5g+fTpHH300V1xxxYDbZCYXXnghRx99NLNmzeLOO+8c0v57O+yww0ak9mfiu9/9Lj/8\n4Q8LO/9IG8zPf/HixRx//PEcf/zxzJgxg6amJh5++GEApk6dysyZMzn++ONpazvgZXRA/b5+BsEK\nmTShZUjtkiRJqqzdu3fzgQ98gFtvvZW7776bL3/5y9x9991P2+7WW2/l3nvv5d5772XJkiWcf/75\nQ9p/MHbt2jWs5zJY9RQEB/vzX7BgAevWrWPdunVcfvnlvOpVr+Lwww/fs/473/kO69atYziTTNbD\n62cQrJAFc6fT0tzUr62luYkFc6cXVJEkSVJtWb62kxOu+DbTLvk6J1zx7WHPtXD77bdz9NFHc9RR\nRzF27FjOOussvva1rz1tu6997Wu8853vJCL4wz/8Q7q6unjwwQcHvf/999/PH/3RHzFz5kz++q//\nek/7d7/7XV75yldy2mmncdxxxwHwr//6r8yYMYMZM2bw0Y9+FIAHHniAF7/4xfzZn/0Zxx57LG95\ny1vYsWMHAKtWrWL27NnMnDmT9773vfzud78DSj1d27ZtA6C9vZ1Xv/rVPPDAA1x77bVcddVVHH/8\n8fzgBz8Y1s9vyNYvg6tmwGUTSo/rlw3rcIP9+ff15S9/mbPPPntI52mU188gWCHzZrdy+ZkzaZ3Q\nQgCtE1q4/MyZzhoqSZI0CJWYeK+zs5MpU6bsWZ48eTKdnU8/3r62G+z+F110Eeeffz4bNmzgBS94\nQb91d955Jx/72Mf4n//5H9asWcNnPvMZfvKTn/DjH/+YT33qU6xduxaAjRs38v73v5977rmHZz/7\n2XziE5/giSee4N3vfjc33HADGzZsYNeuXXzyk5/c5/OdOnUq5513Hn/5l3/JunXreOUrXzn4H9Zw\nrV8GN18I2zcDWXq8+cJhhcHB/vx77dixg2984xu8+c1v3tMWEbz2ta/lZS97GUuWLBlwv0Z5/QyC\nFTRvdiu3XfIa7r/iFG675DWGQEmSpEGq5Yn3brvttj29UO94xzv6rZszZ86e6f5Xr17NGWecwaGH\nHsphhx3GmWeeuafXZ8qUKZxwwgkAvP3tb2f16tVs3LiRadOm8aIXvQiAd73rXXz/+9+v1tMamlWL\noGevuTF6ukvtVXLzzTdzwgkn9BsWunr1atatW8ett97KNddcM+DPr1FeP4NgJY1wd7gkSVKjqMTE\ne62trWzevHnPckdHB62tT/+P+n1tN9j9Yd8zOh566KGDqnXv/Q80Q+SYMWN48sknAYZ1S4ERs71j\naO2DMJSfP8DSpUufNiy0d/vf+73f44wzzuD2228fcN9GeP0MgpVSge5wSZKkRlGJifde/vKXc++9\n93L//fezc+dOli5dymmnnfa07U477TQ+97nPkZn8+Mc/Zvz48bzgBS8Y9P4nnHACS5cuBeCLX/zi\nPut55StfyfLly9mxYwePP/44N910057hf7/85S/50Y9+BMCXvvQlTjzxRKZPn84DDzzApk2bAPj8\n5z/Pq171KqA0jHDNmjUAfPWrX91zjmc961k8+uijz+THNTzjJw+tfRAG+/MH2L59O9/73vc4/fTT\n97Q9/vjje34Wjz/+ON/85jeZMWPG0/ZtlNfPIFgpo6A7XJIkqVZVYuK9MWPG8PGPf5y5c+dy7LHH\nMn/+fP7gD/4AgGuvvZZrr70WgDe+8Y0cddRRHH300bzvfe/jE5/4xAH37+tjH/sY11xzDTNnztzv\nNWwvfelLefe7382cOXN4xStewTnnnMPs2bMBmD59Otdccw3HHnssv/3tbzn//PMZN24cn/nMZ/jT\nP/1TZs6cyUEHHcR5550HwEc+8hEuuugi2traaGp66ud26qmnctNNN1V/spiTFkLzXqG9uaXU/gwN\n9vUDuOmmm3j961/frwfv17/+NSeeeCIveclLmDNnDqeccgonn3zy087TKK9fZOaIHrBIbW1tOZxp\nYEfUZROAgX62AZd1VbsaSZKkwt1zzz0ce+yxg95++dpOFq/cyJaubiZNaGHB3OkNMefCAw88wJve\n9CZ++tOfFl3K8KxfVuoE2d5R6gk8aSHMml90VRVXzddvoN+piFiTmQe8SeKYilXV6MZPLg8LHaBd\nkiRJBzRvdmtDBL+6NWt+QwS/WuXQ0EqpQHe4JEmS6t/UqVNrvzewgdXK62cQrJRZ8+HUq2H8FCBK\nj6de7f+KSJIkSSqcQ0Mrye5wSZKkfjLzgFPpSzqw4c71Yo+gJEmSqmLcuHE89NBDw/4DVmp0mclD\nDz3EuHHjnvEx7BGUJElSVUyePJmOjg62bt1adClSzRs3bhyTJz/ziSgNgpIkSaqK5uZmpk2bVnQZ\nknBoqCRJkiQ1HIOgJEmSJDUYg6AkSZIkNZiop1mbImIr8Iui6xjARGBb0UWobvn+UiX5/lIl+f5S\nJfn+UqWN1vfY72fmEQfaqK6C4GgVEe2Z2VZ0HapPvr9USb6/VEm+v1RJvr9UabX+HnNoqCRJkiQ1\nGIOgJEmSJDUYg2B1LCm6ANU131+qJN9fqiTfX6ok31+qtJp+j3mNoCRJkiQ1GHsEJUmSJKnBGAQr\nKCJOjoiNEbEpIi4puh7Vl4h4ICI2RMS6iGgvuh7Vvoj4dET8JiJ+2qft8Ij4VkTcW358TpE1qnbt\n4/11WUR0lj/H1kXEG4usUbUrIqZExHci4u6IuCsiLiq3+xmmYdvP+6umP8McGlohEdEE/A/wOqAD\nuAM4OzPvLrQw1Y2IeABoy8zReP8a1aCI+BPgMeBzmTmj3HYl8HBmXlH+D63nZOaHi6xTtWkf76/L\ngMcy85+LrE21LyJeALwgM++MiGcBa4B5wLvxM0zDtJ/313xq+DPMHsHKmQNsysz7MnMnsBQ4veCa\nJGmfMvP7wMN7NZ8OXF/+/npK//BJQ7aP95c0IjLzwcy8s/z9o8A9QCt+hmkE7Of9VdMMgpXTCmzu\ns9xBHbxhNKok8F8RsSYizi26GNWt52Xmg+XvfwU8r8hiVJcuiIj15aGjDtvTsEXEVGA28BP8DNMI\n2+v9BTX8GWYQlGrXiZl5PPAG4APlYVdSxWTpWgKvJ9BI+iRwFHA88CDwL8WWo1oXEYcBXwU+lJmP\n9F3nZ5iGa4D3V01/hhkEK6cTmNJneXK5TRoRmdlZfvwNcBOl4cjSSPt1+dqI3mskflNwPaojmfnr\nzNydmU8Cn8LPMQ1DRDRT+iP9i5l5Y7nZzzCNiIHeX7X+GWYQrJw7gGMiYlpEjAXOAlYUXJPqREQc\nWr5YmYg4FHg98NP97yU9IyuAd5W/fxfwtQJrUZ3p/QO97Az8HNMzFBEB/DtwT2b+a59VfoZp2Pb1\n/qr1zzBnDa2g8hSyHwWagE9n5j8UXJLqREQcRakXEGAM8CXfXxquiPgy8GpgIvBr4CPAcmAZcCTw\nC2B+Zjrhh4ZsH++vV1MaUpXAA8Bf9LmeSxq0iDgR+AGwAXiy3PxXlK7j8jNMw7Kf99fZ1PBnmEFQ\nkiRJkhqMQ0MlSZIkqcEYBCVJkiSpwRgEJUmSJKnBGAQlSZIkqcEYBCVJkiSpwRgEJUnaS0Tsjoh1\nfb4uGcFjT42ImrrXlCSp/owpugBJkkah7sw8vugiJEmqFHsEJUkapIh4ICKujIgNEXF7RBxdbp8a\nEd+OiPURsSoijiy3Py8iboqI/y5//XH5UE0R8amIuCsivhkRLYU9KUlSQzIISpL0dC17DQ19a591\n2zNzJvBx4KPltn8Drs/MWcAXgavL7VcD38vMlwAvBe4qtx8DXJOZfwB0AW+u8PORJKmfyMyia5Ak\naVSJiMcy87AB2h8AXpOZ90VEM/CrzHxuRGwDXpCZPeX2BzNzYkRsBSZn5u/6HGMq8K3MPKa8/GGg\nOTP/vvLPTJKkEnsEJUkamtzH90Pxuz7f78Zr9iVJVWYQlCRpaN7a5/FH5e9/CJxV/v7PgB+Uv18F\nnA8QEU0RMb5aRUqStD/+D6QkSU/XEhHr+ix/IzN7byHxnIhYT6lX7+xy2wXAZyJiAbAVeE+5/SJg\nSUT8OaWev/OBBytevSRJB+A1gpIkDVL5GsG2zNxWdC2SJA2HQ0MlSZIkqcHYIyhJkiRJDcYeQUlS\nQynf/D0jwuvkJUkNyyAoSaopEfGNiFg0QPvpEfErA54kSQdmEJQk1ZrrgbdHROzV/g7gi5m5q4Ca\nRkSU+G+zJKni/MdGklRrlgPPBV7Z2xARzwHeBHyuvHxKRKyNiEciYnNEXDbYg0fEJRHx84h4NCLu\njogz9lr/voi4p8/6l5bbp0TEjRGxNSIeioiPl9svi4gv9Nm/39DUiPhuRPxDRNwG7ACOioj39DnH\nfRHxF3vVcHpErCs/v59HxMkR8acRsWav7f53RHxtsM9dktQ4DIKSpJqSmd3AMuCdfZrnAz/LzP8u\nLz9eXj8BOAU4PyLmDfIUP6cUMscDfwt8ISJeABARfwpcVj72s4HTgIciogn4T+AXwFSgFVg6hKf1\nDuBc4FnlY/yGUrB9NqV7El7VJ3DOoRR4F5Sf358ADwArgGkRcexex/3cEOqQJDUIg6AkqRZdD7wl\nIsaVl99ZbgMgM7+bmRsy88nMXA98GXjVYA6cmV/JzC3lfW8A7gXmlFefA1yZmXdkyabM/EV5/SRg\nQWY+nplPZObqITyfz2bmXZm5KzN7MvPrmfnz8jm+B3yTp3pA/xz4dGZ+q1xjZ2b+LDN/B9wAvB0g\nIv6AUij9zyHUIUlqEAZBSVLNKYesbcC8iHghpSD2pd71EfGKiPhOeZjmduA8YOJgjh0R7ywPu+yK\niC5gRp99p1DqMdzbFOAXw7g+cfNeNbwhIn4cEQ+Xa3jjIGqAUhh+W/n6yXcAy8oBUZKkfgyCkqRa\n9TlKPYFvB1Zm5q/7rPsSpaGSUzJzPHAtsPfkMk8TEb8PfAr4IPDczJwA/LTPvpuBFw6w62bgyH3M\nWPo4cEif5ecPsM2em/pGxMHAV4F/Bp5XruGWQdRAZv4Y2Emp9/BtwOcH2k6SJIOgJKlWfQ54LfA+\n+gwLLXsW8HBmPlG+pu5tgzzmoZRC2VaAiHgPpR7BXtcB/zciXlae4fPocni8HXgQuCIiDo2IcRFx\nQnmfdcCfRMSRETEeuPQANYwFDi7XsCsi3gC8vs/6fwfeExEnRcRBEdEaES/us/5zwMeBniEOT5Uk\nNRCDoCSpJmXmA8APKYW3FXutfj+wKCIeBRZSmlxmMMe8G/gX4EfAr4GZwG191n8F+AdKPY6PUprB\n9PDM3A2cChwN/BLoAN5a3udblK7dWw+s4QDX7GXmo8CF5Zp/SynEruiz/nbKE8gA24HvAb/f5xCf\npxRev4AkSfsQmXngrSRJUk2IiBZKs46+NDPvLboeSdLoZI+gJEn15XzgDkOgJGl/BrqoXZIk1aCI\neIDSpDKDvWeiJKlBOTRUkiRJkhqMQ0MlSZIkqcEYBCVJkiSpwdTVNYITJ07MqVOnFl2GJEmSJBVi\nzZo12zLziANtV1dBcOrUqbS3txddhiRJkiQVIiJ+MZjtHBoqSZIkSQ3GIChJkiRJDcYgKEmSJEkN\nxiAoSZIkSQ3GIChJkiRJDcYgKEmSJEkNxiAoSZJUr9Yvg6tmwGUTSo/rlxVdkaRRoq7uIyhJkqSy\n9cvg5guhp7u0vH1zaRlg1vzi6pI0KtgjKEmSVI9WLXoqBPbq6S61S2p4BkFJkqR6tL1jaO0jzWGp\n0qhmEJQkSapH4ycPrX0k9Q5L3b4ZyKeGpRoGpVHDIChJklSPTloIzS3925pbSu2V5rBUadSraBCM\niJMjYmNEbIqISwZYf3pErI+IdRHRHhEnltunRMR3IuLuiLgrIi6qZJ2SJEl1Z9Z8OPVqGD8FiNLj\nqVdXZ6KYooelSjqgis0aGhFNwDXA64AO4I6IWJGZd/fZbBWwIjMzImYBy4AXA7uA/5OZd0bEs4A1\nEfGtvfaVJEnS/syaX8wMoeMnl4eFDtAuaVSoZI/gHGBTZt6XmTuBpcDpfTfIzMcyM8uLhwJZbn8w\nM+8sf/8ocA/QWsFaJUmSNFKKHJYqaVAqGQRbgb7/FdTBAGEuIs6IiJ8BXwfeO8D6qcBs4CcVqVKS\nJEkjq8hhqZIGpfAbymfmTcBNEfEnwN8Br+1dFxGHAV8FPpSZjwy0f0ScC5wLcOSRR1a+YEmSJB1Y\nUcNSJQ1KJXsEO4EpfZYnl9sGlJnfB46KiIkAEdFMKQR+MTNv3M9+SzKzLTPbjjjiiJGpXJIkSZLq\nWCWD4B3AMRExLSLGAmcBK/puEBFHR0SUv38pcDDwULnt34F7MvNfK1ijJEmSJDWcig0NzcxdEfFB\nYCXQBHw6M++KiPPK668F3gy8MyJ6gG7greUZRE8E3gFsiIh15UP+VWbeUql6JUmSJKlRxFOTdta+\ntra2bG9vL7oMSZIkSSpERKzJzLYDbVfRG8pLkiRJkkYfg6AkSZIkNRiDoCRJkqRnZv0yuGoGXDah\n9Lh+WdEVaZAKv4+gJEmSpBq0fhncfCH0dJeWt28uLYP3kKwB9ghKkiRJGrpVi54Kgb16ukvtGvUM\ngpIkSZKGbnvH0No1qhgEJUmSJA3d+MlDa9eoYhCUJEmSNHQnLYTmlv5tzS2ldo16BkFJkiRJQzdr\nPpx6NYyfAkTp8dSrnSimRjhrqCTp6dYvK13sv72jNMTnpIX+w16PfJ0lDdes+X5u1CiDoCSpP6cD\nbwy+zpLU0BwaKknqz+nAG4OvsyQ1NIOgJKk/pwNvDL7OktTQDIKSpP6cDrwx+DpLUkMzCEqS+nM6\n8Mbg66x6tX4ZXDUDLptQely/rOiKpFHJIChJ6s/pwKurqD9afZ1Vj3onQdq+GcinJkEyDEpPE5lZ\ndA0jpq2tLdvb24suQ5Kkwdl75k4o9coZyKRn5qoZ5RC4l/FT4C9/Wv16pAJExJrMbDvQdvYISpJU\nFGfubBwOV6wOJ0GSBs37CEqSVBT/aG0M3rOxana0PJ9Duh8cuL2AelQ5y9d2snjlRrZ0dTNpQgsL\n5k5n3uzWosuqKfYISpJUFGfubAz2/FbNlT1vZUeO7de2I8dyZc9bC6pIlbB8bSeX3riBzq5uEujs\n6ubSGzewfG1n0aXVFIOgJElFcebOxmDPb9Vc/9gcLuk5h44nJ/JkBh1PTuSSnnO4/rE5RZemEbR4\n5Ua6e3b3a+vu2c3ilRsLqqg2OTRUkqSi9A4LXLWoFArGTy6FQIcL1pfxk/cxgYk9vyNt0oQWVnSd\nyIqdJ/Zrb53Qso89VIu2dHUPqV0Dq2iPYEScHBEbI2JTRFwywPrTI2J9RKyLiPaIOHGw+0qSVBdm\nzS/NZnhZV+nREFh/7PmtmgVzp9PS3NSvraW5iQVzpxdUkSph0j6C/b7aNbCK9QhGRBNwDfA6oAO4\nIyJWZObdfTZbBazIzIyIWcAy4MWD3FeSJGn0s+e3anonC3ESkfq2YO50Vt/0CT7EUibFNrbkRD7K\nWZw49/1Fl1ZTKjk0dA6wKTPvA4iIpcDpwJ4wl5mP9dn+UCAHu680Kqxf5j/skqQDmzXffx+qZN7s\nVoNfnZvXdBtvar6OMbufAGBybOOKpusY0/QS+P/t3X2UXHWZ4PHvQ6czaUEIxqikO5mEhYlgEhKn\nDc4EFhUxoPIiKsIi6qrDhlFhPLtxYI8LrLM7oHFFGEFOZPToDIJZDRFWMbrgGyJIQrIJLxPIQEa6\nAQUkKKQxLzz7R1WHSuhOql9uV3fd7+ecPlX3d++teqr69u166vfc3w//zupVZGloO1BbEN9VbdtF\nRLwrIv4F+B7w4YHsKzVU73DgzzwC5IvDgTs3lCRJUnFu+czOJLDXuB3POxLvADV81NDMvCEzXwuc\nAvzdQPePiLOr1xeueuKJJ4Y/QKk/DgcuSYPj5OqShsKReIdFkYlgNzC1Zrmj2tanzPwZcHBEvHIg\n+2bm0szszMzOyZMnDz1qqV6ehCRp4KymkDRUzsE6LIpMBO8CDo2IGRExHjgduLF2g4g4JCKiev/1\nwJ8AT9Wzr9RoW9peM6B2SRJWU0gaOkfiHRaFJYKZuR34OLASuB9Ylpn3RsSiiFhU3ezdwD0RsZbK\nKKHvy4o+9y0qVmkwPrftfWzJ8bu0bcnxfG7b+xoUkSSNAVZTSBqqOafBiVfAAVOBqNyeeIUDMg1Q\noRPKZ+b3ge/v1nZ1zf3PAp+td19pNPn6s/P53T5b+dS4ZUyJp3g0J/G57adx0x/nc3Gjg5Ok0crJ\n1SUNB0fiHbJCE0GpmU2Z2MaNm4/ixq1H7dLe7mSmktS/Yy+sXBNYWx5qSZckjbiGjxoqjVWLF86k\nrbVll7a21hYWL5zZoIgkaQywpGtErVjTzYJLb2XG+d9jwaW3smJNv+P2SSoZewSlQeqdrHbJyg08\nurmHKRPbWLxwppPYStLeWNI1Ilas6eaC5evp2bYDgO7NPVywfD2A/6skmQhKQ3HKvHb/mUqSRqUl\nKzfsTAJ79WzbwZKVG/zfJclEUBqrVqzptjdSTcfjWoVbt6wyVcUzXZUBao69sGl7Jx/d3DOgdknl\nYiIojUGW+6gZeVyrcL2T2fcOVNM7mT00ZTI4ZWIb3X0kfVMc1EwSDhYjjUl7KveRxiqPaxWuZJPZ\nO6iZpD2xR1Aagyz3UTN6dHMPJ+1zW3Vuzid5NF9ZmZtz81F731mqR8kms3dQsxIpUcmzho+JoDQG\nTZnYxp///kcv+cC8ev/jGh2aNGgf3O9XfGrbNbwstgLQEU9yaes1vKJ1PPCOxgan5lDCyewd1Gxk\nNeQ653XL2P7dTzBux/OV5WceqSyDyaD2yNJQaQz64uEP8tnWa+jY50n2CejY50k+23oNXzz8wUaH\nJg3ap1q/tTMJ7PWy2MqnWr/VoIjUdI69sDJ5fS0ns9cw6b3OuXtzD8mL1zkXPXfjlpsvfDEJrBq3\n43m23OxxrT0zEZTGoDf86z/QttsH5rbYyhv+9R8aFJE0dC/reXxA7dKAOZm9CtSo65wn9HOO7K9d\nw2DdMrhsFlw8sXK7blmjIxoUS0Olsahk17moJBpctufUFSOnoe+1k9mrII26fv/RFybRsc+TfbcX\n+swl1USjD9sjKI1F/X0wbuLrXFQCDSzba1RJVxn5XqtZ9TctR9HTdVwz/v1syfG7tG3J8Vwz/v2F\nPm9pNdHowyaC0ljkdS5qRg0s23PqipHje10eK9Z0s+DSW5lx/vdYcOmtTZ/sN2q6jrnvOJsL82y6\nXnglL2TQ9cIruTDPZu47zi70eUuriaqyLA2VxqLeD8YOFT0iLBkcQQ0q23NKlpHje10OvT2/vUl/\nb88v0LTnz1PmtdP+yP9h6t1LeFU+wW9jMo+8fjFvmHd84c8Lf837Vh7r/6mR0ESjD5sISmOV17mM\niDJ+mCmjKRPb6O4jESm6pKuMfK/LYU89v0177ly3jDesvwjogYDX8ASvWX8RTD+w8P/XThMygo69\ncNdrBGHMVmVZGipJe7Bk5QaO2/FTbht/Lg/9yX/gtvHnctyOn45cGVuTjEw22jWqpKuMfK/LoZQ9\nv0107dhAlK0EmDmncdfs/87jTOaFDB5nMnfN/u9j8st5ewQlDcy6ZaUqSe38/Y+4pPWlk5xf8HuA\ntxT75E00Mtlo1/tNuiXAxfO9LodS9vw20bVj9Spj1cyKNd1ccNef0rPt8p1tbXe1cMnU7jH3miMz\nGx3DsOns7MxVq1Y1Ogypee2emEClHKKJ5+F6/OJDeA1PvLSdybzm4o3FPvlls/q5DmEqfPKeYp9b\nkoZg9wQBKj2/l5w6e8x9WK5bCc/ZCy69tc+Ev31iG784v+AvSxtkLLzmiFidmZ17287SUEn1K2HZ\ny6t56dxMe2ofViX8drmMSldWpVI4ZV47l5w6m/aJbQSVD8lNnQRCKUf0LmMJcDO9ZktDJdWvhIlJ\n9Kb1C5sAACAASURBVDM6WIzE6GBNNDKZ+lbGsiqVR+kGMCnhiN5lLAFuptdcaI9gRBwfERsiYmNE\nnN/H+jMjYl1ErI+I2yPiiJp1n4yIeyPinoi4LiImFBmrpDqUcSL7Rn7DW8Jvl8vGOfWkJjPntEoZ\n6MWbK7dNnARCOQd/aqbXXFgiGBEtwJXACcDhwBkRcfhumz0MHJOZs4G/A5ZW920HzgU6M3MW0AKc\nXlSskurU6MSkESNoNnCS84Y+t0ZEM5UYSSqfMpYAN9NrLrI0dD6wMTMfAoiI64GTgft6N8jM22u2\nvwOo7VYYB7RFxDbgZcCjBcYqqR6NLHtp5AiajZyz0fkim1ozlRhJKqfSlQDTPK+5yESwHai9uKUL\nOHIP238EuBkgM7sj4vPAr4Ee4IeZ+cOiApU0AI1KTPY0UE0TJ0or1nQ7zH4TW7xwZp8jK47FEiNJ\n0tgyKgaLiYg3U0kEj6ouH0il93AGsBn43xHx/sz85z72PRs4G2DatGkjFrOkEVbCgWocSKT5Oade\nefiljqTRpshEsBuYWrPcUW3bRUTMAa4BTsjMp6rNbwUezswnqtssB/4SeEkimJlLqV5b2NnZ2TyT\nIkraVQlH0NzTQCJ+gGwezVJipP75pY6k0ajIUUPvAg6NiBkRMZ7KYC831m4QEdOA5cBZmflAzapf\nA2+MiJdFRADHAvcXGKuk0a7RA9U0gAOJSM3B0WEljUaFJYKZuR34OLCSShK3LDPvjYhFEbGoutmF\nwCTgqohYGxGrqvveCXwbuBtYX41zaVGxShoDSjiCZn8DhjiQiDS2+KWOpNEoMpunmrKzszNXrVrV\n6DAkaVjsXk4GlYFExuow1VJZLbj01j5Hh22f2MYvzn9LAyKS1MwiYnVmdu5tu0InlJckDV4zzVUk\nlVkzTUAtqXmMilFDJUl9cyARaexzdFhJo5GJoCRJUsH8UkfSaGNpqCRJAOuWwWWz4OKJldt1yxod\nkSRJhbFHUJKkdcvgpnNhW3VAj2ceqSxDU49MK0kqL3sEJUm65TMvJoG9tvVU2iVJakImgpIkPdM1\nsHZJksY4E0ENH6+vkTRWHdAxsHZJksY4E0ENj97ra555BMgXr68xGZQ0Fhx7IbS27drW2lZplySp\nCTlYjIbHnq6vcaAFacxZsaa7XHOe9Z6nbvlMpRz0gI5KEuj5S5LUpEwENSzymS5iAO2SRq8Va7q5\nYPl6erbtAKB7cw8XLF8P0PzJoImfJKkkLA3VsPgNrxxQu6TRa8nKDTuTwF4923awZOWGBkUkSZKG\nm4mghsUlW9/Llhy/S9uWHM8lW9/boIgkDdajm3sG1K6xa8WabhZceiszzv8eCy69lRVruhsdkiRp\nhFgaqmGxav/jOP/38Klxy5gST/FoTuJz209j9f7HNTo0DbPSXTtWQlMmttHdR9I3ZWJbH1trrCpt\nCbAkCTAR1DBZvHAmFyzfyo1bj9rZ1tbawiULZzYwKg03PziWQ+Xvef0u5aFtrS0s9u+5qeypBNi/\nZ0lqfntNBCPiE8A/Z+bTIxCPxqjeDw32FDU3PziWg3/P5VDWEmCrGiSpop4ewVcDd0XE3cBXgZWZ\nmcWGpbHolHnt/jNtcmX94FhG/j03vzKWAFvVIEkv2utgMZn5aeBQ4B+BDwEPRsTfR8S/Kzg2SaNM\nfx8Qm/mDo9SsFi+cSVtryy5tzV4C7Ii4kvSiukYNrfYAPl792Q4cCHw7Ij5XYGwapFKOArduGVw2\nCy6eWLldt6zRETWlMn5wlJrVKfPaueTU2bRPbCOA9oltXHLq7KbuGbOqQZJeVM81gucBHwCeBK4B\nFmfmtojYB3gQ+FSxIWogSln2sm4Z3HQubKv+I3/mkcoyODn0MPPaMam5lK0EuIzlsJLUn3quEXwF\ncGpm/lttY2a+EBHvLCYsDVYpB/O45TMvJoG9tvVU2k0Eh13ZPjhKah6OiCtJL6qnNPRm4He9CxGx\nf0QcCZCZ9+9px4g4PiI2RMTGiDi/j/VnRsS6iFgfEbdHxBE16yZGxLcj4l8i4v6I+Iv6X1Z5lbLs\n5ZmugbVLkkqpjOWwktSfenoEvwy8vmb52T7aXiIiWoArgeOALiojj96YmffVbPYwcExmPh0RJwBL\ngSOr6y4HfpCZ74mI8cDL6nlBZVfKspcDOirloH21S5JUw6oGSaqop0cwaqeLyMwXqC+BnA9szMyH\nMnMrcD1wcu0GmXl7zfyEdwAdABFxAPDvqYxUSmZuzczNdTxn6ZVyMI9jL4TW3RLd1rZKuyRJkqSX\nqCcRfCgizo2I1urPecBDdezXDtR203RV2/rzESplqAAzgCeAr0XEmoi4JiL2reM5S6+UZS9zToMT\nr4ADpgJRuT3xCq8PlCRJkvpRT8/eIuAK4NNAArcAZw9nEBHxZiqJ4FE1cb0e+ERm3hkRlwPnA/+t\nj33P7o1n2rRpwxnWmFXKspc5p5n4SZIkSXXaayKYmb8FTh/EY3cDU2uWO6ptu4iIOVSmpTghM5+q\nNncBXZl5Z3X521QSwb7iW0rl2kI6Ozuzr20kSZIkSS+qZx7BCVR6614HTOhtz8wP72XXu4BDI2IG\nlQTwdOA/7PbY04DlwFmZ+UDNYz8eEY9ExMzM3AAcC9QOMiNJkiRJGqR6rhH8J+A1wELgp1R69v6w\nt50yczvwcWAlcD+wLDPvjYhFEbGoutmFwCTgqohYGxGrah7iE8C1EbEOmAv8fZ2vSZIkSZK0B1Ez\nIGjfG0Ssycx5EbEuM+dERCvw88x848iEWL/Ozs5ctWrV3jeUJEmSpCYUEaszs3Nv29XTI7iters5\nImYBBwCvGkpwkiRJkqTGqWfU0KURcSCVUUNvBPajj9E7JUmSJEljwx4TwYjYB/h9ddL3nwEHj0hU\nkiRJkqTC7LE0NDNfAD41QrFIkiRJkkZAPdcI/t+I+C8RMTUiXtH7U3hkkiRJkqRC1HON4Puqtx+r\naUssE5UkSZKkMWmviWBmzhiJQCRJkiRJI2OviWBEfKCv9sz8xvCHI0mSJEkqWj2loW+ouT8BOBa4\nGzARlCRJkqQxqJ7S0E/ULkfEROD6wiKSJEmSJBWqnh7B3T0HeN2gRpUVa7pZsnIDj27uYcrENhYv\nnMkp89obHZYkSZI0KtVzjeBNVEYJhcp0E4cDy4oMShqIFWu6uWD5enq27QCge3MPFyxfD2AyKEmS\nJPWhnh7Bz9fc3w78W2Z2FRSPNGBLVm7YmQT26tm2gyUrN5gISpIkSX2oJxH8NfBYZj4PEBFtETE9\nMzcVGplUp0c39wyoXZIkSSq7ferY5n8DL9Qs76i2SaPClIltA2qXJEmSyq6eRHBcZm7tXajeH19c\nSNLALF44k7bWll3a2lpbWLxwZoMikiRJkka3ehLBJyLipN6FiDgZeLK4kKSBOWVeO5ecOpv2iW0E\n0D6xjUtOne31gZIkSVI/6rlGcBFwbUR8qbrcBXyguJCkgTtlXruJnyRJklSneiaU/1fgjRGxX3X5\n2cKjkiRJkiQVZq+loRHx9xExMTOfzcxnI+LAiPgfIxGcJEmSJGn41XON4AmZubl3ITOfBt5eXEiS\nJEmSpCLVkwi2RMSf9C5ERBvwJ3vYXpIkSZI0itWTCF4L3BIRH4mIjwI/Ar5ez4NHxPERsSEiNkbE\n+X2sPzMi1kXE+oi4PSKO2G19S0SsiYj/U8/zSZIkSZL2rp7BYj4bEf8PeCuQwErgT/e2X0S0AFcC\nx1EZafSuiLgxM++r2exh4JjMfDoiTgCWAkfWrD8PuB/Yv87XI6mJrVjTzZKVG3h0cw9TJraxeOFM\nR4uVJEkahHp6BAF+QyUJfC/wFirJ2d7MBzZm5kPVSeivB06u3SAzb69ecwhwB9DRuy4iOoB3ANfU\nGaOkJrZiTTcXLF9P9+YeEuje3MMFy9ezYk13o0OTJEkac/pNBCPizyLiooj4F+AfgF8DkZlvzswv\n9bdfjXbgkZrlrmpbfz4C3Fyz/EXgU8ALdTyXpCa3ZOUGerbt2KWtZ9sOlqzc0KCIJEmSxq49lYb+\nC/Bz4J2ZuREgIj5ZRBAR8WYqieBR1eV3Ar/NzNUR8aa97Hs2cDbAtGnTighP0ijw6OaeAbVLkiSp\nf3sqDT0VeAz4cUR8JSKOBWIAj90NTK1Z7qi27SIi5lAp/zw5M5+qNi8AToqITVRKSt8SEf/c15Nk\n5tLM7MzMzsmTJw8gPEljyZSJbQNqlyRJUv/6TQQzc0Vmng68Fvgx8DfAqyLiyxHxtjoe+y7g0IiY\nERHjgdOBG2s3iIhpwHLgrMx8oOa5L8jMjsycXt3v1sx8/wBfm6QmsnjhTNpaW3Zpa2ttYfHCmQ2K\nSJIkaeyqZ9TQ54BvAt+MiAOpDBjzt8AP97Lf9oj4OJVRRluAr2bmvRGxqLr+auBCYBJwVUQAbM/M\nziG8HklNqnd0UEcNlSRJGrrIzEbHMGw6Oztz1apVjQ5DkiRJkhoiIlbX07lW7/QRkiRJkqQmYSIo\nSZIkSSVjIihJkiRJJWMiKEmSJEklYyIoSZIkSSVjItiM1i2Dy2bBxRMrt+uWNToiSZIkSaPIXucR\n1BizbhncdC5s66ksP/NIZRlgzmmNi0uSJEnSqGGPYLO55TMvJoG9tvVU2iVJkiQJE8Hm80zXwNol\nSZIklY6JYLM5oGNg7ZIkSZJKx0Sw2Rx7IbS27drW2lZplyRJkiRMBJvPnNPgxCvggKlAVG5PvMKB\nYiRJkiTt5KihzWjOaSZ+kiRJkvplj6AkSZIklYyJoCRJkiSVjImgJEmSJJWMiaAkSZIklYyJoCRJ\nkiSVjImgJEmSJJWMiaAkSZIklYyJoCRJkiSVTKGJYEQcHxEbImJjRJzfx/ozI2JdRKyPiNsj4ohq\n+9SI+HFE3BcR90bEeUXGKUmSJEllMq6oB46IFuBK4DigC7grIm7MzPtqNnsYOCYzn46IE4ClwJHA\nduA/Z+bdEfFyYHVE/Gi3fSVJkiRJg1Bkj+B8YGNmPpSZW4HrgZNrN8jM2zPz6eriHUBHtf2xzLy7\nev8PwP1Ae4GxSpIkSVJpFJkItgOP1Cx3sedk7iPAzbs3RsR0YB5w5zDGJkmSJEmlVVhp6EBExJup\nJIJH7da+H/Ad4G8y8/f97Hs2cDbAtGnTCo5UkiRJksa+InsEu4GpNcsd1bZdRMQc4Brg5Mx8qqa9\nlUoSeG1mLu/vSTJzaWZ2Zmbn5MmThy14SZIkSWpWRSaCdwGHRsSMiBgPnA7cWLtBREwDlgNnZeYD\nNe0B/CNwf2Z+ocAYJUmSJKl0CisNzcztEfFxYCXQAnw1M++NiEXV9VcDFwKTgKsquR/bM7MTWACc\nBayPiLXVh/yvmfn9ouKVJEmSpLKIzGx0DMOms7MzV61a1egwJEmSJKkhImJ1tXNtjwqdUF6SJEmS\nNPqYCEqSJElSyZgISpIkSVLJmAhKkiRJUsmYCEqSJElSyZgISpIkSVLJmAhKkiRJUsmYCEqSJElS\nyZgISpIkSVLJmAhKkiRJUsmYCEqSJElSyZgISpIkSVLJmAhKkiRJUsmYCEqSJElSyZgISpIkSVLJ\nmAhKkiRJUsmYCEqSJElSyZgISpIkSVLJmAhKkiRJUsmYCEqSJElSyZgISpIkSVLJFJoIRsTxEbEh\nIjZGxPl9rD8zItZFxPqIuD0ijqh3X0mSJEnS4BSWCEZEC3AlcAJwOHBGRBy+22YPA8dk5mzg74Cl\nA9hXkiRJkjQIRfYIzgc2ZuZDmbkVuB44uXaDzLw9M5+uLt4BdNS7ryRJkiRpcIpMBNuBR2qWu6pt\n/fkIcPMg95UkSZIk1WlcowMAiIg3U0kEjxrEvmcDZwNMmzZtmCOTJEmSpOZTZI9gNzC1Zrmj2raL\niJgDXAOcnJlPDWRfgMxcmpmdmdk5efLkYQlckiRJkppZkYngXcChETEjIsYDpwM31m4QEdOA5cBZ\nmfnAQPaVJEmSJA1OYaWhmbk9Ij4OrARagK9m5r0Rsai6/mrgQmAScFVEAGyv9u71uW9RsUqSJElS\nmURmNjqGYdPZ2ZmrVq1qdBiSJEmS1BARsTozO/e2XaETykuSJEmSRh8TQUmSJEkqGRNBSZIkSSoZ\nE0FJkiRJKplRMaF8s1qxppslKzfw6OYepkxsY/HCmZwyr73RYUmSJEkqORPBgqxY080Fy9fTs20H\nAN2be7hg+XoAk0FJkiRJDWVpaEGWrNywMwns1bNtB0tWbmhQRJIkSZJUYSJYkEc39wyoXZIkSZJG\niolgQaZMbBtQuyRJkiSNFBPBgixeOJO21pZd2tpaW1i8cGaDIpIkSZKkCgeLKUjvgDCOGipJkiRp\ntDERLNAp89pN/CRJkiSNOiaCkiRJGhHbtm2jq6uL559/vtGhSGPehAkT6OjooLW1dVD7mwhKkiRp\nRHR1dfHyl7+c6dOnExGNDkcaszKTp556iq6uLmbMmDGox3CwGEmSJI2I559/nkmTJpkESkMUEUya\nNGlIvesmgpIkSRoxJoHS8Bjq35KJoCRJkkrjBz/4ATNnzuSQQw7h0ksv7XObzOTcc8/lkEMOYc6c\nOdx9990D2n93++2337DEPhg/+clPuP322xv2/MOtnvd/yZIlzJ07l7lz5zJr1ixaWlr43e9+B8D0\n6dOZPXs2c+fOpbOzs67nbNbfn4mgJEmSRqUVa7pZcOmtzDj/eyy49FZWrOke0uPt2LGDj33sY9x8\n883cd999XHfdddx3330v2e7mm2/mwQcf5MEHH2Tp0qWcc845A9q/Htu3bx/Sa6lXQxPBdcvgsllw\n8cTK7bplQ3q4et//xYsXs3btWtauXcsll1zCMcccwyte8Yqd63/84x+zdu1aVq1aNehYmuH3ZyIo\nSZKkUWfFmm4uWL6e7s09JNC9uYcLlq8fUjL4q1/9ikMOOYSDDz6Y8ePHc/rpp/Pd7373Jdt997vf\n5QMf+AARwRvf+EY2b97MY489Vvf+Dz/8MH/xF3/B7Nmz+fSnP72z/Sc/+QlHH300J510EocffjgA\nX/jCF5g1axazZs3ii1/8IgCbNm3ita99LWeeeSaHHXYY73nPe9iyZQsAt9xyC/PmzWP27Nl8+MMf\n5o9//CNQ6el68sknAVi1ahVvetOb2LRpE1dffTWXXXYZc+fO5ec///mg37sBW7cMbjoXnnkEyMrt\nTecOKRms9/2vdd1113HGGWcM6HnK8vszEZQkSdKos2TlBnq27dilrWfbDpas3DDox+zu7mbq1Kk7\nlzs6Oujufmli2d929e5/3nnncc4557B+/XoOOuigXdbdfffdXH755TzwwAOsXr2ar33ta9x5553c\ncccdfOUrX2HNmjUAbNiwgb/+67/m/vvvZ//99+eqq67i+eef50Mf+hDf+ta3WL9+Pdu3b+fLX/5y\nv693+vTpLFq0iE9+8pOsXbuWo48+uv43a6hu+Qxs69m1bVtPpX2Q6n3/e23ZsoUf/OAHvPvd797Z\nFhG89a1v5c///M9ZunRpn/uV5fdnIihJkqRR59HNPQNqH01+8Ytf7OyFOuuss3ZZN3/+/J3D/d92\n2228613vYt9992W//fbj1FNP3dnrM3XqVBYsWADA+9//fm677TY2bNjAjBkz+LM/+zMAPvjBD/Kz\nn/1spF7WwDzTNbD2Atx0000sWLBgl7LQ2267jbVr13LzzTdz5ZVX9vn+leX3V2giGBHHR8SGiNgY\nEef3sf61EfHLiPhjRPyX3dZ9MiLujYh7IuK6iJhQZKyFGOa6aEmSpLKYMrFtQO31aG9v55FHHtm5\n3NXVRXt7e93b1bs/9D+i47777ltXrLvvv7cRIseNG8cLL7wAMKQpBYbNAR0Da6/DQN5/gOuvv/4l\nZaG927/qVa/iXe96F7/61a/63LcMv7/CEsGIaAGuBE4ADgfOiIjDd9vsd8C5wOd327e92t6ZmbOA\nFuD0omItRAF10ZIkSWWxeOFM2lpbdmlra21h8cKZg37MN7zhDTz44IM8/PDDbN26leuvv56TTjrp\nJduddNJJfOMb3yAzueOOOzjggAM46KCD6t5/wYIFXH/99QBce+21/cZz9NFHs2LFCrZs2cJzzz3H\nDTfcsLP879e//jW//OUvAfjmN7/JUUcdxcyZM9m0aRMbN24E4J/+6Z845phjgEoZ4erVqwH4zne+\ns/M5Xv7yl/OHP/xhMG/X0Bx7IbTulrS3tlXaB6ne9x/gmWee4ac//Sknn3zyzrbnnntu53vx3HPP\n8cMf/pBZs2a9ZN+y/P6K7BGcD2zMzIcycytwPXBy7QaZ+dvMvAvY1sf+44C2iBgHvAx4tMBYh18B\nddGSJEllccq8di45dTbtE9sIoH1iG5ecOptT5vXfA7Q348aN40tf+hILFy7ksMMO47TTTuN1r3sd\nAFdffTVXX301AG9/+9s5+OCDOeSQQ/irv/orrrrqqr3uX+vyyy/nyiuvZPbs2Xu8hu31r389H/rQ\nh5g/fz5HHnkkH/3oR5k3bx4AM2fO5Morr+Swww7j6aef5pxzzmHChAl87Wtf473vfS+zZ89mn332\nYdGiRQBcdNFFnHfeeXR2dtLS8mICfeKJJ3LDDTeM/GAxc06DE6+AA6YCUbk98YpK+yDV+/sDuOGG\nG3jb2962Sw/eb37zG4466iiOOOII5s+fzzve8Q6OP/74lzxPWX5/kZnD+oA7HzjiPcDxmfnR6vJZ\nwJGZ+fE+tr0YeDYzP1/Tdh7wP4Ee4IeZeebenrOzszOHMgzssLp4ItDXextw8eaRjkaSJKnh7r//\nfg477LBGhzHqbdq0iXe+853cc889jQ5FgzCSv7++/qYiYnVm7nWSxFE5WExEHEil93AGMAXYNyLe\n38+2Z0fEqohY9cQTT4xkmHtWQF20JEmSJA2HIhPBbmBqzXJHta0ebwUezswnMnMbsBz4y742zMyl\nmdmZmZ2TJ08eUsDDqoC6aEmSJDW/6dOn2xs4ho2V31+RieBdwKERMSMixlMZ7OXGOvf9NfDGiHhZ\nVIbYORa4v6A4i1FAXbQkSZIkDYdxRT1wZm6PiI8DK6mM+vnVzLw3IhZV118dEa8BVgH7Ay9ExN8A\nh2fmnRHxbeBuYDuwBuh7xsfRbM5pJn6SJEk1MnOvQ+lL2ruhjvVSWCIIkJnfB76/W9vVNfcfp1Iy\n2te+FwEXFRmfJEmSRs6ECRN46qmnmDRpksmgNASZyVNPPcWECYOfar3QRFCSJEnq1dHRQVdXF6Nq\ngD9pjJowYQIdHYMfiNJEUJIkSSOitbWVGTNmNDoMSYzS6SMkSZIkScUxEZQkSZKkkjERlCRJkqSS\niaEOOzqaRMQTwL81Oo4+vBJ4stFBqGl5fKlIHl8qkseXiuTxpaKN1mPsTzNz8t42aqpEcLSKiFWZ\n2dnoONScPL5UJI8vFcnjS0Xy+FLRxvoxZmmoJEmSJJWMiaAkSZIklYyJ4MhY2ugA1NQ8vlQkjy8V\nyeNLRfL4UtHG9DHmNYKSJEmSVDL2CEqSJElSyZgIFigijo+IDRGxMSLOb3Q8ai4RsSki1kfE2ohY\n1eh4NPZFxFcj4rcRcU9N2ysi4kcR8WD19sBGxqixq5/j6+KI6K6ex9ZGxNsbGaPGroiYGhE/joj7\nIuLeiDiv2u45TEO2h+NrTJ/DLA0tSES0AA8AxwFdwF3AGZl5X0MDU9OIiE1AZ2aOxvlrNAZFxL8H\nngW+kZmzqm2fA36XmZdWv9A6MDP/tpFxamzq5/i6GHg2Mz/fyNg09kXEQcBBmXl3RLwcWA2cAnwI\nz2Eaoj0cX6cxhs9h9ggWZz6wMTMfysytwPXAyQ2OSZL6lZk/A363W/PJwNer979O5R+fNGD9HF/S\nsMjMxzLz7ur9PwD3A+14DtMw2MPxNaaZCBanHXikZrmLJjhgNKok8H8jYnVEnN3oYNS0Xp2Zj1Xv\nPw68upHBqCl9IiLWVUtHLdvTkEXEdGAecCeewzTMdju+YAyfw0wEpbHrqMycC5wAfKxadiUVJivX\nEng9gYbTl4GDgbnAY8D/amw4GusiYj/gO8DfZObva9d5DtNQ9XF8jelzmIlgcbqBqTXLHdU2aVhk\nZnf19rfADVTKkaXh9pvqtRG910j8tsHxqIlk5m8yc0dmvgB8Bc9jGoKIaKXyIf3azFxebfYcpmHR\n1/E11s9hJoLFuQs4NCJmRMR44HTgxgbHpCYREftWL1YmIvYF3gbcs+e9pEG5Efhg9f4Hge82MBY1\nmd4P6FXvwvOYBikiAvhH4P7M/ELNKs9hGrL+jq+xfg5z1NACVYeQ/SLQAnw1M/9ng0NSk4iIg6n0\nAgKMA77p8aWhiojrgDcBrwR+A1wErACWAdOAfwNOy0wH/NCA9XN8vYlKSVUCm4D/VHM9l1S3iDgK\n+DmwHnih2vxfqVzH5TlMQ7KH4+sMxvA5zERQkiRJkkrG0lBJkiRJKhkTQUmSJEkqGRNBSZIkSSoZ\nE0FJkiRJKhkTQUmSJEkqGRNBSZJ2ExE7ImJtzc/5w/jY0yNiTM01JUlqPuMaHYAkSaNQT2bObXQQ\nkiQVxR5BSZLqFBGbIuJzEbE+In4VEYdU26dHxK0RsS4ibomIadX2V0fEDRHx/6o/f1l9qJaI+EpE\n3BsRP4yItoa9KElSKZkISpL0Um27lYa+r2bdM5k5G/gS8MVq2z8AX8/MOcC1wBXV9iuAn2bmEcDr\ngXur7YcCV2bm64DNwLsLfj2SJO0iMrPRMUiSNKpExLOZuV8f7ZuAt2TmQxHRCjyemZMi4kngoMzc\nVm1/LDNfGRFPAB2Z+ceax5gO/CgzD60u/y3Qmpn/o/hXJklShT2CkiQNTPZzfyD+WHN/B16zL0ka\nYSaCkiQNzPtqbn9ZvX87cHr1/pnAz6v3bwHOAYiIlog4YKSClCRpT/wGUpKkl2qLiLU1yz/IzN4p\nJA6MiHVUevXOqLZ9AvhaRCwGngD+Y7X9PGBpRHyESs/fOcBjhUcvSdJeeI2gJEl1ql4j2JmZE0wW\nZAAAAE5JREFUTzY6FkmShsLSUEmSJEkqGXsEJUmSJKlk7BGUJEmSpJIxEZQkSZKkkjERlCRJkqSS\nMRGUJEmSpJIxEZQkSZKkkjERlCRJkqSS+f//HXTUNnyU3gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xe80178a860>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot train and validation accuracies of the two models\n",
    "\n",
    "train_accs = []\n",
    "val_accs = []\n",
    "for dropout in dropout_choices:\n",
    "  solver = solvers[dropout]\n",
    "  train_accs.append(solver.train_acc_history[-1])\n",
    "  val_accs.append(solver.val_acc_history[-1])\n",
    "\n",
    "plt.subplot(3, 1, 1)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].train_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Train accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "  \n",
    "plt.subplot(3, 1, 2)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].val_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Val accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "\n",
    "plt.gcf().set_size_inches(15, 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Question\n",
    "Explain what you see in this experiment. What does it suggest about dropout?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Answer\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}