assignment2/Dropout.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout\n",
    "Dropout [1] is a technique for regularizing neural networks by randomly setting some features to zero during the forward pass. In this exercise you will implement a dropout layer and modify your fully-connected network to optionally use dropout.\n",
    "\n",
    "[1] [Geoffrey E. Hinton et al, \"Improving neural networks by preventing co-adaptation of feature detectors\", arXiv 2012](https://arxiv.org/abs/1207.0580)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "# As usual, a bit of setup\n",
    "from __future__ import print_function\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from cs231n.classifiers.fc_net import *\n",
    "from cs231n.data_utils import get_CIFAR10_data\n",
    "from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array\n",
    "from cs231n.solver import Solver\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for auto-reloading external modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "def rel_error(x, y):\n",
    "  \"\"\" returns relative error \"\"\"\n",
    "  return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_val:  (1000, 3, 32, 32)\n",
      "X_train:  (49000, 3, 32, 32)\n",
      "y_train:  (49000,)\n",
      "y_val:  (1000,)\n",
      "y_test:  (1000,)\n",
      "X_test:  (1000, 3, 32, 32)\n"
     ]
    }
   ],
   "source": [
    "# Load the (preprocessed) CIFAR10 data.\n",
    "\n",
    "data = get_CIFAR10_data()\n",
    "for k, v in data.items():\n",
    "  print('%s: ' % k, v.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout forward pass\n",
    "In the file `cs231n/layers.py`, implement the forward pass for dropout. Since dropout behaves differently during training and testing, make sure to implement the operation for both modes.\n",
    "\n",
    "Once you have done so, run the cell below to test your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running tests with p =  0.25\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  10.014059116977283\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.749784\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.4\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  9.977917658761159\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.600796\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.7\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  9.987811912159426\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.30074\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(500, 500) + 10\n",
    "\n",
    "for p in [0.25, 0.4, 0.7]:\n",
    "  out, _ = dropout_forward(x, {'mode': 'train', 'p': p})\n",
    "  out_test, _ = dropout_forward(x, {'mode': 'test', 'p': p})\n",
    "\n",
    "  print('Running tests with p = ', p)\n",
    "  print('Mean of input: ', x.mean())\n",
    "  print('Mean of train-time output: ', out.mean())\n",
    "  print('Mean of test-time output: ', out_test.mean())\n",
    "  print('Fraction of train-time output set to zero: ', (out == 0).mean())\n",
    "  print('Fraction of test-time output set to zero: ', (out_test == 0).mean())\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout backward pass\n",
    "In the file `cs231n/layers.py`, implement the backward pass for dropout. After doing so, run the following cell to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dx relative error:  5.44560814873387e-11\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(10, 10) + 10\n",
    "dout = np.random.randn(*x.shape)\n",
    "\n",
    "dropout_param = {'mode': 'train', 'p': 0.2, 'seed': 123}\n",
    "out, cache = dropout_forward(x, dropout_param)\n",
    "dx = dropout_backward(dout, cache)\n",
    "dx_num = eval_numerical_gradient_array(lambda xx: dropout_forward(xx, dropout_param)[0], x, dout)\n",
    "\n",
    "# Error should be around e-10 or less\n",
    "print('dx relative error: ', rel_error(dx, dx_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inline Question 1:\n",
    "What happens if we do not divide the values being passed through inverse dropout by `p` in the dropout layer? Why does that happen?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Answer:\n",
    "The score value expected for an image would not be the same in training and testing.\n",
    "This will fault the prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fully-connected nets with Dropout\n",
    "In the file `cs231n/classifiers/fc_net.py`, modify your implementation to use dropout. Specifically, if the constructor of the net receives a value that is not 1 for the `dropout` parameter, then the net should add dropout immediately after every ReLU nonlinearity. After doing so, run the following to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running check with dropout =  1\n",
      "Initial loss:  2.3004790897684924\n",
      "W1 relative error: 1.48e-07\n",
      "W2 relative error: 2.21e-05\n",
      "W3 relative error: 3.53e-07\n",
      "b1 relative error: 5.38e-09\n",
      "b2 relative error: 2.09e-09\n",
      "b3 relative error: 5.80e-11\n",
      "\n",
      "Running check with dropout =  0.75\n",
      "Initial loss:  2.302371489704412\n",
      "W1 relative error: 1.90e-07\n",
      "W2 relative error: 4.76e-06\n",
      "W3 relative error: 2.60e-08\n",
      "b1 relative error: 4.73e-09\n",
      "b2 relative error: 1.82e-09\n",
      "b3 relative error: 1.70e-10\n",
      "\n",
      "Running check with dropout =  0.5\n",
      "Initial loss:  2.3042759220785896\n",
      "W1 relative error: 3.11e-07\n",
      "W2 relative error: 1.84e-08\n",
      "W3 relative error: 5.35e-08\n",
      "b1 relative error: 2.58e-08\n",
      "b2 relative error: 2.99e-09\n",
      "b3 relative error: 1.13e-10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "N, D, H1, H2, C = 2, 15, 20, 30, 10\n",
    "X = np.random.randn(N, D)\n",
    "y = np.random.randint(C, size=(N,))\n",
    "\n",
    "for dropout in [1, 0.75, 0.5]:\n",
    "  print('Running check with dropout = ', dropout)\n",
    "  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,\n",
    "                            weight_scale=5e-2, dtype=np.float64,\n",
    "                            dropout=dropout, seed=123)\n",
    "\n",
    "  loss, grads = model.loss(X, y)\n",
    "  print('Initial loss: ', loss)\n",
    "  \n",
    "  # Relative errors should be around e-6 or less; Note that it's fine\n",
    "  # if for dropout=1 you have W2 error be on the order of e-5.\n",
    "  for name in sorted(grads):\n",
    "    f = lambda _: model.loss(X, y)[0]\n",
    "    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)\n",
    "    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regularization experiment\n",
    "As an experiment, we will train a pair of two-layer networks on 500 training examples: one will use no dropout, and one will use a keep probability of 0.25. We will then visualize the training and validation accuracies of the two networks over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "(Iteration 1 / 125) loss: 7.856644\n",
      "(Epoch 0 / 25) train acc: 0.260000; val_acc: 0.184000\n",
      "(Epoch 1 / 25) train acc: 0.416000; val_acc: 0.258000\n",
      "(Epoch 2 / 25) train acc: 0.482000; val_acc: 0.276000\n",
      "(Epoch 3 / 25) train acc: 0.532000; val_acc: 0.277000\n",
      "(Epoch 4 / 25) train acc: 0.600000; val_acc: 0.271000\n",
      "(Epoch 5 / 25) train acc: 0.708000; val_acc: 0.299000\n",
      "(Epoch 6 / 25) train acc: 0.722000; val_acc: 0.282000\n",
      "(Epoch 7 / 25) train acc: 0.832000; val_acc: 0.255000\n",
      "(Epoch 8 / 25) train acc: 0.878000; val_acc: 0.269000\n",
      "(Epoch 9 / 25) train acc: 0.902000; val_acc: 0.275000\n",
      "(Epoch 10 / 25) train acc: 0.890000; val_acc: 0.261000\n",
      "(Epoch 11 / 25) train acc: 0.930000; val_acc: 0.282000\n",
      "(Epoch 12 / 25) train acc: 0.958000; val_acc: 0.300000\n",
      "(Epoch 13 / 25) train acc: 0.964000; val_acc: 0.305000\n",
      "(Epoch 14 / 25) train acc: 0.962000; val_acc: 0.318000\n",
      "(Epoch 15 / 25) train acc: 0.964000; val_acc: 0.304000\n",
      "(Epoch 16 / 25) train acc: 0.980000; val_acc: 0.308000\n",
      "(Epoch 17 / 25) train acc: 0.968000; val_acc: 0.322000\n",
      "(Epoch 18 / 25) train acc: 0.990000; val_acc: 0.317000\n",
      "(Epoch 19 / 25) train acc: 0.984000; val_acc: 0.302000\n",
      "(Epoch 20 / 25) train acc: 0.974000; val_acc: 0.307000\n",
      "(Iteration 101 / 125) loss: 0.163264\n",
      "(Epoch 21 / 25) train acc: 0.980000; val_acc: 0.305000\n",
      "(Epoch 22 / 25) train acc: 0.964000; val_acc: 0.317000\n",
      "(Epoch 23 / 25) train acc: 0.962000; val_acc: 0.316000\n",
      "(Epoch 24 / 25) train acc: 0.986000; val_acc: 0.306000\n",
      "(Epoch 25 / 25) train acc: 0.976000; val_acc: 0.304000\n",
      "0.25\n",
      "(Iteration 1 / 125) loss: 17.318480\n",
      "(Epoch 0 / 25) train acc: 0.230000; val_acc: 0.177000\n",
      "(Epoch 1 / 25) train acc: 0.378000; val_acc: 0.243000\n",
      "(Epoch 2 / 25) train acc: 0.402000; val_acc: 0.254000\n",
      "(Epoch 3 / 25) train acc: 0.502000; val_acc: 0.276000\n",
      "(Epoch 4 / 25) train acc: 0.528000; val_acc: 0.298000\n",
      "(Epoch 5 / 25) train acc: 0.562000; val_acc: 0.296000\n",
      "(Epoch 6 / 25) train acc: 0.620000; val_acc: 0.290000\n",
      "(Epoch 7 / 25) train acc: 0.624000; val_acc: 0.299000\n",
      "(Epoch 8 / 25) train acc: 0.682000; val_acc: 0.310000\n",
      "(Epoch 9 / 25) train acc: 0.714000; val_acc: 0.298000\n",
      "(Epoch 10 / 25) train acc: 0.736000; val_acc: 0.302000\n",
      "(Epoch 11 / 25) train acc: 0.762000; val_acc: 0.305000\n",
      "(Epoch 12 / 25) train acc: 0.782000; val_acc: 0.282000\n",
      "(Epoch 13 / 25) train acc: 0.822000; val_acc: 0.302000\n",
      "(Epoch 14 / 25) train acc: 0.818000; val_acc: 0.350000\n",
      "(Epoch 15 / 25) train acc: 0.854000; val_acc: 0.356000\n",
      "(Epoch 16 / 25) train acc: 0.856000; val_acc: 0.304000\n",
      "(Epoch 17 / 25) train acc: 0.846000; val_acc: 0.302000\n",
      "(Epoch 18 / 25) train acc: 0.870000; val_acc: 0.336000\n",
      "(Epoch 19 / 25) train acc: 0.876000; val_acc: 0.317000\n",
      "(Epoch 20 / 25) train acc: 0.884000; val_acc: 0.306000\n",
      "(Iteration 101 / 125) loss: 4.688574\n",
      "(Epoch 21 / 25) train acc: 0.862000; val_acc: 0.307000\n",
      "(Epoch 22 / 25) train acc: 0.910000; val_acc: 0.300000\n",
      "(Epoch 23 / 25) train acc: 0.894000; val_acc: 0.321000\n",
      "(Epoch 24 / 25) train acc: 0.890000; val_acc: 0.319000\n",
      "(Epoch 25 / 25) train acc: 0.898000; val_acc: 0.314000\n"
     ]
    }
   ],
   "source": [
    "# Train two identical nets, one with dropout and one without\n",
    "np.random.seed(231)\n",
    "num_train = 500\n",
    "small_data = {\n",
    "  'X_train': data['X_train'][:num_train],\n",
    "  'y_train': data['y_train'][:num_train],\n",
    "  'X_val': data['X_val'],\n",
    "  'y_val': data['y_val'],\n",
    "}\n",
    "\n",
    "solvers = {}\n",
    "dropout_choices = [1, 0.25]\n",
    "for dropout in dropout_choices:\n",
    "  model = FullyConnectedNet([500], dropout=dropout)\n",
    "  print(dropout)\n",
    "\n",
    "  solver = Solver(model, small_data,\n",
    "                  num_epochs=25, batch_size=100,\n",
    "                  update_rule='adam',\n",
    "                  optim_config={\n",
    "                    'learning_rate': 5e-4,\n",
    "                  },\n",
    "                  verbose=True, print_every=100)\n",
    "  solver.train()\n",
    "  solvers[dropout] = solver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4gAAAJNCAYAAABz4lt9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3XucXWV99/3PzzDIiJhwiEhOJEgapEkgdgARCkiEoHIIqBE8UKyKWil97H2nhvtp05i7Fkqo+FC49Y4VAQ9ASmMMKkYF0eIBmTQ0ETASDpqZYA2HRNQBQ/g9f+w9YWeYSWaS2Xvtw+f9es1rz7rW2mv99mTPZL5zXeu6IjORJEmSJOklRRcgSZIkSaoPBkRJkiRJEmBAlCRJkiSVGRAlSZIkSYABUZIkSZJUZkCUJEmSJAEGREmSiIgREfHbiJhQdC2SJBUpXAdRktRoIuK3FZsvA54Ftpa3P5iZX6p9VZIkNT4DoiSpoUXEo8D7M/M7Ozhmj8x8rnZV1U4zvzZJUu05xFSS1HQi4h8i4uaIuDEingbeHRHHRsSPI2JTRDwWEVdFRFv5+D0iIiNiYnn7i+X9t0XE0xHxo4iYNMC1XhIRt0TEr8rnvjMiXlOx/2URcWVE/DIiNkfE9yPipeV9J5Rr2hwR6yPiPeX2uyLigopzvD8i7uxT619ExDrgZ+X2qyOiKyJ+ExH3RMTrK56/R0T8XUQ8VN7fGRFjIuL/RsQ/9Xk9t0XERbv/ryBJakQGRElSszob+DIwErgZeA74K+AA4DjgNOCDO3j+O4G/A/YDfgn87x0c+zVgMvAq4KfAFyr2XQlMB44pn+t/Ac+XA+fXgU8C+wMzgDVDeH1nAkcB08rbd5evsx9wC/BvvUEUmAu8jdJrHgW8H3gGuB54Z0S8BCAiDgROBG4aQh2SpCZiQJQkNau7MvPWzHw+M3sy857MvDszn8vMh4HFlMLQQG7JzM7M3AJ8CTiyv4PK578uM5/OzGeABcCfRMTeETECuAC4ODMfy8ytmXlX+ZzvBr6ZmUvKNT2emfcO4fX9Y2Y+lZk95Tq+kJlPloebXg68Aji0fOz7gf+VmQ+W6723fOwPgZ6Kr8N5wHcy8/Eh1CFJaiIGRElSs1pfuRERh0XE18tDQX8DLKTUmziQX1V8/nvg5f0dVJ4B9fKIeLh83nXlXQcABwJ7Ag/189TxA7QPVt/X9zcR8bOI2Aw8BezNC69vR9e6gVJYpfz4hQGOkyS1AAOiJKlZ9Z2F7f9SGv55aGa+ApgPxDBc53zgzcDJlIaz9vbaBfDfwB+AV/fzvPUDtAP8jtLsrL1e1c8x215fRLwB+GvgrZSGkO4L/JYXXt+OrvUF4JyImFE+5tYBjpMktQADoiSpVewDbAZ+V55EZkf3Hw71vM8CT1AKdZ/o3ZGZW4HrgE9FxKvKvY3HlSfH+SJwWkS8tTyJzAERcUT5qfcCb42I9oj4I+DPB1HDc8DjQBulYa57V+z/V+AfIuLVUXJkROxXrvEX5etdD/xbeZisJKlFGRAlSa3ifwB/BjxNqTfx5mE67+eBDeWP+4Af9tn/UeABYCXwJPCPlJaZegQ4A/hYuf0/eWHCmSso9RD+GriWUpjckW8A3wEeBB4FfgM8VrF/EbAMuL28bzGwV8X+68vXdnipJLU410GUJKnFRcTJwOeAQ9JfDCSppdmDKElSC4uIPSkt//FZw6EkyYAoSVKLiohplGY83Q+4quByJEl1wCGmkiRJkiTAHkRJkiRJUpkBUZIkSZIEwB5FF1ALBxxwQE6cOLHoMiRJkiSpECtXrnw8M0fv7LiWCIgTJ06ks7Oz6DIkSZIkqRAR8YvBHOcQU0mSJEkSYECUJEmSJJXVVUCMiGsj4tcR8dMB9kdEXBUR6yJidUS8ttY1SpIkSVKzqquACFwHnLaD/W8CJpc/LgQ+XYOaJEmSJKkl1FVAzMzvA0/u4JCzgBuy5MfAqIg4qDbVSZIkSVJzq6uAOAhjgfUV213lNkmSJEnSbmq0ZS6in7bs98CICykNQ2XChAnVrEmSJKnhLFvVzaIVa9mwqYcxo9qZO2sKs2f4d3ep1TVaQOwCxldsjwM29HdgZi4GFgN0dHT0GyIlSZJa0bJV3VyydA09W7YC0L2ph0uWrgEwJKqh+YeP3ddoQ0yXA+eXZzN9HbA5Mx8ruihJkqRGsmjF2m3hsFfPlq0sWrG26tdetqqb4y67g0nzvs5xl93BslXdVb+mWkPvHz66N/WQvPCHD99jQ1NXPYgRcSNwEnBARHQBfw+0AWTmZ4BvAG8G1gG/B95bTKWSJKmZtFqvw4ZNPUNqHy6t2nPZau+vouzoDx9+vQevrgJiZp63k/0JfKRG5UiSpBbQiqFlzKh2uvsJg2NGtVf1uq34C3wrvr+KUtQfPno1yx8CGm2IqSRJ0rAqcrhlUebOmkJ724jt2trbRjB31pSqXrcefoGv9fDWVnx/FWWgP3BU+w8f0FzDWw2IkiSppRUZWoq6H2/2jLFces40xo5qJ4Cxo9q59JxpVe/taMVf4IsOxa2kqD98QHP9IaCuhphKkiTVWlHDLYseejh7xtiaD3+bO2vKdq8Z6uMX+Gp+HYp6f7Wi3n/HIoZ5NtMfAgyIkiTVqaLuZ2m16xYVWlrxfrxW/AW+yFBc5D1xRV27iD98QHP9IcCAKEnSDhQZloroXWq161aev9b/zs3U4zAUrfYLfFHvryK/p4ruHS9CkX8IGG4GREmSBlDkLzlF9S612nV7FRFamqnHoREU+Qt8Ee+vIr+niv5+LkKRvePDzYAoSdIAivwlp6jepVa7bpGaqcehETTTL/CDUeT3VCt+P0NxvePDzYAoSdIAivwlp6jepVa7bpFaLbDUg2b5BX4wivyeasXv52biMheSJA2gyCn5i5quvdWuW7TZM8byg3kn88hlb+EH805umfCi6ivye6pVv5+bhQFRkqQBFPlLTlHr1LXadaVmVeT3lN/PjS0ys+gaqq6joyM7OzuLLkOS1ICKnCZekqThEhErM7NjZ8d5D6IkSTvQSvcsSZJkQJQkNQR78iRJqj4DoiSp7rXiosuSJBXBSWokSXVvR+sRSpKk4WNAlCTVvVZddFmSpFozIEqS6l6R6xFKktRKDIiSpLrnosuSJNWGk9RIkupe70Q0zmIqSUOwegncvhA2d8HIcTBzPkyfU3RVqnMGRElSQ3A9QkkagtVL4NaLYUv5Xu3N60vbYEjUDtXdENOIOC0i1kbEuoiY18/+gyPi9ohYHRF3RsS4IuqUpCItW9XNcZfdwaR5X+e4y+5g2aruokuSJNWT2xe+EA57bekptUs7UFcBMSJGANcAbwIOB86LiMP7HHYFcENmTgcWApfWtkpJKlbvmoDdm3pIXlgT0JAoSdpmc9fQ2rX7Vi+BK6fCglGlx9VLiq5ol9RVQASOBtZl5sOZ+QfgJuCsPsccDtxe/vy7/eyXpKbmmoCSpJ0aOcAgu4HatXt6h/RuXg/kC0N6GzAk1ltAHAusr9juKrdV+i/greXPzwb2iYj9a1CbJNUF1wSUJO3UzPnQ1mcpoLb2UnstNElv2qA10ZDeepukJvppyz7b/xO4OiIuAL4PdAPPvehEERcCFwJMmDBheKuUpAKNGdVOdz9hsBZrAi5b1e1MopLUCHonoiliFtMiJ8gpaubWJhrSW28BsQsYX7E9DthQeUBmbgDOAYiIlwNvzczNfU+UmYuBxQAdHR19Q6YkNay5s6ZwydI12w0zrcWagL33PvZet/feR8CQKKkxtNqyD9PnFPP6dtSbVs16igymI8eVh5f2095g6m2I6T3A5IiYFBF7AucCyysPiIgDIqK37kuAa2tcoyQVavaMsVx6zjTGjmongLGj2rn0nGlVD2ne+yipoTXRPWJ1r6jetCKHeRY9pHcY1VUPYmY+FxEXASuAEcC1mXlfRCwEOjNzOXAScGlEJKUhph8prGBJLa3I4ZZFrAnovY+SGlpRvVqtqKjetCKHeRY5pHeY1VVABMjMbwDf6NM2v+LzW4Bbal2XJFVqxeGWRd77KKmJeI9Y85s5f/uhnlCb3rSih3kWNaR3mNXbEFNJagitONxy7qwptLeN2K6tFvc+SmoiRQ7zdNmH2pk+B864CkaOB6L0eMZV1Q9PTTTMs0h114MoSY2gFYdb9vaMOoup1CSK6MkrcphnUb1araqI3rQmGuZZJAOiJO2CVh1uWcS9j5KqoKjZHr1HTNXWJMM8i2RAlKRdUNRSE5I0LIrqyWvVe8RabXkNNTTvQZSkXVDUUhOSNCyK6slrxXvEXF5DDcYeREnaRQ63lIZZkb0srdbDU1RPXisO83R5DTUYA6IkSSpeUffEFX3tohQ5YUur3SPm8hpqMA4xlSRJxdtRL0szX3v1ErhyKiwYVXqs1bDDopYhaEUur6EGYw+iJEkqXpG9LEVdu+iey1brySuKy2uowdiDKEmSildkL0tR1y6y51K1Y2+tGow9iJIkqXhF9rIUdW3vTWsd9taqgRgQJTW8Zau6WbRiLRs29TBmVDtzZ01xdlGp0RQ5u2VR1y56TUBJ6kdkZtE1VF1HR0d2dnYWXYakKli2qrvfBetdk1BS3et7DyKUei4dfiipCiJiZWZ27Ow470GU1NAWrVi7XTgE6NmylUUr1hZUkSQNkvemSapDDjGV1NA2bOoZUrvUUFpt8fZW5L1pkuqMPYiSGtqYUe1DapcaRu/ww83rgXxhCYRarJNX1Np8kqTCGRAlNbS5s6bQ3jZiu7b2thHMnTWloIqkYVLUEghFBlNJUuEMiJIa2uwZY7n0nGmMHdVOAGNHtTtBjZpDUUsguDafJLU070GU1PBmzxhrIFTzKWoJBNfmk6SWZg+iJEn1aOb80pIHlWqxePtAAdS1+SSpJdRdQIyI0yJibUSsi4h5/eyfEBHfjYhVEbE6It5cRJ2SJFVVUUsgFBVMJUl1oa6GmEbECOAa4BSgC7gnIpZn5v0Vh/0tsCQzPx0RhwPfACbWvFhJkqqtiCUQeq/n8hqS1JLqKiACRwPrMvNhgIi4CTgLqAyICbyi/PlIYENNK5Qkqdm5Np8ktax6C4hjgco78ruAY/ocswD4VkT8JbA38MbalCZJkiRJza3e7kGMftqyz/Z5wHWZOQ54M/CFiHjR64iICyOiMyI6N27cWIVSJUmSJKm51FtA7ALGV2yP48VDSN8HLAHIzB8BewEH9D1RZi7OzI7M7Bg9enSVypUkSZKk5lFvAfEeYHJETIqIPYFzgeV9jvklMBMgIl5DKSDaRShJqo7VS+DKqbBgVOlx9ZKiK5IkqWrq6h7EzHwuIi4CVgAjgGsz876IWAh0ZuZy4H8An42Ij1IafnpBZvYdhipJ0u5bvQRuvRi29JS2N68vbYOTuEiSmlK0Qrbq6OjIzs7OosuQJDWaK6eWQmFfI8fDR39a+3okSdpFEbEyMzt2dly9DTGVJKl+bO4aWrskSQ3OgChJ0kBGjhtauyRJDc6AKElqDEVMFjNzPrS1b9/W1l5qlySpCdXVJDWSJPWrqMlies99+8LSsNKR40rh0AlqJElNyoAoSap/ty98IRz22tJTaq92WJs+x0AoSWoZDjGVJNU/J4uRJKkmDIiSpPrnZDGSJNWEAVGSVP+cLEaSpJowIEqS6t/0OXDGVaUF6onS4xlXeW+gJEnDzElqJEmNwcliJEmqOnsQJUmDV8RahJIkqWbsQZQkDU5RaxFKkqSasQdRkjQ4O1qLUJIkNQUDoiRpcFyLUJKkpucQU0nDYtmqbhatWMuGTT2MGdXO3FlTmD1jbNFlaTiNHFcaVtpfuyRJagr2IErabctWdXPJ0jV0b+ohge5NPVyydA3LVnUXXZqGk2sRSpLU9AyIknbbohVr6dmydbu2ni1bWbRibUEVtYAiZhN1LUJJkpqeQ0wl7bYNm3qG1K7dVORsoq5FKElSU7MHUWoyy1Z1c9xldzBp3tc57rI7ajLMc8yo9iG1azc5m6gkSaqSuguIEXFaRKyNiHURMa+f/VdGxL3lj59HxKYi6pTqUVH3As6dNYX2thHbtbW3jWDurClVvW7LcjZRSZJUJXUVECNiBHAN8CbgcOC8iDi88pjM/GhmHpmZRwL/AiytfaVSfSrqXsDZM8Zy6TnTGDuqnQDGjmrn0nOmOYtptQw0a6iziUqSpN1Ub/cgHg2sy8yHASLiJuAs4P4Bjj8P+Psa1SbVvSLvBZw9Y6yBsFZmzt/+HkRwNlFJkjQs6qoHERgLVC6y1VVue5GIOBiYBNxRg7qkhuC9gC3C2UQlSVKV1FsPYvTTlgMcey5wS2Zu7W9nRFwIXAgwYcKE4alOqnNzZ03hkqVrthtm6r2ATcrZRCVJUhXUWw9iFzC+YnscsGGAY88FbhzoRJm5ODM7MrNj9OjRw1iiVL+8F1CSJEm7o956EO8BJkfEJKCbUgh8Z9+DImIKsC/wo9qWJ9U/7wWsodVLSktLbO4qTRAzc769epIkqaHVVUDMzOci4iJgBTACuDYz74uIhUBnZi4vH3oecFNmDjT8VJKqq8jF6iVJkqokWiFjdXR0ZGdnZ9FlSGomV04thcK+Ro6Hj/609vVIkiTtQESszMyOnR1XlXsQI+KiiNi3GueWpLrgYvWSJKkJVWuSmlcB90TEkog4LSL6m51UkhqXi9VLkqQmVJWAmJl/C0wGPgdcADwYEf8YEa+uxvUkqeZmzi8tTl/JxeolSVKDq9oyF+UJZH5V/niO0qyjt0TE5dW6piTVjIvVS5KkJlSVWUwj4mLgz4DHgX8F5mbmloh4CfAg8DfVuK4k1ZSL1UuSpCZTrWUuDgDOycxfVDZm5vMRcXqVrilJkiRJ2g3VGmL6DeDJ3o2I2CcijgHIzAeqdE1JkiRJ0m6oVkD8NPDbiu3fldskSZIkSXWqWgExypPUAKWhpVRvOKskSZIkaRhUKyA+HBEXR0Rb+eOvgIerdC1JkiRJ0jCoVkD8EPB6oBvoAo4BLqzStSRJkiRJw6Aqwz4z89fAudU4tyRJkiSpOqq1DuJewPuAPwb26m3PzD+vxvUkSZIkSbuvWkNMvwC8CpgFfA8YBzxdpWtJkiRJkoZBtQLioZn5d8DvMvN64C3AtCpdS1KrW70ErpwKC0aVHlcvKboiSZKkhlStpSe2lB83RcRU4FfAxCpdS1I9WL0Ebl8Im7tg5DiYOR+mz6nNdW+9GLb0lLY3ry9tQ22uL0mS1ESq1YO4OCL2Bf4WWA7cD/xTla4lqWi9IW3zeiBfCGm16Mm7feEL4bDXlp5SuyRJkoZk2HsQI+IlwG8y8yng+8Ahw30NSXVmRyGt2r14m7uG1i5JkqQBDXsPYmY+D1w03OeVVMeKDGkjxw2tXZIkSQOq1hDTb0fE/4yI8RGxX+9Hla4lqWhFhrSZ86Gtffu2tvZSuyRJkoakWgHxz4GPUBpiurL80TmYJ0bEaRGxNiLWRcS8AY6ZExH3R8R9EfHlYata0q4pMqRNnwNnXAUjxwNRejzjKieokSRJ2gVVmcU0MyftyvMiYgRwDXAK0AXcExHLM/P+imMmA5cAx2XmUxHxyuGoWRpuy1Z1s2jFWjZs6mHMqHbmzprC7Bljiy6rOnrDWBGzmPZe30AoSZK026oSECPi/P7aM/OGnTz1aGBdZj5cPs9NwFmUZkHt9QHgmvIkOGTmr3e/Yml4LVvVzSVL19CzZSsA3Zt6uGTpGoDmDomGNEmSpIZWrSGmR1V8/CmwADhzEM8bC6yv2O4qt1X6I+CPIuIHEfHjiDht98uVhteiFWu3hcNePVu2smjF2oIqkiRJknauWkNM/7JyOyJGAl8YxFOjv9P12d4DmAycBIwD/iMipmbmpj7XvBC4EGDChAmDK1waJhs29QypXZIkSaoH1epB7Ov3lELdznQB4yu2xwEb+jnmq5m5JTMfAdb2d+7MXJyZHZnZMXr06F0sW9o1Y0a1D6ldkiRJqgdVCYgRcWtELC9/fI1SiPvqIJ56DzA5IiZFxJ7AucDyPscsA95Qvs4BlIacPjx81Uu7b+6sKbS3jdiurb1tBHNnTSmoIkmSJGnnqjLEFLii4vPngF9k5k5XzM7M5yLiImAFMAK4NjPvi4iFQGdmLi/vOzUi7ge2AnMz84nhfwnSruudiKZlZjGVJElSU4jMvrf4DcNJIyYBj2XmM+XtduDAzHx02C82CB0dHdnZOahlGCVJkiSp6UTEyszs2Nlx1boH8d+A5yu2t5bbJEmSJEl1qloBcY/M/EPvRvnzPat0LUmSJEnSMKhWQNwYEdvWPYyIs4DHq3QtSZIkSdIwqNYkNR8CvhQRV5e3u4Dzq3QtSZVWL4HbF8LmLhg5DmbOh+lziq5KkiRJDaAqATEzHwJeFxEvpzQRztPVuI6kPlYvgVsvhi09pe3N60vbYEiUJEnSTlVrHcR/jIhRmfnbzHw6IvaNiH+oxrUkVbh94QvhsNeWnlK7JEmStBPVugfxTZm5qXcjM58C3lyla0nqtXmA5UYHapckSZIqVCsgjoiIl/ZulNdBfOkOjpc0HEaOG1q7JEmSVKFaAfGLwO0R8b6IeB/wbeD6Kl1LUq+Z86Gtffu2tvZSuyRJkrQT1Zqk5vKIWA28EQjgm8DB1biWpAq9E9E4i6kkSZJ2QbWWuQD4FfA8MAd4BPj3Kl5LUq/pcwyEkiRJ2iXDGhAj4o+Ac4HzgCeAmyktc/GG4byONFjLVnWzaMVaNmzqYcyodubOmsLsGWOLLkuSJEmqS8Pdg/gz4D+AMzJzHUBEfHSYryENyrJV3VyydA09W7YC0L2ph0uWrgEwJEqSJEn9GO5Jat5KaWjpdyPisxExk9I9iFLNLVqxdls47NWzZSuLVqytTQGrl8CVU2HBqNLj6iW1ua4kSZK0i4Y1IGbmVzLzHcBhwJ3AR4EDI+LTEXHqcF5L2pkNm3qG1D6sVi+BWy+GzeuBLD3eerEhUZIkSXWtKstcZObvMvNLmXk6MA64F5hXjWtJAxkzqn1I7cPq9oWwpU8Q3dJTapckSZLqVLXWQdwmM5/MzP+bmSdX+1pSpbmzptDeNmK7tva2EcydNaX6F9/cNbR2SZIkqQ5Uc5kLqVC9E9EUMovpyHHl4aX9tEuSJEl1yoCopjZ7xthiZiydOb90z2HlMNO29lK7JEmSVKeqPsRUaknT58AZV8HI8UCUHs+4ygXsJUmSVNfqrgcxIk4D/j9gBPCvmXlZn/0XAIuA7nLT1Zn5rzUtUo1j9ZLSxDCbu0rDO2fOr11Imz7HQChJkqSGUlcBMSJGANcApwBdwD0RsTwz7+9z6M2ZeVHNC1Rj6V1qoneYZ+9SE2BwkyRJkvpRb0NMjwbWZebDmfkH4CbgrIJrUqNyqQlJkiRpSOotII4FKqd+7Cq39fXWiFgdEbdExPjalKaG41ITkiRJ0pDUW0CMftqyz/atwMTMnA58B7i+3xNFXBgRnRHRuXHjxmEuUw1hoCUlXGpCkiRJ6le9BcQuoLJHcBywofKAzHwiM58tb34W+JP+TpSZizOzIzM7Ro8eXZViVedmzi8tLVHJpSYkSZKkAdXVJDXAPcDkiJhEaZbSc4F3Vh4QEQdl5mPlzTOBB2pbonbFslXdtV+wvncimqJmMZUkSZIaTF0FxMx8LiIuAlZQWubi2sy8LyIWAp2ZuRy4OCLOBJ4DngQuKKxgDcqyVd1csnQNPVu2AtC9qYdLlq4BqE1INBBKkiRJgxKZfW/xaz4dHR3Z2dlZdBkt67jL7qB7U8+L2seOaucH804uoCJJkiSptUTEyszs2Nlx9XYPoprQhn7C4Y7aJUmSJBXDgKiqGzOqfUjtkiRJkophQFTVzZ01hfa2Edu1tbeNYO6sKQVVJEmSJKk/dTVJjZpT70Q0NZ/FVJIkSdKQGBBVE7NnjDUQSpIkSXXOIaaSJEmSJMCAKEmSJEkqMyBKkiRJkgADoiRJkiSpzIAoSZIkSQIMiJIkSZKkMgOiJEmSJAkwIEqSJEmSygyIkiRJkiTAgChJkiRJKjMgSpIkSZIA2KPoAlQ7y1Z1s2jFWjZs6mHMqHbmzprC7Bljiy5LkiRJUp0wILaIZau6uWTpGnq2bAWge1MPlyxdA2BIlCRJkgQ4xLRlLFqxdls47NWzZSuLVqwtqCJJkiRJ9abuAmJEnBYRayNiXUTM28Fxb4uIjIiOWtbXqDZs6hlSuyRJkqTWU1cBMSJGANcAbwIOB86LiMP7OW4f4GLg7tpW2LjGjGofUvuwW70ErpwKC0aVHlcvqc11JUmSJA1aXQVE4GhgXWY+nJl/AG4CzurnuP8NXA48U8viGtncWVNobxuxXVt72wjmzppS/YuvXgK3Xgyb1wNZerz1YkOiJEmSVGfqLSCOBdZXbHeV27aJiBnA+Mz8Wi0La3SzZ4zl0nOmMXZUOwGMHdXOpedMq80ENbcvhC19hrJu6Sm1S5IkSaob9TaLafTTltt2RrwEuBK4YKcnirgQuBBgwoQJw1ReY5s9Y2wxM5Zu7hpauyRJkqRC1FsPYhcwvmJ7HLChYnsfYCpwZ0Q8CrwOWN7fRDWZuTgzOzKzY/To0VUsWTs1ctzQ2iVJkiQVot4C4j3A5IiYFBF7AucCy3t3ZubmzDwgMydm5kTgx8CZmdlZTLkalJnzoa3PZDht7aV2SZIkSXWjrgJiZj4HXASsAB4AlmTmfRGxMCLOLLY67bLpc+CMq2DkeCBKj2dcVWqXJEmSVDciM3d+VIPr6OjIzk47GVm9pDQxzOau0vDOmfMNaZIkSVILiIiVmbnTNeTrbZIaVUvvUhO9s4n2LjUBhkRJkiRJQJ0NMVUVudSEJEmSpJ0wILYKl5qQJEmStBMGxFbhUhOSJEmSdsKA2CpcakKSJEnSThgQW4VLTUiSJEnaCWcxbSXT5xgIJUmSJA3IHkRJkiRJEmBAlCRJkiSVGRAlSZIkSYABUZIkSZJUZkCUJEmSJAEGREmSJElSmQFRkiRJkgQYECVJkiRJZQZESZIkSRJgQJQkSZIklRkQJUmSJEmAAVGSJEmSVGZAlCRJkiQBdRgQI+K0iFgbEesiYl4/+z8UEWsi4t6IuCsiDi+iTkl/R2kaAAAgAElEQVSSJElqNnUVECNiBHAN8CbgcOC8fgLglzNzWmYeCVwOfLLGZUqSJElSU6qrgAgcDazLzIcz8w/ATcBZlQdk5m8qNvcGsob1SZIkSVLT2qPoAvoYC6yv2O4Cjul7UER8BPhrYE/g5NqUJkmSJEnNrd56EKOfthf1EGbmNZn5auBjwN/2e6KICyOiMyI6N27cOMxlSpIkSVLzqbeA2AWMr9geB2zYwfE3AbP725GZizOzIzM7Ro8ePYwlSpIkSVJzqreAeA8wOSImRcSewLnA8soDImJyxeZbgAdrWJ8kSZIkNa26ugcxM5+LiIuAFcAI4NrMvC8iFgKdmbkcuCgi3ghsAZ4C/qy4iiVJkiSpedRVQATIzG8A3+jTNr/i87+qeVGSJEmS1ALqbYipJEmSJKkgBkRJkiRJEmBAlCRJkiSV1d09iJIkSWotW7Zsoauri2eeeaboUqSGt9deezFu3Dja2tp26fkGxAIsW9XNohVr2bCphzGj2pk7awqzZ4wtuixJkqRCdHV1sc8++zBx4kQiouhypIaVmTzxxBN0dXUxadKkXTqHQ0xrbNmqbi5ZuobuTT0k0L2ph0uWrmHZqu6iS5MkSSrEM888w/777284lHZTRLD//vvvVm+8AbHGFq1YS8+Wrdu19WzZyqIVawuqSJIkqXiGQ2l47O73kgGxxjZs6hlSuyRJkqrvz//8z3nlK1/J1KlTBzwmM7n44os59NBDmT59Ov/5n/+5bd/111/P5MmTmTx5Mtdff/1Or3fnnXdy+umnD0vtu+K6665jw4YNhV1/uH3zm99kypQpHHrooVx22WX9HvPJT36Sww8/nOnTpzNz5kx+8YtfbNs3YsQIjjzySI488kjOPPPMnV7v0Ucf3eF7pdqWLVvG/fffX5VzGxBrbMyo9iG1S5IkqfouuOACvvnNb+7wmNtuu40HH3yQBx98kMWLF/PhD38YgCeffJKPf/zj3H333fzkJz/h4x//OE899dQu1fHcc8/t0vOGqpkC4tatW/nIRz7Cbbfdxv3338+NN97Yb3iaMWMGnZ2drF69mre97W38zd/8zbZ97e3t3Hvvvdx7770sX758l2up1b+fAbGJzJ01hfa2Edu1tbeNYO6sKQVVJEmS1FiWrermuMvuYNK8r3PcZXcMy1wOJ5xwAvvtt98Oj/nqV7/K+eefT0Twute9jk2bNvHYY4+xYsUKTjnlFPbbbz/23XdfTjnllH7D5je/+U0OO+wwjj/+eJYuXbqtfcGCBVx44YWceuqpnH/++TzzzDO8973vZdq0acyYMYPvfve7QCnUnXXWWZx22mlMmTKFj3/849vO8clPfpKpU6cydepUPvWpTwEv7uW64oorWLBgAbfccgudnZ28613v4sgjj6Snp8Yj2VYvgSunwoJRpcfVS3brdD/5yU849NBDOeSQQ9hzzz0599xz+epXv/qi497whjfwspe9DIDXve51dHV1Dek6K1eu5IgjjuDYY4/lmmuu2dZ+3XXX8fa3v50zzjiDU089lcxk7ty5TJ06lWnTpnHzzTcDpV7jE044gbPPPpvDDz+cD33oQzz//PMA3HjjjUybNo2pU6fysY99bNu5X/7yl2/7/JZbbuGCCy7ghz/8IcuXL2fu3LkceeSRPPTQQ0N6HTvjLKY11jtbqbOYSpIkDV3vhH+9czr0TvgHVP33qe7ubsaPH79te9y4cXR3dw/YXumZZ57hAx/4AHfccQeHHnoo73jHO7bbv3LlSu666y7a29v553/+ZwDWrFnDz372M0499VR+/vOfA6Uw9NOf/pSXvexlHHXUUbzlLW8hIvj85z/P3XffTWZyzDHHcOKJJ7Lvvvv2+zre9ra3cfXVV3PFFVfQ0dExLF+bQVu9BG69GLaUQ+nm9aVtgOlzdumU/X3977777h0+53Of+xxvetObtm0/88wzdHR0sMceezBv3jxmz579oue8973v5V/+5V848cQTmTt37nb7fvSjH7F69Wr2228//v3f/517772X//qv/+Lxxx/nqKOO4oQTTgBK/373338/Bx98MKeddhpLly7l9a9/PR/72MdYuXIl++67L6eeeirLli3rtwaA17/+9Zx55pmcfvrpvO1tbxv012mwDIgFmD1jrIFQkiRpF+xowr9q/36VmS9qi4gB2yv97Gc/Y9KkSUyePBmAd7/73SxevHjb/jPPPJP29tItR3fddRd/+Zd/CcBhhx3GwQcfvC0gnnLKKey///4AnHPOOdx1111EBGeffTZ77733tvb/+I//GNS9dDV3+8IXwmGvLT2l9l0MiIP5+lf64he/SGdnJ9/73ve2tf3yl79kzJgxPPzww5x88slMmzaNV7/61dv2b968mU2bNnHiiScC8J73vIfbbrtt2/7eHmQo/fudd955jBgxggMPPJATTzyRe+65h1e84hUcffTRHHLIIQCcd9553HXXXbS1tXHSSScxevRoAN71rnfx/e9/f8CAWG0OMS3CMHerS5IktYoiJ/wbN24c69ev37bd1dXFmDFjBmzva0ehpTfcQf+BZ6BzDBRQAfbYY49tQxiB3Vr6YNhsHmBY50DtgzDYrz/Ad77zHT7xiU+wfPlyXvrSl25r7z3+kEMO4aSTTmLVqlXbPS8za/7v1/f4Wv37GRBrrbdbffN6IF/oVjckSpIk7VSRE/6deeaZ3HDDDWQmP/7xjxk5ciQHHXQQs2bN4lvf+hZPPfUUTz31FN/61reYNWvWds897LDDeOSRR7bdL3bjjTcOeJ0TTjiBL33pSwD8/Oc/55e//CVTppTmq/j2t7/Nk08+SU9PD8uWLeO4447jhBNOYNmyZfz+97/nd7/7HV/5ylf40z/9Uw488EB+/etf88QTT/Dss8/yta99bds19tlnH55++unh/hLt3MhxQ2sfhKOOOooHH3yQRx55hD/84Q/cdNNN/faerlq1ig9+8IMsX76cV77yldvan3rqKZ599lkAHn/8cX7wgx9w+OGHb/fcUaNGMXLkSO666y6Abf8+/TnhhBO4+eab2bp1Kxs3buT73/8+Rx99NFAaYvrII4/w/PPPc/PNN3P88cdzzDHH8L3vfY/HH3+crVu3cuONN27rqTzwwAN54IEHeP755/nKV76y7RrV/PczINbajrrVJUmStEPVmvDvvPPO49hjj2Xt2rWMGzeOz33ucwB85jOf4TOf+QwAb37zmznkkEM49NBD+cAHPsD/+T//B4D99tuPv/u7v+Ooo47iqKOOYv78+S+a8GavvfZi8eLFvOUtb+H444/n4IMPHrCWv/iLv2Dr1q1MmzaNd7zjHVx33XXberuOP/543vOe93DkkUfy1re+lY6ODl772tdywQUXcPTRR3PMMcfw/ve/nxkzZtDW1sb8+fM55phjOP300znssMO2XeOCCy7gQx/6UO0nqZk5H9r6hPm29lL7Ltpjjz24+uqrmTVrFq95zWuYM2cOf/zHfwzA/Pnzt81KOnfuXH7729/y9re/fbvlLB544AE6Ojo44ogjeMMb3sC8efNeFBABPv/5z/ORj3yEY489dttw4P6cffbZTJ8+nSOOOIKTTz6Zyy+/nFe96lUAHHvsscybN4+pU6cyadIkzj77bA466CAuvfRS3vCGN3DEEUfw2te+lrPOOguAyy67jNNPP52TTz6Zgw46aNs1zj33XBYtWsSMGTOGfZKa2FGXZrPo6OjIzs7OossoWTAK6O9rHrBgU62rkSRJKtwDDzzAa17zmkEfv2xVd0tO+HfdddfR2dnJ1VdfXXQpu2f1klLnyOauUs/hzPm7fP9hI7nzzju54oortuvJrZb+vqciYmVm7nRWIiepqbWR48rDS/tplyRJ0k454V+Dmz6nJQJho3KIaa1VoVtdkiRJze+CCy5o/N7DFnbSSSfVpPdwdxkQa236HDjjKhg5HojS4xlX+VcUSZIkSYVziGkR7FaXJEnazs6WEZA0OLs7x0zd9SBGxGkRsTYi1kXEvH72/3VE3B8RqyPi9ogYeAooSZIk1b299tqLJ554Yrd/sZVaXWbyxBNPsNdee+3yOeqqBzEiRgDXAKcAXcA9EbE8M++vOGwV0JGZv4+IDwOXA++ofbWSJEkaDuPGjaOrq4uNGzcWXYrU8Pbaay/Gjdv1CTDrKiACRwPrMvNhgIi4CTgL2BYQM/O7Fcf/GHh3TSuUJEnSsGpra2PSpElFlyGJ+htiOhaoXAOiq9w2kPcBt1W1IkmSJElqEfXWg9jfncn9DkaPiHcDHcCJA+y/ELgQYMKECcNVnyRJkiQ1rXrrQewCxldsjwM29D0oIt4I/L/AmZn5bH8nyszFmdmRmR2jR4+uSrGSJEmS1EyinmaLiog9gJ8DM4Fu4B7gnZl5X8UxM4BbgNMy88FBnncj8Ivhr3i3HQA8XnQRalq+v1RNvr9UTb6/VG2+x1RN9fr+Ojgzd9pzVlcBESAi3gx8ChgBXJuZn4iIhUBnZi6PiO8A04DHyk/5ZWaeWVC5uyUiOjOzo+g61Jx8f6mafH+pmnx/qdp8j6maGv39VW/3IJKZ3wC+0adtfsXnb6x5UZIkSZLUAurtHkRJkiRJUkEMiMVaXHQBamq+v1RNvr9UTb6/VG2+x1RNDf3+qrt7ECVJkiRJxbAHUZIkSZIEGBALERGnRcTaiFgXEfOKrkfNJyIejYg1EXFvRHQWXY8aW0RcGxG/joifVrTtFxHfjogHy4/7FlmjGtcA768FEdFd/hl2b3mGc2nIImJ8RHw3Ih6IiPsi4q/K7f4M027bwfuroX+GOcS0xiJiBKW1Hk8Buiit9XheZt5faGFqKhHxKNCRmfW4Bo8aTEScAPwWuCEzp5bbLgeezMzLyn/o2jczP1ZknWpMA7y/FgC/zcwriqxNjS8iDgIOysz/jIh9gJXAbOAC/Bmm3bSD99ccGvhnmD2ItXc0sC4zH87MPwA3AWcVXJMkDSgzvw882af5LOD68ufXU/oPURqyAd5f0rDIzMcy8z/Lnz8NPACMxZ9hGgY7eH81NANi7Y0F1ldsd9EEbyTVnQS+FRErI+LCootRUzowMx+D0n+QwCsLrkfN56KIWF0egurwP+22iJgIzADuxp9hGmZ93l/QwD/DDIi1F/20Oc5Xw+24zHwt8CbgI+UhXJLUKD4NvBo4EngM+Odiy1Gji4iXA/8O/D+Z+Zui61Fz6ef91dA/wwyItdcFjK/YHgdsKKgWNanM3FB+/DXwFUpDm6Xh9N/ley9678H4dcH1qIlk5n9n5tbMfB74LP4M026IiDZKv7x/KTOXlpv9GaZh0d/7q9F/hhkQa+8eYHJETIqIPYFzgeUF16QmEhF7l2+UJiL2Bk4FfrrjZ0lDthz4s/LnfwZ8tcBa1GR6f3EvOxt/hmkXRUQAnwMeyMxPVuzyZ5h220Dvr0b/GeYspgUoT3X7KWAEcG1mfqLgktREIuIQSr2GAHsAX/Y9pt0RETcCJwEHAP8N/D2wDFgCTAB+Cbw9M51oREM2wPvrJEpDsxJ4FPhg7/1i0lBExPHAfwBrgOfLzf+L0n1i/gzTbtnB++s8GvhnmAFRkiRJkgQ4xFSSJEmSVGZAlCRJkiQBBkRJkiRJUpkBUZIkSZIEGBAlSZIkSWUGREmSBikitkbEvRUf84bx3BMjoqHWypIkNZ89ii5AkqQG0pOZRxZdhCRJ1WIPoiRJuykiHo2If4qIn5Q/Di23HxwRt0fE6vLjhHL7gRHxlYj4r/LH68unGhERn42I+yLiWxHRXtiLkiS1JAOiJEmD195niOk7Kvb9JjOPBq4GPlVuuxq4ITOnA18Criq3XwV8LzOPAF4L3Fdunwxck5l/DGwC3lrl1yNJ0nYiM4uuQZKkhhARv83Ml/fT/ihwcmY+HBFtwK8yc/+IeBw4KDO3lNsfy8wDImIjMC4zn604x0Tg25k5ubz9MaAtM/+h+q9MkqQSexAlSRoeOcDnAx3Tn2crPt+KcwVIkmrMgChJ0vB4R8Xjj8qf/xA4t/z5u4C7yp/fDnwYICJGRMQralWkJEk74l8mJUkavPaIuLdi+5uZ2bvUxUsj4m5Kf3w9r9x2MXBtRMwFNgLvLbf/FbA4It5Hqafww8BjVa9ekqSd8B5ESZJ2U/kexI7MfLzoWiRJ2h0OMZUkSZIkAfYgSpIkSZLK7EGUJKksIiZGREaE9+hLklqSAVGS1DQiYkVELOyn/ayI+JXBT5KkHTMgSpKayXXAeyIi+rS/B/hSZj5X+5KGR5T4/7Ykqar8j0aS1EyWAfsBf9rbEBH7AqcDN5S33xIRqyLiNxGxPiIWDPbkETEvIh6KiKcj4v6IOLvP/g9ExAMV+19bbh8fEUsjYmNEPBERV5fbF0TEFyuev90Q14i4MyI+ERE/AH4PHBIR7624xsMR8cE+NZwVEfeWX99DEXFaRLw9Ilb2Oe5/RMSywb52SVJrMCBKkppGZvYAS4DzK5rnAD/LzP8qb/+uvH8U8BbgwxExe5CXeIhS+BwJfBz4YkQcBBARbwcWlM/9CuBM4ImIGAF8DfgFMBEYC9w0hJf1HuBCYJ/yOX5NKfC+gtK6ildWBNGjKQXhueXXdwLwKLAcmBQRr6k477uBLwyhDklSCzAgSpKazfXA2yOivbx9frkNgMy8MzPXZObzmbkauBE4cTAnzsx/y8wN5efeDDwIHF3e/X7g8sy8J0vWZeYvyvvHAHMz83eZ+Uxm3jWE13NdZt6Xmc9l5pbM/HpmPlS+xveAb/FCj+n7gGsz89vlGrsz82eZ+SxwM6VQSET8MaWw+rUh1CFJagEGRElSUymHr43AWRFxCHAU8OXe/RFxTER8tzzcczPwIeCAwZw7Is4vD9/cFBGbgKkVzx1PqYexr/HAL3bj/sf1fWp4U0T8OCKeLNfw5kHUAKWQ/M7y/ZnvAZaUg6MkSdsYECVJzegGSj2H7wG+lZn/XbHvy5SGXI7PzJHAZ4C+k9q8SEQcDHwWuAjYPzNHAT+teO564NX9PHU9MGGAGVR/B7ysYvtV/RyzbcHiiHgp8O/AFcCB5Rq+MYgayMwfA3+g1Nv4ThxeKknqhwFRktSMbgDeCHyAiuGlZfsAT2bmM+V79t45yHPuTSmsbQSIiPdS6kHs9a/A/4yIPynPOHpoOVT+BHgMuCwi9o6IvSLiuPJz7gVOiIgJETESuGQnNewJvLRcw3MR8Sbg1Ir9nwPeGxEzI+IlETE2Ig6r2H8DcDXw3BCHuUqSWoQBUZLUdDLzUeCHlELd8j67/wJYGBFPA/MpTWozmHPeD/wz8CPgv4FpwA8q9v8b8AlKPZRPU55RNTO3AmcAhwK/BLqAd5Sf821K9wauBlayk3sCM/Np4OJyzU9RCrfLK/b/hPLENcBm4HvAwRWn+AKlUGvvoSSpX5GZOz9KkiQ1vPLEPb8GXpuZDxZdjySp/tiDKElS6/gwcI/hUJI0kP5umJckSU0mIh6lNJnNYNd8lCS1IIeYSpIkSZIAh5hKkiRJksoMiJIkSZIkoEXuQTzggANy4sSJRZchSZIkSYVYuXLl45k5emfHtURAnDhxIp2dnUWXIUmSJEmFiIhfDOY4h5hKkiRJkgADoiRJkiSpzIAoSZIkSQIMiJIkSZKkMgOiJEmSJAkwIEqSJEmSygyIkiSpPqxeAldOhQWjSo+rlxRdkSS1nJZYB1GSJNW51Uvg1othS09pe/P60jbA9DnF1SVJLcYeREmSVLzbF74QDntt6Sm1S5JqxoAoSZKKt7lraO2SpKowIEqSpOKNHDe0dklSVRgQJUlS8WbOh7b27dva2kvtkqSaqWpAjIjTImJtRKyLiHn97P9QRKyJiHsj4q6IOLzcPjEiesrt90bEZyqe8yfl56yLiKsiIqr5GiRJUg1MnwNnXAUjxwNRejzjKieokaQaq9osphExArgGOAXoAu6JiOWZeX/FYV/OzM+Ujz8T+CRwWnnfQ5l5ZD+n/jRwIfBj4Bvl42+rzquQJKkFrV5Smhxmc1dpiOfM+bUJatPnGAglqWDV7EE8GliXmQ9n5h+Am4CzKg/IzN9UbO4N5I5OGBEHAa/IzB9lZgI3ALOHt2xJklpY73ITm9cD+cJyE65JKEktoZoBcSywvmK7q9y2nYj4SEQ8BFwOXFyxa1JErIqI70XEn1acs3I6s37PKUmSdpHLTUhSS6tmQOzv3sAX9RBm5jWZ+WrgY8DflpsfAyZk5gzgr4EvR8QrBntOgIi4MCI6I6Jz48aNu/QCJElqOS43IUktrZoBsQsYX7E9Dtiwg+NvojxcNDOfzcwnyp+vBB4C/qh8zsr5rgc8Z2YuzsyOzOwYPXr0Lr8ISZJaistNSFJLq2ZAvAeYHBGTImJP4FxgeeUBETG5YvMtwIPl9tHlSW6IiEOAycDDmfkY8HREvK48e+n5wFer+BokSWotLjchSS2tarOYZuZzEXERsAIYAVybmfdFxEKgMzOXAxdFxBuBLcBTwJ+Vn34CsDAingO2Ah/KzCfL+z4MXAe0U5q91BlMJUkaLr2ziBYxi6kkqXBRmgy0uXV0dGRnZ2fRZUiSJElSISJiZWZ27Oy4ag4xlSRJkiQ1EAOiJEmSJAkwIEqSJEmSygyIkiRJkiTAgChJkiRJKjMgSpIkSZIAA6IkSZIkqcyAKEmSJEkCDIiSJEmSpDIDoiRJUitavQSunAoLRpUeVy8puiJJdWCPoguQJElSja1eArdeDFt6Stub15e2AabPKa4uSYWzB1GSJKnV3L7whXDYa0tPqV1SSzMgSpIktZrNXUNrl9QyDIiSJEmtZuS4obVLahkGREmSpP+/vbsPk7MsD/7/PdkEsogmvEQkL0CQPKExicnTBVRoQBESFEKgGEEUERRR+VHr0ZTwVCnkpw8UqPKj0Na0ClgRiAgxViFSXrSxiNmYNOEtEkFhN1gSICiyIW/n74+ZXSbLJtlNdvbOznw/xzHHzH3Ofd9z3TN3JnvOdV3nXW+OuxQGNm4ZG9hYikuqayaIkiRJ9WbCDDj5Ohg8EojS/cnXWaBGklVMJUmS6tKEGSaEkt7AHkRJkiRJEmCCKEmSJEkqM0GUJEmSJAEmiJIkSZKkMhNESZIkSRJQ5QQxIqZGxIqIWBkRs7p4/oKIWB4RSyNiYUSMLcePj4jF5ecWR8T7KrZ5sLzPpeXbW6t5DJIkSZJUL6p2mYuIaABuAI4HWoBFETE/Mx+rWO07mfnP5fWnAV8FpgJrgJMzc1VEjAMWAMMrtjsrM5ur1XZJkiRJqkfV7EE8AliZmU9l5nrgNuCUyhUy8/cVi28CshxfkpmryvFHgUERsUcV2ypJkiRJda9qPYiUevyerVhuAY7svFJEfA74ArA78L7OzwN/DizJzNcqYjdGxCbge8CXMzN7rdWSJEmSVKeq2YMYXcTekMhl5g2Z+XbgYuCLW+wg4h3A3wGfrgiflZnjgT8r3z7W5YtHnB8RzRHRvHr16h08BEmSpOqZt6SVo668n1GzfshRV97PvCWtRTdJUp2rZoLYAoysWB4BrNrKulAagjq9fSEiRgB3AWdn5q/b45nZWr7/A/AdSkNZ3yAz52RmU2Y2DR06dIcPQpIkqRrmLWnlkjuX07q2jQRa17ZxyZ3LTRIlFaqaCeIiYHREjIqI3YEzgPmVK0TE6IrFDwJPluNDgB8Cl2TmzyrWHxAR+5UfDwROAh6p4jFIkiRVxdULVtC2YdMWsbYNm7h6wYqCWiT1omVz4Wvj4LIhpftlc4tukbqpanMQM3NjRFxIqQJpA/DNzHw0ImYDzZk5H7gwIt4PbABeAj5e3vxC4FDgSxHxpXLsBOCPwIJyctgA/AfwL9U6BkmSpGpZtbatR3Gp31g2F35wEWwon8svP1taBpgwo7h2qVuqWaSGzPwR8KNOsUsrHv/FVrb7MvDlrez2T3utgZIkSQUZNqSR1i6SwWFDGgtojdSL7pv9enLYbkNbKV7tBHHZ3NLrvNwCg0fAcZealPZQNYeYSpKkfsjCKX1j5pQxNA5s2CLWOLCBmVPGFNQiqZe83NKzeG9p77l8+VkgX++5dHhrj1S1B1GSJPUv7YVT2ufGtRdOAZg+aXiRTas57e/n1QtWsGptG8OGNDJzypg+e5/nLWkt5rXt4ekzhX3Gg0eUk7Qu4tVUZM9lDTFBlCRJHbZVOMUEsfdNnzS8kPe1sB8CnJvWZwr9see4S7f8nAEGNpbi1VRUz2WNcYipJEnqYOGU+lBYBdVt9fCoVxVaJXfCDBaNv5zfMZTNGfyOoSwaf3n1fwTYWg9ltXsua4w9iJIkqYOFU+pDYT8E2MPTZ4r8sWfeklYuWXQQbRv+v45Y46IGrhjZWt3ey6J6LmuMPYiSJKmDhVPqw9YS/qr/EGAPT58p7DOmwN7LCTPg5Otg8EggSvcnX9d3w5dr5NqPJoiSJKnD9EnDueK08Qwf0kgAw4c0csVp451/WGMK+yHguEtLPTqV7OGpiiJ/7Cl0qPqEGfCXj8Bla0v3fZkc1kgFVYeYSpKkLRRVOEV9p7AKqu1/rFvFtOqKrJJbl0PVa6iCamRm0W2ouqampmxubi66GZIkSVLN61xBFUq9lzU9GuGyIUBXeVWUejN3ARGxODObtreePYiSJEmSek3R1/gsRFHXfqwCE0RJ2lFe7FmSpC7V3VD1GqqgaoIoSTvCiz1LkqR2NTS/1gRRknZEDU1Gl1SaM1VXw+Ek9b4JM2ribwATREnaEV7sWaoZnQtqtK5t45I7lwOYJEqqO14HUZJ2hBd7lmpGYRf1Vp+bt6SVo668n1GzfshRV97PvCWtffPCNXIBddUHexAlaUfU0GR0qd4VelFv9Zl5S1pZeNc/cju3MWyPNax6dT+uvesM4LPV7Sl2znqfcrj4zrMHUZJ2xIQZcPJ1MHgkEKX7k6/zP/tqqsNf4Avr7agzW7t4d01f1Bb9dB8AACAASURBVLsOLf3hHGbHHEbstobdAkbstobZMYelP5xT3Rfe1px19ar24eKta9tIXh8u7ndnz9iDKEk7qkYmo/cLdfgLfD3Oiyvql/+ZU8aw8K5/5PPcxrBYw6rcj2s5g6OnfLbqr62+88n132bP3dZvEdsz1vPJ9d8GLq/a6+bLLUQP4tpx2xou3hffJbXSe2kPoiRp11eHv8DX27y4In/5n97wM64c+K9b9CxdOfBfmd7ws6q/tvrOsN1e6FG8t/wP+/Uorh1X5HDxWuq9NEGUJO366rBqbL3Niys0Ib5vNgM2rdsiNGDTupr+AaIerWt8W4/iveWK9R/i1dx9i9iruTtXrP9QVV+3HhU5XLyWftQzQZQkdV9R8wDrsGrssCGNTNttIQt3v4in9vgIC3e/iGm7LazZeXGFJsRF/gDh3No+62HZ88TZbGwYtEVsY8Mg9jyxuj8ENL/leGZt+CQtm/djcwYtm/dj1oZP0vyW46v6uvVo5pQxNA5s2CLWOLCBmVPGVP21a+lHvaomiBExNSJWRMTKiJjVxfMXRMTyiFgaEQsjYmzFc5eUt1sREVO6u09JUpW0zwN8+VkgX58H2Bd/0B53aalKbKUarxp77dgn+btOwx7/buC/cu3YJ4tuWlUUWiimqB8givw3VZBCh+FNmMGAU/5hi+JiA075h6rPY545ZQz3NhzD0euv45DXbuHo9ddxb8MxfZK01Jvpk4ZzxWnjGT6kkQCGD2nkitPG98k8wFoqdhWZWZ0dRzQAvwKOB1qARcCZmflYxTpvyczflx9PAz6bmVPLieKtwBHAMOA/gP9V3myb++xKU1NTNjc39+bhSVL9+dq48h+ynQweCX/5SPVff9nc0pC/l1tKf7gfd2mfFKgprOhA0e93H+tclAdKv/z3yR93nYsgQekHiGpXJq6zzxjgqCvvp7WLHpXhQxr52az3FdCivlHY90hB35v1qNDvsG6KiMWZ2bS99apZxfQIYGVmPlVu0G3AKUBHMteeHJa9CWjPVk8BbsvM14CnI2JleX9sb59SvauVClo94n+AfaPoeYAFVI0ttJJo0e93H2t/Pwv5/mo/r/r6e6TOPmOorWF4PTF90vC+/7+4Dqs/F6nQ77BeVs0EcThQ+bNYC3Bk55Ui4nPAF4DdgfafjoYDP++0bfu7u919SvWqHsvi+x9gHxo8Yiu9HbU7D7DQkul1+H4X8kd0uyIuW1OHn/GwIY1d9iD2x2F4u7xtVX/2/8eqKPQ7rBdVcw5il5d8eUMg84bMfDtwMfDF7WzbrX0CRMT5EdEcEc2rV6/uZpOlXlRA4YFaqqDVbXV4+QOgmMIWdTgPsNDejjp8v+tOHX7GRRYRqTt12EOt3lHNBLEFGFmxPAJYtY31bwOmb2fbbu8zM+dkZlNmNg0dOrSHTZd2UkGFB+py6E49/gdYVGGLCTNKc7IqCjxUfY5WwQotOlCH73fdqcPPuMgiInWnDqs/q3dUc4jpImB0RIwCWoEzgI9UrhARozOzvRzbB4H2x/OB70TEVykVqRkN/IJSD+I29yntEgoa1jFsSCN/+vt7+esBcxkWa1iV+3HVxhksruVS2nU4RKvQYUNFDMMr0MwpY7osOtBnvR119n7XpTr8jGtlGN4u77hLuy6+VMM91OodVUsQM3NjRFwILAAagG9m5qMRMRtozsz5wIUR8X5gA/AS8PHyto9GxFxKxWc2Ap/LzE0AXe2zWscg7bCCerWuHfsk4xb/K42xHoARUSqL/8jYg3l9im+Nqcf/AOux17QgtVR0QFKdKar4kvq9ql3mYlfiZS7U54oqXV6HJdOB+qtiWq+fsyRJ2mG7wmUupPpVVK9WvfYs1dsQrXrsNZUkSX2imkVqpPpVVOEBJ6TXhzosbCFJkvqGQ0ylWtL5moBQ6lkyeZAkSapr3R1iag+iVEvsWZIkSdJOcA6iVGuKmo9Xb4ViJEmSapAJoqSd13loa/uF28EkUZIkqR9xiKmknbetC7dLkiSp3zBBlLTz6vXyGpIkSTXGBFHSzvPyGpIkSTXBBFHSzjvu0tLlNCp54XZJkqR+xwRR0s7z8hqSJEk1wSqmknpHUZfXkCRJUq+xB1GSJEmSBJggSpIkSZLKHGIqqVfMW9LK1QtWsGptG8OGNDJzyhimTxpedLNUQwo7x5bNLV3T8+WWUmXe4y51OLUkqWZtN0GMiAuBWzLzpT5oj6R+aN6SVi65czltGzYB0Lq2jUvuXA5gkqheUdg5tmwu/OAi2NBWWn752dIymCRKkmpSd4aYvg1YFBFzI2JqRES1GyWpf7l6wYqOP9zbtW3YxNULVhTUItWaws6x+2a/nhy229BWikuSVIO2myBm5heB0cA3gHOAJyPi/0bE26vcNkn9xKq1bT2KSz1V2Dn2ckvP4pIk9XPdKlKTmQn8rnzbCOwN3BERV1WxbZL6iWFDGnsUl3qqsHNs8IiexSVJ6ue2myBGxEURsRi4CvgZMD4zPwP8KfDnVW6fpH5g5pQxNA5s2CLWOLCBmVPGFNQi1ZrCzrHjLoWBnZLQgY2luCRJNag7VUz3A07LzN9WBjNzc0ScVJ1mSb3Dypp9o/099b1WtRR2jrUXorGKqSSpTkRp9Og2Voh4F/BoZv6hvPxmYGxmPtwH7esVTU1N2dzcXHQz6loRidq8Ja0svOsf+Ty3MSzWsCr341rO4OhTP2viIkmSpLoSEYszs2l763VnDuI/Aa9ULP+xHOtOI6ZGxIqIWBkRs7p4/gsR8VhELIuI+yLioHL8vRGxtOK2LiKml5+7KSKernhuYnfaouK0l6dvXdtG8np5+nlLWqv6ukt/OIfZMYcRu61ht4ARu61hdsxh6Q/nVPV1JUmSpP6qOwliZEU3Y2ZupnvXT2wAbgBOBMYCZ0bE2E6rLQGaMnMCcAeleY5k5gOZOTEzJwLvA14Fflyx3cz25zNzaTeOQQUqqjz9J9d/mz1j/RaxPWM9n1z/7aq+rtQX5i1p5agr72fUrB9y1JX3V/0HF0mSVB+6kyA+VS5UM7B8+wvgqW5sdwSwMjOfysz1wG3AKZUrlBPBV8uLPwe6Kgt3OnB3xXrqZ4oqTz9stxd6FJf6i6J65SVJUu3rToJ4AfAeoBVoAY4Ezu/GdsOBZyuWW8qxrTkPuLuL+BnArZ1iXykPS/1aROzRjbaoQEWVp1/X+LYexaX+orCLxkuSpJq33QQxM5/PzDMy862ZuX9mfiQzn+/GvqOr3XW5YsRHgSbg6k7xA4DxwIKK8CXAYcDhwD7AxVvZ5/kR0RwRzatXr+5Gc1UtRZWn3/PE2WxsGLRFbGPDIPY8cXZVX1eqtsIuGi9Jkmped+YSDqLUu/cOoOOv7cw8dzubtgAjK5ZHAKu62P/7gb8BjsnM1zo9PQO4KzM3VLzuc+WHr0XEjcBfdfXimTkHmAOlKqbbaauqqMjy9ANgi/L0A/qwPL2X2FC1DBvSSGsXyWDVLxovSZJqXneug/hvwBPAFGA2cBbweDe2WwSMjohRlIanngF8pHKFiJgEfB2YupVeyTMp9RhWbnNAZj4XEQFMBx7pRltUsOmThheTHE2YUcj1ytrniLUPA2yfIwaYJGqnzZwyZovzC/roovGSJKnmdWcO4qGZ+SXgj5l5M/BBSsM+tykzNwIXUhoe+jgwNzMfjYjZETGtvNrVwF7Ad8uXrJjfvn1EHEypB/InnXZ9S0QsB5YD+wFf7sYxSH3KOWKqpumThnPFaeMZPqSRAIYPaeSK08b744MkSdpp3elBbB/euTYixgG/Aw7uzs4z80fAjzrFLq14/P5tbPsbuihqk5nv685rS0Vyjlh9KHIYcWG98pIkqaZ1J0GcExF7A18E5lPq8ftSVVsl9XPOEat9DiOWJEm1aJtDTCNiN+D3mflSZv40Mw8pVzP9eh+1T+qXiqrcqr7jMGJJklSLtpkgZuZmSvMIJfWAc8Rqn8OIJUlSLerOENN7I+KvgNuBP7YHM/PFqrVKqgHOEattDiOWJEm1qDtVTM8FPgf8FFhcvjVXs1GStKtzGLEkSapF2+1BzMxRfdEQSepP2nuHi6piKkmSVA3bTRAj4uyu4pn5rd5vjiT1Hw4jliRJtaY7cxAPr3g8CDgO+CVggihJkiRJNaQ7Q0z/n8rliBgM/FvVWiRJkiRJKkR3ehA7exUY3dsNkSTt+uYtaXXepSRJNaw7cxB/AGR5cTdgLDC3mo2SJO165i1p5ZI7l9O2YRMArWvbuOTO5QAmiZIk1Yju9CBeU/F4I/DbzGypUnskSbuoqxes6EgO27Vt2MTVC1aYIEqSVCO6kyA+AzyXmesAIqIxIg7OzN9UtWXqdQ4Nk7QzVq1t61FckiT1P7t1Y53vApsrljeVY+pH2oeGta5tI3l9aNi8Ja1FN01SPzFsSGOP4pIkqf/pToI4IDPXty+UH+9evSapGrY1NEySumPmlDE0DmzYItY4sIGZU8YU1CJJktTbupMgro6Iae0LEXEKsKZ6TVI1ODRM0s6aPmk4V5w2nuFDGglg+JBGrjhtvEPVJUmqId2Zg3gBcEtEXF9ebgHOrl6TVA3DhjTS2kUy6NAw1QLn1/ad6ZOG+95KklTDttuDmJm/zsx3Ubq8xTsy8z2ZubL6TVNvKnxo2LK58LVxcNmQ0v0yr5Si3uH8WkmSpN6z3QQxIv5vRAzJzFcy8w8RsXdEfLkvGqfeU+jQsGVz4QcXwcvPAlm6/8FFJonqFc6vlSRJ6j3dGWJ6Ymb+n/aFzHwpIj4AfLF6zVI1FDY07L7ZsKHT8NYNbaX4hBl93x7VFOfXSpIk9Z7uFKlpiIg92hciohHYYxvrS1t6uaVncakHvPSCJElS7+lOgvht4L6IOC8izgPuBW7uzs4jYmpErIiIlRExq4vnvxARj0XEsoi4LyIOqnhuU0QsLd/mV8RHRcTDEfFkRNweEV5yY1c3eETP4lIPFD6/VpIkqYZ0p0jNVcCXgT+hVKjmHuCgbW4EREQDcANwYnm7MyNibKfVlgBNmTkBuAO4quK5tsycWL5Nq4j/HfC1zBwNvASct722qGDHXQoDO/XmDGwsxaWd5KUXJEmSek935iAC/A7YDMwAnga+141tjgBWZuZTABFxG3AK8Fj7Cpn5QMX6Pwc+uq0dRkQA7wM+Ug7dDFwG/FN3DkIFaZ9neN/s0rDSwSNKyaHzD9VLvPSCJElS79hqghgR/ws4AzgTeAG4HYjMfG839z0ceLZiuQU4chvrnwfcXbE8KCKagY3AlZk5D9gXWJuZGyv26V+F/cGEGSaEkiRJ0i5uWz2ITwD/CZzcft3DiPjLHuw7uohllytGfBRoAo6pCB+Ymasi4hDg/ohYDvy+B/s8Hzgf4MADD+xBsyVJkiSpPm1rDuKfUxpa+kBE/EtEHEfXSd/WtAAjK5ZHAKs6rxQR7wf+BpiWma+1xzNzVfn+KeBBYBKwBhgSEe2JbZf7LG83JzObMrNp6NChPWi2JEmSJNWnrSaImXlXZn4YOIxSgvaXwP4R8U8RcUI39r0IGF2uOro7peGq8ytXiIhJwNcpJYfPV8T3br+0RkTsBxwFPJaZCTwAnF5e9ePA97t1pJIkSZKkbepOFdM/ZuYtmXkSpR67pcAbLlnRxXYbgQuBBcDjwNzMfDQiZkdEe1XSq4G9gO92upzFnwDNEfHflBLCKzOzvbjNxcAXImIlpTmJ3+juwUqSJEmSti5KnXK1rampKZubm4tuhiRJkiQVIiIWZ2bT9tbbbg+iJEmSJKk+mCDWk2Vz4Wvj4LIhpftlc4tukSRJkqRdyLYuc6Fasmwu/OAi2NBWWn752dIyeH1CSZIkSYA9iPXjvtmvJ4ftNrSV4pIkSZKECWL9eLmlZ3FJkiRJdccEsV4MHtGzuCRJkqS6Y4JYL467FAY2bhkb2FiKS5IkSRImiPVjwgw4+ToYPBKI0v3J11mgRpIkSVIHq5jWkwkzTAglSZIkbZU9iJIkSZIkwARRkiRJklRmgihJkiRJAkwQJUmSJEllJoiSJEmSJMAEUZIkSZJUZoIoSZIkSQJMECVJkiRJZSaIkiRJkiTABFGSJEmSVGaCKEmSJEkCTBAlSZIkSWVVTRAjYmpErIiIlRExq4vnvxARj0XEsoi4LyIOKscnRsRDEfFo+bkPV2xzU0Q8HRFLy7eJ1TwGSZIkSaoXVUsQI6IBuAE4ERgLnBkRYzuttgRoyswJwB3AVeX4q8DZmfkOYCpwbUQMqdhuZmZOLN+WVusYJEmSJKmeVLMH8QhgZWY+lZnrgduAUypXyMwHMvPV8uLPgRHl+K8y88ny41XA88DQKrZVkiRJkupeNRPE4cCzFcst5djWnAfc3TkYEUcAuwO/rgh/pTz09GsRsUdvNFaSJEmS6l01E8ToIpZdrhjxUaAJuLpT/ADg34BPZObmcvgS4DDgcGAf4OKt7PP8iGiOiObVq1fv2BFIkiRJUh2pZoLYAoysWB4BrOq8UkS8H/gbYFpmvlYRfwvwQ+CLmfnz9nhmPpclrwE3UhrK+gaZOSczmzKzaehQR6dKkiRJ0vZUM0FcBIyOiFERsTtwBjC/coWImAR8nVJy+HxFfHfgLuBbmfndTtscUL4PYDrwSBWPQZIkSZLqxoBq7TgzN0bEhcACoAH4ZmY+GhGzgebMnE9pSOlewHdL+R7PZOY0YAYwGdg3Is4p7/KccsXSWyJiKKUhrEuBC6p1DJIkSZJUTyKzy2mBNaWpqSmbm5uLboYkSZIkFSIiFmdm0/bWq+YQU0mSJElSP2KCKEmSJEkCTBAlSZIkSWUmiJIkSZIkwARRkiRJklRmgihJkiRJAkwQJUmSJEllJoiSJEmSJMAEUZIkSZJUZoIoSZIkSQJMECVJkiRJZSaIkiRJkiTABFGSJEmSVGaCKEmSJEkCTBAlSZIkSWUmiJIkSZIkwARRkiRJklRmgihJkiRJAkwQJUmSJEllJoiSJEmSJMAEUZIkSZJUZoIoSZIkSQKqnCBGxNSIWBERKyNiVhfPfyEiHouIZRFxX0QcVPHcxyPiyfLt4xXxP42I5eV9XhcRUc1jkCRJkqR6UbUEMSIagBuAE4GxwJkRMbbTakuApsycANwBXFXedh/gb4EjgSOAv42Ivcvb/BNwPjC6fJtarWOQJEmSpHpSzR7EI4CVmflUZq4HbgNOqVwhMx/IzFfLiz8HRpQfTwHuzcwXM/Ml4F5gakQcALwlMx/KzAS+BUyv4jFIkiRJUt2oZoI4HHi2YrmlHNua84C7t7Pt8PLj7u5TkiRJktRNA6q4767mBmaXK0Z8FGgCjtnOtj3Z5/mUhqJy4IEHbq+tkiRJklT3qtmD2AKMrFgeAazqvFJEvB/4G2BaZr62nW1beH0Y6lb3CZCZczKzKTObhg4dusMHIUmSJEn1opoJ4iJgdESMiojdgTOA+ZUrRMQk4OuUksPnK55aAJwQEXuXi9OcACzIzOeAP0TEu8rVS88Gvl/FY5AkSZKkulG1IaaZuTEiLqSU7DUA38zMRyNiNtCcmfOBq4G9gO+Wr1bxTGZOy8wXI+L/pZRkAszOzBfLjz8D3AQ0UpqzeDeSJEmSpJ0WpWKgta2pqSmbm5uLboYkSZIkFSIiFmdm0/bWq+YQU0mSJElSP2KCKEmSJEkCTBAlSZIkSWUmiJIkSZIkwARRkiRJklRmgihJkiRJAkwQJUmSJEllJoiSJEmSJMAEUZIkSZJUZoIoSZIkSQJMECVJkiRJZQOKbkA9mreklasXrGDV2jaGDWlk5pQxTJ80vOhmSZIkFWLDhg20tLSwbt26opsi9XuDBg1ixIgRDBw4cIe2N0HsY/OWtHLJnctp27AJgNa1bVxy53IAk0RJklSXWlpaePOb38zBBx9MRBTdHKnfykxeeOEFWlpaGDVq1A7twyGmfezqBSs6ksN2bRs2cfWCFQW1SJIkqVjr1q1j3333NTmUdlJEsO++++5Ub7wJYh9btbatR3FJkqR6YHIo9Y6d/bdkgtjHhg1p7FFckiRJ1Xfuuefy1re+lXHjxm11nczkoosu4tBDD2XChAn88pe/7Hju5ptvZvTo0YwePZqbb755u6/34IMPctJJJ/VK23fETTfdxKpVqwp7/d52zz33MGbMGA499FCuvPLKLtf56le/ytixY5kwYQLHHXccv/3tbzuea2hoYOLEiUycOJFp06Zt9/V+85vfbPNcqbZ58+bx2GOPVWXfJoh9bOaUMTQObNgi1jiwgZlTxhTUIkmSpP5l3pJWjrryfkbN+iFHXXk/85a07vQ+zznnHO65555trnP33Xfz5JNP8uSTTzJnzhw+85nPAPDiiy9y+eWX8/DDD/OLX/yCyy+/nJdeemmH2rFx48Yd2q6nCk0Ql82Fr42Dy4aU7pfN3andbdq0ic997nPcfffdPPbYY9x6661dJk+TJk2iubmZZcuWcfrpp/PXf/3XHc81NjaydOlSli5dyvz583e4LX31+Zkg1pDpk4ZzxWnjGT6kkQCGD2nkitPGW6BGkiSpG9oL/rWubSN5veDfziaJkydPZp999tnmOt///vc5++yziQje9a53sXbtWp577jkWLFjA8ccfzz777MPee+/N8ccf32Wyec8993DYYYdx9NFHc+edd3bEL7vsMs4//3xOOOEEzj77bNatW8cnPvEJxo8fz6RJk3jggQeAUlJ3yimnMHXqVMaMGcPll1/esY+vfvWrjBs3jnHjxnHttdcCb+zluuaaa7jsssu44447aG5u5qyzzmLixIm0tfXhVKdlc+EHF8HLzwJZuv/BRTuVJP7iF7/g0EMP5ZBDDmH33XfnjDPO4Pvf//4b1nvve9/LnnvuCcC73vUuWlpaevQ6ixcv5p3vfCfvfve7ueGGGzriN910Ex/60Ic4+eSTOeGEE8hMZs6cybhx4xg/fjy33347UOo1njx5Mqeeeipjx47lggsuYPPmzQDceuutjB8/nnHjxnHxxRd37HuvvfbqeHzHHXdwzjnn8F//9V/Mnz+fmTNnMnHiRH7961/36Di2xyqmBZg+abgJoSRJ0g7YVsG/av991draysiRIzuWR4wYQWtr61bjldatW8enPvUp7r//fg499FA+/OEPb/H84sWLWbhwIY2Njfz93/89AMuXL+eJJ57ghBNO4Fe/+hVQSoYeeeQR9txzTw4//HA++MEPEhHceOONPPzww2QmRx55JMcccwx77713l8dx+umnc/3113PNNdfQ1NTUK+9Nt903GzZ0Skg3tJXiE2bs0C67ev8ffvjhbW7zjW98gxNPPLFjed26dTQ1NTFgwABmzZrF9OnT37DNJz7xCf7hH/6BY445hpkzZ27x3EMPPcSyZcvYZ599+N73vsfSpUv57//+b9asWcPhhx/O5MmTgdLn99hjj3HQQQcxdepU7rzzTt7znvdw8cUXs3jxYvbee29OOOEE5s2b12UbAN7znvcwbdo0TjrpJE4//fRuv0/dZQ+iJEmS+o0iC/5l5htiEbHVeKUnnniCUaNGMXr0aCKCj370o1s8P23aNBobSzUpFi5cyMc+9jEADjvsMA466KCOBPH4449n3333pbGxkdNOO42FCxeycOFCTj31VN70pjex1157cdppp/Gf//mfvXLMve7lrfTabS3eDd15/yt9+9vfprm5eYsk75lnnqG5uZnvfOc7fP7zn39Dr9zLL7/M2rVrOeaYYwA6Pp927T3IUPr8zjzzTBoaGth///055phjWLRoEQBHHHEEhxxyCA0NDZx55pksXLiQRYsWceyxxzJ06FAGDBjAWWedxU9/+tMdezN6gQliEXp53LUkSVK9KLLg34gRI3j22Wc7lltaWhg2bNhW451tK2l505ve1PG4q4Rna/vYWoIKMGDAgI4hjMBOXfqg1wwe0bN4N3T3/Qf4j//4D77yla8wf/589thjj454+/qHHHIIxx57LEuWLNliu8zs88+v8/p99flVNUGMiKkRsSIiVkbErC6enxwRv4yIjRFxekX8vRGxtOK2LiKml5+7KSKernhuYjWPoddVYdy1JElSvSiy4N+0adP41re+RWby85//nMGDB3PAAQcwZcoUfvzjH/PSSy/x0ksv8eMf/5gpU6Zsse1hhx3G008/3dEzdeutt271dSZPnswtt9wCwK9+9SueeeYZxowpHd+9997Liy++SFtbG/PmzeOoo45i8uTJzJs3j1dffZU//vGP3HXXXfzZn/0Z+++/P88//zwvvPACr732Gv/+7//e8RpvfvOb+cMf/tDbb9H2HXcpDOyUzA9sLMV30OGHH86TTz7J008/zfr167ntttu6rES6ZMkSPv3pTzN//nze+ta3dsRfeuklXnvtNQDWrFnDz372M8aOHbvFtkOGDGHw4MEsXLgQoOPz6crkyZO5/fbb2bRpE6tXr+anP/0pRxxxBFAaYvr000+zefNmbr/9do4++miOPPJIfvKTn7BmzRo2bdrErbfe2tFTuf/++/P444+zefNm7rrrro7XqObnV7U5iBHRANwAHA+0AIsiYn5mVpbbeQY4B/irym0z8wFgYnk/+wArgR9XrDIzM++oVturqgrjriVJkupF+zzDqxesYNXaNoYNaWTmlDE7Pf/wzDPP5MEHH2TNmjWMGDGCyy+/nPPOO49//ud/BuCCCy7gAx/4AD/60Y849NBD2XPPPbnxxhsB2GefffjSl77E4YcfDsCll176hoI3gwYNYs6cOXzwgx9kv/324+ijj+aRRx7psi2f/exnueCCCxg/fjwDBgzgpptu6ujtOvroo/nYxz7GypUr+chHPtIxh/Ccc87pSEI++clPMmnSpI62HHnkkYwaNYrDDjus4zXOOeccLrjgAhobG3nooYc6hrdWXfvfu/fNLg0rHTyilBzuxN/BAwYM4Prrr2fKlCls2rSJc889l3e84x1A6RtDgAAAB/1JREFU6fibmpqYNm0aM2fO5JVXXuFDH/oQAAceeCDz58/n8ccf59Of/jS77bYbmzdvZtasWW9IEAFuvPFGzj33XPbcc883/ABQ6dRTT+Whhx7ine98JxHBVVddxdve9jaeeOIJ3v3udzNr1iyWL1/eUbBmt91244orruC9730vmckHPvABTjnlFACuvPJKTjrpJEaOHMm4ceN45ZVXADjjjDP41Kc+xXXXXccdd9zB29/+9h1+/zqLbXVp7tSOI94NXJaZU8rLlwBk5hVdrHsT8O9dJX0RcT5wTGaetb11t6apqSmbm5t35DB632VDgK7e84DL1vZ1ayRJkgr3+OOP8yd/8idFN2OXd9NNN9Hc3Mz1119fdFO0Ax588EGuueaaLXpyq6Wrf1MRsTgzt1uVqJpDTIcDz1Yst5RjPXUG0LkP/isRsSwivhYRe3S10S6rCuOuJUmSJKk3VDNB7GoWZ4+6KyPiAGA8sKAifAlwGHA4sA9wcRebEhHnR0RzRDSvXr26Jy9bXVUYdy1JkqTad84559h72I8de+yxfdJ7uLOqmSC2ACMrlkcAq3q4jxnAXZm5oT2Qmc9lyWvAjcARXW2YmXMysykzm4YOHdrDl62iCTPg5Otg8EggSvcnX+f8Q0mSJEmFq1qRGmARMDoiRgGtlIaKfqSH+ziTUo9hh4g4IDOfi1LN1+lA17N7d2UTZpgQSpIkVdjeZQQkdc/O1pipWg9iZm4ELqQ0PPRxYG5mPhoRsyNiGkBEHB4RLcCHgK9HxKPt20fEwZR6IH/Sade3RMRyYDmwH/Dlah2DJEmSqm/QoEG88MILO/2HrVTvMpMXXniBQYMG7fA+qlbFdFeyS1UxlSRJ0hY2bNhAS0vLrnEhd6mfGzRoECNGjGDgwIFbxLtbxbSaQ0wlSZKk7Ro4cCCjRo0quhmSqG6RGkmSJElSP2KCKEmSJEkCTBAlSZIkSWV1UaQmIlYDvy26HV3YD1hTdCNUszy/VE2eX6omzy9Vm+eYqmlXPb8OysztXiC+LhLEXVVENHenkpC0Izy/VE2eX6omzy9Vm+eYqqm/n18OMZUkSZIkASaIkiRJkqQyE8RizSm6Aappnl+qJs8vVZPnl6rNc0zV1K/PL+cgSpIkSZIAexAlSZIkSWUmiAWIiKkRsSIiVkbErKLbo9oTEb+JiOURsTQimotuj/q3iPhmRDwfEY9UxPaJiHsj4sny/d5FtlH911bOr8siorX8HbY0Ij5QZBvVf0XEyIh4ICIej4hHI+IvynG/w7TTtnF+9evvMIeY9rGIaAB+BRwPtACLgDMz87FCG6aaEhG/AZoyc1e8Bo/6mYiYDLwCfCszx5VjVwEvZuaV5R+69s7Mi4tsp/qnrZxflwGvZOY1RbZN/V9EHAAckJm/jIg3A4uB6cA5+B2mnbSN82sG/fg7zB7EvncEsDIzn8rM9cBtwCkFt0mStiozfwq82Cl8CnBz+fHNlP5DlHpsK+eX1Csy87nM/GX58R+Ax4Hh+B2mXrCN86tfM0Hse8OBZyuWW6iBE0m7nAR+HBGLI+L8ohujmrR/Zj4Hpf8ggbcW3B7VngsjYll5CKrD/7TTIuJgYBLwMH6HqZd1Or+gH3+HmSD2vegi5jhf9bajMvN/AycCnysP4ZKk/uKfgLcDE4HngL8vtjnq7yJiL+B7wOcz8/dFt0e1pYvzq19/h5kg9r0WYGTF8ghgVUFtUY3KzFXl++eBuygNbZZ60/+U5160z8F4vuD2qIZk5v9k5qbM3Az8C36HaSdExEBKf7zfkpl3lsN+h6lXdHV+9ffvMBPEvrcIGB0RoyJid+AMYH7BbVINiYg3lSdKExFvAk4AHtn2VlKPzQc+Xn78ceD7BbZFNab9D/eyU/E7TDsoIgL4BvB4Zn614im/w7TTtnZ+9ffvMKuYFqBc6vZaoAH4ZmZ+peAmqYZExCGUeg0BBgDf8RzTzoiIW4Fjgf2A/wH+FpgHzAUOBJ4BPpSZFhpRj23l/DqW0tCsBH4DfLp9vpjUExFxNPCfwHJgczn8fyjNE/M7TDtlG+fXmfTj7zATREmSJEkS4BBTSZIkSVKZCaIkSZIkCTBBlCRJkiSVmSBKkiRJkgATREmSJElSmQmiJEndFBGbImJpxW1WL+774IjoV9fKkiTVngFFN0CSpH6kLTMnFt0ISZKqxR5ESZJ2UkT8JiL+LiJ+Ub4dWo4fFBH3RcSy8v2B5fj+EXFXRPx3+fae8q4aIuJfIuLRiPhxRDQWdlCSpLpkgihJUvc1dhpi+uGK536fmUcA1wPXlmPXA9/KzAnALcB15fh1wE8y853A/wYeLcdHAzdk5juAtcCfV/l4JEnaQmRm0W2QJKlfiIhXMnOvLuK/Ad6XmU9FxEDgd5m5b0SsAQ7IzA3l+HOZuV9ErAZGZOZrFfs4GLg3M0eXly8GBmbml6t/ZJIkldiDKElS78itPN7aOl15reLxJqwVIEnqYyaIkiT1jg9X3D9UfvxfwBnlx2cBC8uP7wM+AxARDRHxlr5qpCRJ2+Ivk5IkdV9jRCytWL4nM9svdbFHRDxM6cfXM8uxi4BvRsRMYDXwiXL8L4A5EXEepZ7CzwDPVb31kiRth3MQJUnaSeU5iE2ZuabotkiStDMcYipJkiRJAuxBlCRJkiSV2YMoSZIkSQJMECVJkiRJZSaIkiRJkiTABFGSJEmSVGaCKEmSJEkCTBAlSZIkSWX/P0um+KH/ZO76AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1080x1080 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot train and validation accuracies of the two models\n",
    "\n",
    "train_accs = []\n",
    "val_accs = []\n",
    "for dropout in dropout_choices:\n",
    "  solver = solvers[dropout]\n",
    "  train_accs.append(solver.train_acc_history[-1])\n",
    "  val_accs.append(solver.val_acc_history[-1])\n",
    "\n",
    "plt.subplot(3, 1, 1)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].train_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Train accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "  \n",
    "plt.subplot(3, 1, 2)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].val_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Val accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "\n",
    "plt.gcf().set_size_inches(15, 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inline Question 2:\n",
    "Compare the validation and training accuracies with and without dropout -- what do your results suggest about dropout as a regularizer?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Answer:\n",
    "Prevent overfitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inline Question 3:\n",
    "Suppose we are training a deep fully-connected network for image classification, with dropout after hidden layers (parameterized by keep probability p). How should we modify p, if at all, if we decide to decrease the size of the hidden layers (that is, the number of nodes in each layer)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Answer:\n",
    "As we decrease the size of the hidden layers, one shall increase the probability parameter p. The system will be less likely to overfitting, therefore no need for a strong regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}