03. Gradient Descent and its variants/3.02 Performing Gradient Descent in Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Performing Gradient Descent in Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have understood how the gradient descent algorithm finds the optimal parameters of the model, in this section, we will understand how can we use gradient descent in linear regression and find the optimal parameter.\n",
    "\n",
    "\n",
    "The equation of a simple linear regression can be expressed as:\n",
    "\n",
    "$$ \\hat{y} = mx + b  -- (1)$$ \n",
    "\n",
    "Thus, we have two parameters $m$ and $b$. We will see how can we use gradient descent and find the optimal values for these two parameters $m$ and $b$. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import the Libraries\n",
    "\n",
    "Import the required libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "\n",
    "import random\n",
    "import math\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prepare the dataset\n",
    "\n",
    "We generate some random data points with 500 rows and 2 columns (x and y) and use for training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = np.random.randn(500, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see our data has two columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-1.16100447, -1.86143301])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First column indicates the $x$ value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.1610044684790515"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Second column indicates the $y$ value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.8614330055403117"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We know that equation of a simple linear regression is expressed as:\n",
    "\n",
    "$$ \\hat{y} = mx + b  $$\n",
    "\n",
    "Thus, we have two parameters $m$ and $b$.  We store both of these parameter $m$ and $b$ in an array called theta. First, we initialize theta with zeros as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "theta = np.zeros(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0.])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "theta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "theta[0] represents the value of $m$ and theta[1] represents the value of $b$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loss Function \n",
    "\n",
    "Mean Squared Error (MSE) of Regression is given as:\n",
    "\n",
    "$$J=\\frac{1}{N} \\sum_{i=1}^{N}(y-\\hat{y})^{2} -- (2) $$\n",
    "\n",
    "\n",
    "Where $N$ is the number of training samples, $y$ is the actual value and $\\hat{y}$ is the predicted value.\n",
    "\n",
    "The above loss function can be implemented as:\n",
    "\n",
    "We feed the data and the model parameter theta to the loss function which returns the MSE. Remember, data[,0] has $x$ value and data[,1] has $y$ value. Similarly, theta [0] has a value of $m$ and theta[1] has a value of $b$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def loss_function(data,theta):\n",
    "    \n",
    "    #get m and b\n",
    "    m = theta[0]\n",
    "    b = theta[1]\n",
    "    \n",
    "    loss = 0\n",
    "    \n",
    "    #for each iteration\n",
    "    for i in range(0, len(data)):\n",
    "        \n",
    "        #get x and y\n",
    "        x = data[i, 0]\n",
    "        y = data[i, 1]\n",
    "        \n",
    "        #predict the value of y \n",
    "        y_hat = (m*x + b)\n",
    "        \n",
    "        #compute loss as given in quation (2)\n",
    "        loss = loss + ((y - (y_hat)) ** 2)\n",
    "        \n",
    "    #mean sqaured error\n",
    "    mse = loss / float(len(data))\n",
    "        \n",
    "    return mse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we feed our randomly initialized data and model parameter theta, the loss_function returns the mean squared loss as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0217161066273013"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss_function(data, theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we need to minimize this loss. In order to minimize the loss, we need to calculate the gradient of the loss function $J$ with respect to the model parameters $m$ and $b$ and update the parameter according to the parameter update rule. So, first, we will calculate the gradient of the loss function.\n",
    "\n",
    "# Gradients of Loss Function \n",
    "\n",
    "\n",
    "Gradients of loss function $J$ with respect to parameter $m$ is given as:\n",
    "\n",
    "\n",
    "$$ \\frac{d J}{d m}=\\frac{2}{N} \\sum_{i=1}^{N}-x_{i}\\left(y_{i}-\\left(m x_{i}+b\\right)\\right) -- (3) $$\n",
    "\n",
    "\n",
    "Gradients of loss function $J$ with respect to parameter $b$ is given as:\n",
    "\n",
    "\n",
    "$$ \\frac{d J}{d b}=\\frac{2}{N} \\sum_{i=1}^{N}-\\left(y_{i}-\\left(m x_{i}+b\\right)\\right) -- (4) $$\n",
    "\n",
    "We define a function called compute_gradients which takes the data and parameter theta as an input and returns the computed gradients: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def compute_gradients(data, theta):\n",
    "\n",
    "    gradients = np.zeros(2)\n",
    "    \n",
    "    #total number of data points\n",
    "    N = float(len(data))\n",
    "    \n",
    "    m = theta[0]\n",
    "    b = theta[1]\n",
    "    \n",
    "    #for each data point\n",
    "    for i in range(len(data)):\n",
    "        x = data[i, 0]\n",
    "        y = data[i, 1]\n",
    "           \n",
    "        #gradient of loss function with respect to m as given in (3)\n",
    "        gradients[0] += - (2 / N) * x * (y - (( m* x) + b))\n",
    "        \n",
    "        #gradient of loss funcction with respect to b as given in (4)\n",
    "        gradients[1] += - (2 / N) * (y - ((theta[0] * x) + b))\n",
    "    \n",
    "    #add epsilon to avoid division by zero error\n",
    "    epsilon = 1e-6 \n",
    "    gradients = np.divide(gradients, N + epsilon)\n",
    "\n",
    "    return gradients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.00018707,  0.00020307])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "compute_gradients(data,theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, when we feed the randomly initialized data and model parameter theta, the compute_gradients function returns the gradients of $m$ i.e $\\frac{d J}{d m}$ and gradients of $b$ i.e  $\\frac{d J}{d b}$. \n",
    "\n",
    "# Update Rule\n",
    "\n",
    "After computing gradients we need to update our model paramater according to our update rule as given below:\n",
    "\n",
    "$$m=m-\\alpha \\frac{d J}{d m} -- (5) $$ \n",
    "\n",
    "$$ b=b-\\alpha \\frac{d J}{d b} --(6) $$\n",
    "\n",
    "\n",
    "Since we stored $m$ in theta[0] and $b$ in theta[1], we can write our update equation as: \n",
    "\n",
    "$$\\theta = \\theta - \\alpha \\frac{dJ}{d\\theta} -- (7) $$\n",
    "\n",
    "As we learned in the previous section, updating gradients for just one time will not lead us to the convergence i.e minimum of the cost function, so we need to compute gradients and the update the model parameter for several iterations:\n",
    "\n",
    "\n",
    "Set the number of iterations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_iterations = 50000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the learning rate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lr = 1e-2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a list called loss for storing the loss on every iterations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loss = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On each iteration, calculate gradients and update the gradients according to our paramater update rule (7):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "theta = np.zeros(2)\n",
    "\n",
    "for t in range(num_iterations):\n",
    "    \n",
    "    #compute gradients\n",
    "    gradients = compute_gradients(data, theta)\n",
    "    \n",
    "    #update parameter\n",
    "    theta = theta - (lr*gradients)\n",
    "    \n",
    "    #store the loss\n",
    "    loss.append(loss_function(data,theta))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The below plot shows how the loss(cost) decreases over the training iterations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5,1,'Gradient Descent')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8VvX5//HXlU02JCQEwgZFkCFLUNC4F9ZaraPuOuqore23y1+/be382tpaa7XurXVrVbQqtQYcgLJkKXtvwghhBcj1++M+0FtKQkhy5yR33s/H437knM85n3NfH72TN2fc55i7IyIiUlcJYRcgIiLNm4JERETqRUEiIiL1oiAREZF6UZCIiEi9KEhERKReFCQiB2Bmi83s5GD6/5nZw2HXJNJUKUik2TGzi8xsopltNbO1wfSNZmaxeD93/527X1Pf7ZhZFzNzM0uqYZ3bzGyXmW0JXnPN7B4zK6rv+8dKMKYeYdch4VGQSLNiZv8D/AW4A2gHFALXA8cCKdX0SWy0AhvG8+6eBbQBziUyzslNOUykZVOQSLNhZjnAr4Ab3f0ld9/iEVPd/RJ33xms97iZ3Wdmb5nZVuAEMzvLzKaaWbmZLTOz2/bb9mVmtsTMyszsp/stu83Mno6aH2ZmH5vZJjP7zMxKopaVmtmvzeyjYI/iXTPLDxaPC35uMrMKMxte03jdfZe7zwIuBNYB/xP1PqPMbFpQw8dm1i9q2Y/NbEXw/nPM7KSgPTE4TLcgWDbZzDoGy3qZ2Rgz2xD0uSBqe4+b2b1m9mbQb6KZdQ+W7R3TZ8GYLqxpTBKn3F0vvZrFCzgd2A0kHWS9x4HNRPZSEoA0oAToG8z3A9YAXw3W7w1UAMcBqcCdwfucHCy/DXg6mO4AlAFnBts6JZhvGywvBRYAhwGtgvnbg2VdAK+p/uj32q/9V8DEYPooYC1wNJAIXAEsDmo/HFgGtI96z+7B9A+BGcE6BvQH8oCMoM9VQFKw/fVA76j/nmXA0GD5M8BzUbU50CPsz4de4b20RyLNST6w3t13722I2jPYbmbHRa37mrt/5O5V7r7D3UvdfUYwPx14Fjg+WPd8YLS7j/PIXs3PgKpqargUeMvd3wq2NQaYRCRY9nrM3ee6+3bgBWBAA4x9JZFDXQDXAQ+4+0R33+PuTwA7gWHAHiKB0tvMkt19sbsvCPpdA/yvu8/xiM/cvQwYBSx298fcfbe7TwVeBr4e9f6vuvsnwX/7ZxpoTBInFCTSnJQB+dEnq939GHfPDZZFf56XRXc0s6PN7H0zW2dmm4mcV9l7yKl99PruvjXY3oF0Br4ehNcmM9sEjACiz1+sjpreBmQeyiCr0QHYEFXD/+xXQ0cieyHzgVuI7NmsNbPnzKx90K8jkb2lA43p6P22dwmRczOxHJPECQWJNCfjifzL+5xarLv/ba3/DrwOdHT3HOB+Iod3AFYR+SMLgJmlEznkcyDLgKfcPTfqleHut9ehploxswTgbOCDqBp+u18N6e7+LIC7/93dRxAJCAd+H9WvezVjGrvf9jLd/Ya61Cstj4JEmg133wT8EvibmZ1vZllmlmBmA4gc569JFrDB3XeY2VDgG1HLXgJGmdkIM0shcj6iut+Np4Gzzey04OR1mpmVmFlxLYawjsghs261WBczSzKzI4gchmtH5NwNwEPA9cFelplZRnAxQZaZHW5mJ5pZKrAD2M5/DtM9DPzazHoG/fqZWR4wGjgsuOAgOXgNCd67NtbUdkwSnxQk0qy4+x+A7wM/IvIHbA3wAPBj4OMaut4I/MrMtgA/J3LuYu82ZwE3EdlrWQVsBJZX8/7LiOwR/T8iwbCMyEnsg/4uufs24LfAR8EhpGHVrHqhmVUQuWDgdSKH2Qa5+8pgO5OAa4F7glrnA1cGfVOB24mcLF8NFAC3BsvuDMb9LlAOPAK0cvctwKnARUTOxawmsheTerAxBW4DngjGdMHBVpb4Y+56sJWIiNSd9khERKReFCQiIlIvChIREakXBYmIiNRLtXchjSf5+fnepUuXOvXdunUrGRkHu7I0vmjMLYPGHP/qO97Jkyevd/e2B1uvRQRJly5dmDRpUp36lpaWUlJS0rAFNXEac8ugMce/+o7XzJbUZj0d2hIRkXpRkIiISL0oSEREpF4UJCIiUi8KEhERqRcFiYiI1IuCRERE6kVBUoM3p6/i/aW7wi5DRKRJU5DU4K0Zq3hpXiU7du0JuxQRkSZLQVKDS4Z1YuuuyJ6JiIgcmIKkBsO75dEuw3hmYq3uEiAi0iIpSGpgZpzQMZkpSzcxe2V52OWIiDRJCpKDGNEhidSkBJ7WXomIyAEpSA4iI9k4u397Xpu6goqdu8MuR0SkyVGQ1MKlwzqztXIPr05dEXYpIiJNjoKkFvoX59CnfTbPTFiCu4ddjohIk6IgqQUz49Jhnfli9RamLN0YdjkiIk2KgqSWvtK/PVmpSTw9YWnYpYiINCkKklrKSE3i3IEdeHPGKjZsrQy7HBGRJkNBcgguHdaZyt1VPPep9kpERPZSkByCwwqzOLZHHk+NX8KuPVVhlyMi0iQoSA7RVcd0ZdXmHbw7a03YpYiINAkKkkN0Qq8COrVJ57GPFoVdiohIk6AgOUSJCcYVx3Rh0pKNzFi+OexyRERCpyCpg68PLiYjJZHHPtZeiYiIgqQOstOSOX9QMaM/W8W6LTvDLkdEJFQKkjq64pguVO6p4u8TdSmwiLRsCpI66tY2k5LD2/L0xCVU7talwCLScilI6uGqY7uybstO3pqhR/GKSMulIKmHkT3y6d42g4c/XKi7AotIi6UgqYeEBOOakd2YuaKc8QvKwi5HRCQUCpJ6OveoDuRnpvLAuIVhlyIiEgoFST2lJSdy5TGdGTt3HV+sLg+7HBGRRqcgaQCXDutMq+REHtReiYi0QDELEjN71MzWmtnMapabmd1tZvPNbLqZDQzaB5jZeDObFbRfGNXn28H6bmb5sar9UOWmp3DhkI68Pm0lqzZvD7scEZFGFcs9kseB02tYfgbQM3hdB9wXtG8DLnf3PkH/u8wsN1j2EXAysCQWBdfH1SO6UuXOYx8tDrsUEZFGFbMgcfdxwIYaVjkHeNIjJgC5Zlbk7nPdfV6wjZXAWqBtMD/V3RfHqub66NgmnTP7FvH3iUsp37Er7HJERBpNUojv3QFYFjW/PGjb9+0+MxsKpAALDnXjZnYdkT0dCgsLKS0trVORFRUVte47OGMPo3fu5rfPlnJG1+Q6vV9TcChjjhcac8vQ0sbcWOMNM0hqZGZFwFPAFe5+yPcgcfcHgQcBBg8e7CUlJXWqo7S0lEPp+86aCYxdtZVfX3YcKUnN81qGQx1zPNCYW4aWNubGGm+Yf+lWAB2j5ouDNswsG3gT+Glw2KvZ+Nbx3VhdvoNXpy4PuxQRkUYRZpC8DlweXL01DNjs7qvMLAV4lcj5k5dCrK9Ojj+sLX075HBf6QJ267nuItICxPLy32eB8cDhZrbczK42s+vN7PpglbeAhcB84CHgxqD9AuA44Eozmxa8BgTb/I6ZLSey9zLdzB6OVf11ZWbcdEJ3Fpdt403dzFFEWoCYnSNx94sPstyBmw7Q/jTwdDV97gbubpACY+jU3u3oWZDJve/P5+x+7UlIsLBLEhGJmeZ5NriJS0gwbjqhB3PXVPDu7DVhlyMiElMKkhgZ1a+Iznnp3Pv+fN1iXkTimoIkRpISE7jh+O7MWLGZsXPXhV2OiEjMKEhi6GsDiynKSePe9+eHXYqISMwoSGIoJSmBbx3XjU8Xb2TiQj34SkTik4Ikxi4a2on8zFT+8t68sEsREYkJBUmMpSUnckNJdz5eUKbH8YpIXFKQNIJLju5EYXYqfx4zV1dwiUjcUZA0grTkRG46oQefLN7Ah/PXh12OiEiDUpA0kguHdKR9Thp/eld7JSISXxQkjSQ1KZFvn9iTacs28f6ctWGXIyLSYBQkjejrg4vp2KYVd+pciYjEEQVJI0pOTOA7J/Zk5opy3YNLROKGgqSRnXtUB7rmZ/DnMXOpqtJeiYg0fwqSRpaUmMAtJ/fki9VbeGP6yrDLERGpNwVJCM7u154jirK545057Ny9J+xyRETqRUESgoQE4ydn9GL5xu38feLSsMsREakXBUlIjuuZzzHd8/jrv+ezZceusMsREakzBUlIzCJ7JRu2VvLQuIVhlyMiUmcKkhD1K85lVL8iHvpgEWvLd4RdjohInShIQvaDUw9n154q3WZeRJotBUnIuuRn8I2jO/Hcp8tYuK4i7HJERA6ZgqQJuPnEnqQlJfD7t78IuxQRkUOmIGkC2malckNJd96ZtUYPvxKRZkdB0kRcM7IbHXJb8evRs9mjW6eISDOiIGki0pIT+ckZvZi9qpwXJy0LuxwRkVpTkDQho/oVMahza/747hx9SVFEmg0FSRNiZvx8VG/WV1Ry7/sLwi5HRKRWFCRNTP+OuZw3sJhHP1zE0rJtYZcjInJQCpIm6EenH05igvG7tz4PuxQRkYNSkDRBhdlp3FjSnbdnrebj+evDLkdEpEYxCxIze9TM1prZzGqWm5ndbWbzzWy6mQ0M2geY2XgzmxW0XxjVp6uZTQz6PG9mKbGqP2zXHteNTm3S+dlrM6ncXRV2OSIi1YrlHsnjwOk1LD8D6Bm8rgPuC9q3AZe7e5+g/11mlhss+z3wZ3fvAWwEro5B3U1CWnIit32lNwvWbeWRDxeFXY6ISLViFiTuPg7YUMMq5wBPesQEINfMitx9rrvPC7axElgLtDUzA04EXgr6PwF8NVb1NwUn9irklN6F3P3ePFZs2h52OSIiB5QU4nt3AKK/ebc8aFu1t8HMhgIpwAIgD9jk7rv3W/+AzOw6Ins6FBYWUlpaWqciKyoq6ty3IZyaX8XYL/bwncfGcvNRaY3ynmGPOQwac8vQ0sbcWOMNM0hqZGZFwFPAFe5eFdkhqT13fxB4EGDw4MFeUlJSpzpKS0upa9+GsiZtHn98dy4U9abk8IKYv19TGHNj05hbhpY25sYab5hXba0AOkbNFwdtmFk28Cbw0+CwF0AZkcNfSfuvH++uPa4b3fIz+MXrs9ixa0/Y5YiIfEmYQfI6cHlw9dYwYLO7rwquxHqVyPmTvedDcHcH3gfOD5quAF5r7KLDkJqUyC/P6cOSsm08MFaP5RWRpiWWl/8+C4wHDjez5WZ2tZldb2bXB6u8BSwE5gMPATcG7RcAxwFXmtm04DUgWPZj4PtmNp/IOZNHYlV/UzOyZ1vO6lfEvaXzWaAHYIlIExKzcyTufvFBljtw0wHanwaerqbPQmBogxTYDP3i7N58MHcdt74yg+euHUZCwqGdNxIRiQV9s70ZKchK46dnHcEnizbw3Ke61byINA0KkmbmgsEdGd4tj/9763PWlO8IuxwREQVJc2Nm/O5rfancU8XPXzvg3WdERBqVgqQZ6pqfwS0nH8Y7s9bw9sxVB+8gIhJDCpJm6pqRXeldlM3PX5vF5u16mqKIhEdB0kwlJybw+/P6sb5iJ78ZPTvsckSkBVOQNGN9i3O4oaQ7L05eznufrwm7HBFpoRQkzdx3TupJr3ZZ/OSVGWzcWhl2OSLSAilImrnUpET+dEF/Nm6t5Oevzwq7HBFpgRQkcaBP+xy+e1JP3vhsJW9O11VcItK4FCRx4oaS7vQvzuF//zGDdVt2hl2OiLQgCpI4kZSYwJ8u6M/Wyj3c+soMIrcyExGJPQVJHOlRkMWPTjucf32+hud1Ly4RaSQKkjjzzWO7cmyPPH75xmzmr9Xt5kUk9hQkcSYhwbjzggGkJSfw3eemsnO3nqgoIrGlIIlDhdlp/OH8/sxaWc4f35kTdjkiEudqFSRm9lRt2qTpOKV3IZcN68xDHyxi3Nx1YZcjInGstnskfaJnzCwRGNTw5UhD+ulZR9CzIJPvv/AZ6yt0SbCIxEaNQWJmt5rZFqCfmZUHry3AWuC1RqlQ6iwtOZG/fuMoynfs4ocvfkZVlS4JFpGGV2OQuPv/uXsWcIe7ZwevLHfPc/dbG6lGqYde7bL52VlH8P6cddw/bkHY5YhIHKrtoa3RZpYBYGaXmtmdZtY5hnVJA7p0WGfO7t+eP74zh/ELysIuR0TiTG2D5D5gm5n1B/4HWAA8GbOqpEGZGf/3tb50yc/g5menslbPeheRBlTbINntkXtunAPc4+73AlmxK0saWmZqEvddMoiKnbu4+dmp7N5TFXZJIhInahskW8zsVuAy4E0zSwCSY1eWxMLh7bL47Vf7MnHRBu4cMzfsckQkTtQ2SC4EdgLfdPfVQDFwR8yqkpg5b1AxFw3pyN9KF+ipiiLSIGoVJEF4PAPkmNkoYIe76xxJM3XbV/rQp302tzw/jQXrdD8uEamf2n6z/QLgE+DrwAXARDM7P5aFSeykJSfywGWDSE5M4NonJ1G+Y1fYJYlIM1bbQ1s/BYa4+xXufjkwFPhZ7MqSWCtunc7fLhnI0rJtfO+5afqyoojUWW2DJMHd10bNlx1CX2mihnXL4+dn9+a9L9bq5LuI1FlSLdd728zeAZ4N5i8E3opNSdKYLhvWmVkryrnn/fkcUZTNWf2Kwi5JRJqZGoPEzHoAhe7+QzP7GjAiWDSeyMl3aebMjF99tQ/z1m7hBy9+Rpf89LBLEpFm5mCHp+4CygHc/RV3/767fx94NVhWLTN71MzWmtnMapabmd1tZvPNbLqZDYxa9raZbTKz0fv1OdHMppjZTDN7wsxqu0clNUhNSuT+SweR0yqZqx+fxMYd+rKiiNTewYKk0N1n7N8YtHU5SN/HgdNrWH4G0DN4XUfkNix73UHky4/7BF+CfAK4yN2PBJYAVxykBqmlguw0Hr1yCFt27OKuKTvZunN32CWJSDNxsCDJrWFZq5o6uvs4YEMNq5wDPOkRE4BcMysK+r4HbNlv/Tyg0t33nhUeA5xXUw1yaHq3z+aeSwaytLyK7zw7lT26kktEauFgh4Ymmdm17v5QdKOZXQNMrud7dwCWRc0vD9pWVbP+eiDJzAa7+yTgfKBjdRs3s+uI7OlQWFhIaWlpnYqsqKioc9/myIALujvPf7GWb93/Lpf2Tg27pEbR0v4/g8bcEjTWeA8WJLcAr5rZJfwnOAYDKcC5sSxsf+7uZnYR8GczSwXeBfbUsP6DwIMAgwcP9pKSkjq9b2lpKXXt23yVklVUwMMfLuLY/odx1bFdwy4o5lri/2eNOf411nhrDBJ3XwMcY2YnAEcGzW+6+78b4L1X8OU9iuKgraZ6xgMjAczsVOCwBqhDDuDWM49g2cZt/Gr0bIpyWnH6ke3CLklEmqja3mvrfXf/a/BqiBABeB24PLh6axiw2d2rO6wFgJkVBD9TgR8D9zdQLbKfxATjrguPYkDHXL7z3FQ9EEtEqhWzb6eb2bNEvm9yuJktN7Orzex6M7s+WOUtYCEwH3gIuDGq7wfAi8BJQd/TgkU/NLPPgenAGw0YanIArVISeezKIXRuk861T05i5orNYZckIk1QzL6H4e4XH2S5AzdVs2xkNe0/BH5Y/+qktnLTU3jy6qGc97ePufKxT3jp+mPokp8Rdlki0oTofllyUEU5rXjy6qPZU+Vc9uhEPapXRL5EQSK10qMgk8euGkpZRSWXP/oJm7fp1vMiEqEgkVob0DGXBy4bxIJ1FVz+2Cds0XNMRAQFiRyikT3bcu83BjJrxWaufOxT3UpFRBQkcuhO7dOOuy8+imnLNnH1E5+yvbLa74WKSAugIJE6ObNvEXde0J+JizZw3VOT2LFLYSLSUilIpM7OGdCBP5zXjw/mreeGpyezc7fCRKQlUpBIvXx9cEd+d25f3p+zjm89NVl7JiItkIJE6u0bR3fi9q/1ZezcdXzz8U/ZVqkT8CItiYJEGsRFQztx5wX9mbCwjMsf0aXBIi2JgkQazLlHFfPXiwcybdkmLn1EX1oUaSkUJNKgzupXxP2XDuLzleVc/NAEyip2hl2SiMSYgkQa3Mm9C3n4isEsXF/B+fePZ9mGbWGXJCIxpCCRmDjusLY8c83RbNhaydfu+5jZK8vDLklEYkRBIjEzqHMbXrp+OEkJxoUPjGfCQj0cSyQeKUgkpnoWZvHyDcdQmJPG5Y9+wtsza3wIpog0QwoSibn2ua146frhHNk+mxuemcITHy8OuyQRaUAKEmkUuekpPHPNME7qVcgvXp/Fba/PYveeqrDLEpEGoCCRRtMqJZEHLhvENSO68vjHi7nmyUn64qJIHFCQSKNKTDD+d1RvfnvukXwwbz1fv388KzZtD7ssEakHBYmE4pKjO/P4VUNYsWk759zzEVOXbgy7JBGpIwWJhGZkz7a8csMxtEpJ4MIHJ/DCpGVhlyQidaAgkVD1LMzitZtGMKRLa3700nR+9o+ZVO7WSXiR5kRBIqFrk5HCE1cN5VvHdeOpCUv4xkMTWFu+I+yyRKSWFCTSJCQlJnDrmUfw14uPYtbKckb99UMmL9F5E5HmQEEiTcrZ/dvz6k3HkJacyEUPjueRDxfh7mGXJSI1UJBIk9OrXTZvfHsEJYcX8OvRs7n2ycls2lYZdlkiUg0FiTRJOenJPHjZIH5xdm/Gzl3LmX/5gMlLNoRdlogcgIJEmiwz46pju/LyDceQlJjABQ9M4L7SBVRV6VCXSFOiIJEmr19xLqO/M4LTj2zH79/+gssf/YRVm/VteJGmQkEizUJ2WjL3XHwUvzu3L5OXbOS0P4/jtWkrwi5LRIhhkJjZo2a21sxmVrPczOxuM5tvZtPNbGDUsrfNbJOZjd6vz0lmNsXMppnZh2bWI1b1S9NjZnzj6E7887sj6V6QyXefm8bNz07ViXiRkMVyj+Rx4PQalp8B9Axe1wH3RS27A7jsAH3uAy5x9wHA34H/bZBKpVnpkp/Bi98azg9OPYx/zljFaXeN44N568IuS6TFilmQuPs4oKbLbM4BnvSICUCumRUFfd8Dthxos0B2MJ0DrGzAkqUZSUpM4Nsn9uQfNx1LVloylz3yCbe+Mp3N23VbepHGZrH8speZdQFGu/uRB1g2Grjd3T8M5t8Dfuzuk4L5EuAH7j4qqs9I4B/AdqAcGObu5dW893VE9nQoLCwc9Nxzz9VpDBUVFWRmZtapb3PV3MZcucd5Zd4u3lm8i5xU4/LeKQwsTDqkbTS3MTcEjTn+1Xe8J5xwwmR3H3yw9Q7tty183wPOdPeJZvZD4E7gmgOt6O4PAg8CDB482EtKSur0hqWlpdS1b3PVHMd86kkwffkmfvTSdO6euoWz+rXhtrP70DYrtVb9m+OY60tjjn+NNd4wr9paAXSMmi8O2g7IzNoC/d19YtD0PHBM7MqT5qZfcS5v3DyCH5x6GGNmreHkO8fy4qRlusWKSIyFGSSvA5cHV28NAza7+6oa1t8I5JjZYcH8KcDnsS5Smpfk4NzJW98dSc+CTH740nQueGA8n6864BFQEWkAMTu0ZWbPAiVAvpktB34BJAO4+/3AW8CZwHxgG3BVVN8PgF5AZtD3and/x8yuBV42syoiwfLNWNUvzVuPgkxe+NZwXpy8jNv/+QWj/vohVwzvwi2n9CQ7LTns8kTiSsyCxN0vPshyB26qZtnIatpfBV6tf3XSEiQkGBcO6cRpfdpxxztzeOzjRbwxfSU/PfMIzhnQHjMLu0SRuKBvtkvcy01P4bfn9uW1m46lfU4atzw/jQsfmMD05ZvCLk0kLihIpMXoV5zLqzcey/99rS8L11fwlXs+4pbnprJik+7bJVIfze3yX5F6SUgwLh7aiVH9irivdAGPfLiIf85czSmdEhk0bBdZOn8icsi0RyItUlZaMj86vRf//kEJZ/YtYvTCXZTcUcpT4xdTubsq7PJEmhUFibRoHXJb8ecLB/CL4Wl0L8jkZ6/N4qQ7S3lp8nJ271GgiNSGgkQE6JqTyPPXDePxq4aQ2yqFH7z4GafeNY43PlupB2mJHISCRCRgZpQcXsDr3z6W+y8dRFKCcfOzUznz7g8YM3uNviEvUg0Fich+zIzTj2zHP797HH+5aAA7du3h2icnccZfPuCNz1ayR3soIl+iIBGpRmKCcc6ADvzr+8fzx6/3Z9eeKm5+dion3zmWFz5dppPyIgEFichBJCUmcP6gYt793vHcd8lA0lMS+dHL0ym5430e/2gR2yv3hF2iSKgUJCK1lJhgnNG3iNE3j+Dxq4bQoXUrbntjNsNvf48/vP0FqzfvCLtEkVDoC4kih2jvSfmSwwv4dPEGHv5gIfeNXcCD4xYyql8RV4/oRt/inLDLFGk0ChKRehjSpQ1DurRhadk2Hvt4ES98uox/TFvJ0C5t+OaIrpx8RAFJidrxl/imIBFpAJ3y0vnF2X343imH8cKny3jso8Vc//Rk2mWnceGQjlw0tCNFOa3CLlMkJhQkIg0oOy2Za0Z248pjuvDeF2v5+8Sl3P3vefz13/M46YhCLjm6E8f1bEtCgm5hL/FDQSISA0mJCZzWpx2n9WnH0rJtPPvpUl74dBljZq+huHUrLh7aia8N7KC9FIkLChKRGOuUl86PT+/F904+jHdnr+aZCUu54505/PHdOYzokc95A4s5rU87WqUkhl2qSJ0oSEQaSUpSAqP6tWdUv/YsKdvKy1NW8PLk5dzy/DQyU5M4q28R5w0qZkiX1np6ozQrChKREHTOy+D7pxzGLSf1ZOKiDbw8ZTlvTF/J85OW0alNOqP6FTGqX3uOKMpSqEiTpyARCVFCgjG8ex7Du+fxy6/04e2Zq/nHtBU8MG4hfytdQPe2GYzq156z+xfRoyAr7HJFDkhBItJEZKQmcd6gYs4bVExZxU7+OXM1o6ev5O5/z+Mv782jV7sszupbxBl9i+hRkBl2uSL7KEhEmqC8zFQuHdaZS4d1Zm35Dt6asYrR01fxpzFz+dOYuXTLz+CU3oWc0ruQozq1JlGXE0uIFCQiTVxBdhpXHtuVK4/tyqrN2/nX7DW8O3sNj3y4iAfGLSQ/M4UTexVwSu92jOyZT1qyrv6SxqUgEWlGinJacdnwLlw2vAvlO3ZROmcdY2av4Z8zVvPCpOWkJScwvFsexx/WluMPL6BLXrpO1kvMKUghQ167AAAPAUlEQVREmqnstGS+0r89X+nfnsrdVUxcVMa/Zq9h7Nx1vD9nHbwxm05t0jnusHyOP6yA4d3zyEzVr7w0PH2qROJASlICI3u2ZWTPtgAsKdvKuLnrGDt3Ha9MWcHTE5aSnGgM6tyaY7vnM7x7Hrv1pEdpIAoSkTjUOS+Dy4ZncNnwLuzcvYfJizcydt46xs5Zx5/GzIUxkJIIRy+ayLBukcuP+3XI0Z2KpU4UJCJxLjUpkWN65HNMj3xuPeMINmytZOLCMl76YDrLyndwxztzAMhISWRI1zYc3TWPQZ1b0684RyfupVYUJCItTJuMFM7oW0SrsjmUlBzP+oqdTFhYxoSFZYxfUEbpnHUAJCcafdrnMKhz632vwuy0kKuXpkhBItLC5Wem7rsHGEBZxU6mLN3E5CUbmbJkI09PWMIjHy4CoENuKwZ1bs1RnXLpV5xD76Ic3WxSFCQi8mV5man7vuwIULm7itmryvcFy8RFZbz+2UoAEgwOK8yib4cc+hXn0Lc4l17tsnRIrIWJWZCY2aPAKGCtux95gOUG/AU4E9gGXOnuU4JlbwPDgA/dfVRUnw+AvTccKgA+cfevxmoMIhK5ImxAx1wGdMzl6hFdcXfWlO9k+vJNzFixmenLN/PeF2t5cfJyAJISbF+49G6fTa92WfRql01OenLII5FYieUeyePAPcCT1Sw/A+gZvI4G7gt+AtwBpAPfiu7g7iP3TpvZy8BrDVqxiByUmdEuJ412Oe04tU87ANydlZt3MGP5JqYv38yMFZt5Z/Zqnp+0bF+/9jlp9CoKgqUomyPaZdE1P0NXisWBmAWJu48zsy41rHIO8KS7OzDBzHLNrMjdV7n7e2ZWUl1HM8sGTgSuasiaRaRuzIwOua3okNuK048sAti35/L56nK+WLWFL1aXM2f1FsbNXbfvOywpiQn0KMjc9+reNvKzS346qUk6PNZchHmOpAOwLGp+edC2qhZ9vwq85+7lsShMROrvP3suaZxweMG+9srdVSxYV8EX+wJmC5OXbNx33gUi5146tUmne9tMuhdk0qNtJt0LMuiWn0luerJu+9LENNeT7RcDD9e0gpldB1wHUFhYSGlpaZ3eqKKios59myuNuWUIe8ytgeHpMLwb0C2BnXvSWb21ilUVzqqtVazcupM5y7czdu5adlf9p1+rJChIT6Aw3ShIT6BtulGYnkBBupGbaiTUEDJhj7mxNdZ4wwySFUDHqPnioK1GZpYPDAXOrWk9d38QeBBg8ODBXlJSUqciS0tLqWvf5kpjbhmay5j3VDnLN25j/toKFq3fytIN21hcto2lZVuZsnj7l271kpqUQKc26XTOS6djm3Q65LaiuHUrOuSm0z43jemfftwsxtxQGuv/cZhB8jrwbTN7jshJ9s3uXpvDWucDo919R0yrE5EmITHB6JyXQee8jP9atntPFSs37WDJhq0sKdvGkrLIz6UbtvHR/DK279rzpfVTEqB4Sum+gGmf04oOrVvRPji/U5CdqnMzdRDLy3+fBUqAfDNbDvwCSAZw9/uBt4hc+jufyOW/V0X1/QDoBWQGfa9293eCxRcBt8eqbhFpPpISE+iUl06nvHRG9vzyMndn07ZdrNi0nRWbtrNy03YmzJhLYlYWKzZuZ8yqLayv2Plf22yTkUJBViqF2WkUZkd+FmSnURi0tctJIy8jRVebRYnlVVsXH2S5AzdVs2zkgdqDZSX1q0xEWgIzo3VGCq0zUjiyQw4AXXctoaRk0L51duzaw6rNO1ixMRI0a8p3sGbLDtaU72Rt+Q6+WF3Oui072f9GyQkWuSNAYXYa+Zkp5GWmkpeZQn5G5GdeZip5GSnkZ6bSJiOFlKT4Dp3merJdRKTe0pIT6ZqfQdf8/z5stteeKqesYidrynf+V9CsLt/B+opK5qzewvqtlVRGXxUQJTstifwgbNpkRIKmTXoKuenJ5KankNsq+T/T6cnktEomuRnt8ShIRERqkJhgFASHt/qSU+167k7Fzt2UVVRStnUn6ysqI9MVOynbWsn6ip2UVVSyaP1WPl28kU3bKv9rTydaVmoSOelBwLTaGzr/mc5OSya7VRLZaclkfWk6qdEPuylIREQagJmRFfxR71LDHs5eVVXOlp272bxtFxu3VbJp+y42batk07Zdkdf2vdORZSs2bWfTtko2b99VYwABpKckkp2WTELVTv5+5NZa1VMfChIRkRAkJBg5rSKHsTrlpde6X1WVs2XHbsp37KJ8x67I9PZd+9r2zpfv2MXCZatIT439VWgKEhGRZiQhwchJT67VTTBLSzdSkBX7Z8g0n7M5IiLSJClIRESkXhQkIiJSLwoSERGpFwWJiIjUi4JERETqRUEiIiL1oiAREZF6schNeOObma0DltSxez6wvgHLaQ405pZBY45/9R1vZ3dve7CVWkSQ1IeZTXL3wWHX0Zg05pZBY45/jTVeHdoSEZF6UZCIiEi9KEgO7sGwCwiBxtwyaMzxr1HGq3MkIiJSL9ojERGRelGQiIhIvShIamBmp5vZHDObb2Y/CbueQ2Fmj5rZWjObGdXWxszGmNm84GfroN3M7O5gnNPNbGBUnyuC9eeZ2RVR7YPMbEbQ524zs8Yd4X8zs45m9r6ZzTazWWb23aA9bsdtZmlm9omZfRaM+ZdBe1czmxjU+byZpQTtqcH8/GB5l6ht3Rq0zzGz06Lam9zvgZklmtlUMxsdzMf1eAHMbHHw2ZtmZpOCtqbx2XZ3vQ7wAhKBBUA3IAX4DOgddl2HUP9xwEBgZlTbH4CfBNM/AX4fTJ8J/BMwYBgwMWhvAywMfrYOplsHyz4J1rWg7xlNYMxFwMBgOguYC/SO53EHdWQG08nAxKC+F4CLgvb7gRuC6RuB+4Ppi4Dng+newWc8FegafPYTm+rvAfB94O/A6GA+rscb1LwYyN+vrUl8trVHUr2hwHx3X+julcBzwDkh11Rr7j4O2LBf8znAE8H0E8BXo9qf9IgJQK6ZFQGnAWPcfYO7bwTGAKcHy7LdfYJHPoFPRm0rNO6+yt2nBNNbgM+BDsTxuIPaK4LZ5ODlwInAS0H7/mPe+9/iJeCk4F+e5wDPuftOd18EzCfyO9Dkfg/MrBg4C3g4mDfieLwH0SQ+2wqS6nUAlkXNLw/amrNCd18VTK8GCoPp6sZaU/vyA7Q3GcEhjKOI/As9rscdHOaZBqwl8odhAbDJ3XcHq0TXuW9swfLNQB6H/t8iTHcBPwKqgvk84nu8eznwrplNNrPrgrYm8dlOqu2KEl/c3c0sLq/9NrNM4GXgFncvjz7UG4/jdvc9wAAzywVeBXqFXFLMmNkoYK27TzazkrDraWQj3H2FmRUAY8zsi+iFYX62tUdSvRVAx6j54qCtOVsT7MIS/FwbtFc31praiw/QHjozSyYSIs+4+ytBc9yPG8DdNwHvA8OJHMrY+w/F6Dr3jS1YngOUcej/LcJyLPAVM1tM5LDTicBfiN/x7uPuK4Kfa4n8g2EoTeWzHfYJpKb6IrK3tpDIibi9J936hF3XIY6hC18+2X4HXz4x94dg+iy+fGLuk6C9DbCIyEm51sF0m2DZ/ifmzmwC4zUix3bv2q89bscNtAVyg+lWwAfAKOBFvnzy+cZg+ia+fPL5hWC6D18++byQyInnJvt7AJTwn5PtcT1eIAPIipr+GDi9qXy2Q/8wNOUXkSsf5hI55vzTsOs5xNqfBVYBu4gc77yayLHh94B5wL+iPkAG3BuMcwYwOGo73yRyInI+cFVU+2BgZtDnHoK7JIQ85hFEjiNPB6YFrzPjedxAP2BqMOaZwM+D9m7BH4b5wR/Z1KA9LZifHyzvFrWtnwbjmkPUFTtN9feALwdJXI83GN9nwWvW3rqaymdbt0gREZF60TkSERGpFwWJiIjUi4JERETqRUEiIiL1oiAREZF6UZBIXDCzvOCuqNPMbLWZrYiaT6nlNh4zs8MPss5NZnZJA9X8oZkNMLOEhr7LrJl908zaRc0fdGwidaXLfyXumNltQIW7/3G/diPyma86YMdGZmYfAt8mcu3+enfPPcT+iR65PUq123b3afWvVKRm2iORuGZmPSzyfJJniHyRq8jMHjSzSRZ5fsfPo9bdu4eQZGabzOx2izznY3xwfyPM7DdmdkvU+rdb5Hkgc8zsmKA9w8xeDt73peC9BtRQ5u1AVrD39GSwjSuC7U4zs78Fey1767rLzKYDQ83sl2b2qZnNNLP7g+dQXAgMAJ7fu0e2d2zBti8Nnjsx08x+F7TVNOaLgnU/M7P3G/h/kcQBBYm0BL2AP7t7b4/cr+gn7j4Y6A+cYma9D9AnBxjr7v2B8US+DXwg5u5DgR8Ce0PpZmC1u/cGfk3kLsQ1+Qmwxd0HuPvlZnYkcC5wjLsPIHLbjoui6hrn7v3cfTzwF3cfAvQNlp3u7s8T+Vb/hcE2K/cVG7kF+2+AE4K6jg1uhFjTmH8BnBS0n3uQsUgLpCCRlmCBu0+Kmr/YzKYAU4AjiDzkaH/b3f2fwfRkIvctO5BXDrDOCCI3FMTd997S4lCcDAwBJgW3hz8e6B4sqyRyw769TjKzT4jcOuN4IveQqsnRwL/dfb277yLycKjjgmXVjfkj4Ekzuwb9zZAD0G3kpSXYunfCzHoC3wWGuvsmM3uayP2Y9lcZNb2H6n9XdtZinUNlwKPu/rMvNUbuXrvd995MySydyD2RBnrk9uK/4cBjqa3qxnwtkQAaBUwxs6M88lAkEUD/upCWJxvYApTbf54Y19A+Ai4AMLO+HHiPZx8PHsgUdRv0fwEXmFl+0J5nZp0O0LUVkYc7rTezLOC8qGVbiDxueH8TgROCbe49ZDb2IOPp5pGn7P0M2EjTedCTNBHaI5GWZgowG/gCWELkj35D+yuRQ0Gzg/eaTeTJfDV5BJhuZpOC8yS/BP5lZglE7uB8PbAyuoO7l5nZE8H2VxEJib0eAx42s+1Enluxt89yM/sZUEpkz+cNd38zKsQO5M9m1jVY/113n3mQsUgLo8t/RRpY8Ec5yd13BIfS3gV6+n8eBSsSV7RHItLwMoH3gkAx4FsKEYln2iMREZF60cl2ERGpFwWJiIjUi4JERETqRUEiIiL1oiAREZF6+f/ErF33DFw4VgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss)\n",
    "plt.grid()\n",
    "plt.xlabel('Training Iterations')\n",
    "plt.ylabel('Cost')\n",
    "plt.title('Gradient Descent')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, we learned that gradient descent can be used for finding the optimal parameters of the model with which we can minimize the loss. In the next section, we will learn several variants of gradient descent algorithm."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}