09-11-neural-network-intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-07T02:19:02.272939Z",
     "start_time": "2019-04-07T02:19:02.268259Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "\n",
    "# 第七章 神经网络与深度学习\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "![image.png](./images/author.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](./images/neural.png)\n",
    "\n",
    "Sung Kim <hunkim+ml@gmail.com> HKUST\n",
    "\n",
    "- Code: https://github.com/hunkim/PyTorchZeroToAll \n",
    "- Slides: http://bit.ly/PyTorchZeroAll \n",
    "- Videos: https://www.bilibili.com/video/av15823922/\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](./images/neural2.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src = './img/neuralnetwork/synapse.jpg' width = 500 align = 'right'>\n",
    "\n",
    "**The Neuron: A Biological Information Processor**\n",
    "\n",
    "- dentrites -   the receivers\n",
    "- soma -   neuron cell body (sums input signals)\n",
    "- axon  -   the transmitter\n",
    "- synapse 突触 -   point of transmission\n",
    "\n",
    "Neuron activates after a certain threshold is met.\n",
    "\n",
    "Learning occurs via electro-chemical changes in effectiveness of synaptic junction.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src = ./images/neural3.png  width=450 align = 'right'>\n",
    "\n",
    "**An Artificial Neuron: The Perceptron simulated on hardware or by software**. Learning occurs via changes in value of the connection weights. \n",
    "\n",
    "- input connections -   the receivers\n",
    "- node simulates neuron body\n",
    "- output connection -   the transmitter\n",
    "- **activation function** employs a threshold or bias\n",
    "- connection weights act as synaptic junctions (突触)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src = ./images/neural3.png  width=450 align = 'right'>\n",
    "\n",
    "Neural Networks consist of the following components\n",
    "- An **input layer**, **x**\n",
    "- An arbitrary amount of **hidden layers**\n",
    "- An **output layer**, **ŷ**\n",
    "- A set of **weights** and **biases** between each layer, **W and b**\n",
    "- A choice of **activation function** for each hidden layer, **σ**. \n",
    "    - e.g., Sigmoid activation function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Each iteration of the training process consists of the following steps:\n",
    "\n",
    "1. Calculating the predicted output **ŷ**, known as `feedforward`\n",
    "1. Updating the weights and biases, known as `backpropagation`\n",
    "\n",
    "![image.png](./images/neural4.png)\n",
    "\n",
    "**activation function** for each hidden layer, **σ**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](./images/neural5.png)\n",
    "\n",
    "https://blog.ttro.com/artificial-intelligence-will-shape-e-learning-for-good/ \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-07T05:41:09.971531Z",
     "start_time": "2019-04-07T05:41:09.967172Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](./images/neural6.png)\n",
    "\n",
    "http://playground.tensorflow.org/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](./images/neural7.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Batch, Iteration, & Epoch\n",
    "\n",
    "Batch Size is the total number of training examples present in a single batch.\n",
    "\n",
    "![image.png](./images/neural8.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Note: The number of batches is equal to number of iterations for one epoch. Batch size and number of batches (iterations) are two different things.\n",
    "\n",
    "\n",
    "Let’s say we have 2000 training examples that we are going to use .\n",
    "\n",
    "We can divide the dataset of 2000 examples into batches of 500 then it will take 4 iterations to complete 1 epoch.\n",
    "\n",
    "Where Batch Size is 500 and Iterations is 4, for 1 complete epoch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Gradient Descent\n",
    "\n",
    "![](images/gradient.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](./images/neural9.png)\n",
    "\n",
    "Let's represent parameters as $\\Theta$, learning rate as $\\alpha$, and gradient as $\\bigtriangledown J(\\Theta)$, "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Mannual Gradient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-13T03:38:13.578136Z",
     "start_time": "2020-08-13T03:38:13.345432Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD7CAYAAAB+B7/XAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXiTVaIG8DdNum90SRcKgjKKiAiVRQraitKyFCgUBGQpy7CLjOjAICCMeMdxkBFlkDt4h1KWghZKS0UoICiKgEBVEK2IYBFamqT7nvXcPyoZK0saaPKl6ft7nnnG5CT2fT4PL+lJco5MCCFAREROw0XqAERE1LRY7ERETobFTkTkZFjsREROhsVORORkWOxERE6GxU5E5GQUUgcAgNLSaphM1n+cPijIB8XFVTZIdHccNRfguNmYyzrMZR1ny+XiIkNAgPctxx2i2E0mcUfFfv25jshRcwGOm425rMNc1mlJubgUQ0TkZFjsREROhsVORORkGlXshw8fRmJiIgYNGoT/+Z//uWE8NzcXiYmJGDBgAJYsWQKDwdDkQYmIqHEsFvuVK1ewfPlyrFu3DllZWfj+++9x5MiRBo9ZsGABli1bhv3790MIgbS0NJsFJiJq7ipOHMOlhS/hi+GjcGnhS6g4caxJ//0Wi/3gwYMYPHgwwsLC4OrqitWrV6Nr167m8fz8fNTV1aFbt24AgMTERGRnZzdpSCIiZ1Fx4hhUm1NgKCkGhIChpBiqzSlNWu4Wi/3y5cswGo2YNWsWEhISsG3bNvj7+5vH1Wo1lEql+bZSqYRKpWqygEREzqRoVzqETtfgPqHToWhXepP9DIufYzcajTh9+jS2bNkCLy8vzJ49GxkZGUhMTAQAmEwmyGSy/wYUosHtxggK8rEy9n8plb53/FxbctRcgONmYy7rMJd1HCGXrrS0/pX6TRhKS5oso8ViDw4ORlRUFAIDAwEA/fv3x9mzZ83FHhYWBo1GY358UVERQkJCrApRXFx1Rx/SVyp9odFUWv08W3PUXIDjZmMu6zCXdaTOJYRAxbGj0Hzw/i0fowgIbHRGFxfZbV8QW1yK6devH44ePYqKigoYjUZ8/vnn6Ny5s3k8IiIC7u7uyMnJAQDs3r0b0dHRjQpHROTs9EUa5L/9T6g2boB7RASCR42GzM2twWNkbm4IThzZZD/T4iv2rl27Ytq0aRg3bhz0ej369u2LkSNHYvr06Zg3bx66dOmCVatWYenSpaiqqkLnzp2RlJTUZAGJiJojYTKh7JNDKNq1E4AMIeMnwj+mH2QuLlC0aoWiXekwlJZAERCI4MSR8Ovdp8l+tswRDrPmUoz9OGo25rIOc1nH3rm0BQVQbUpG3cWf4PVwF4ROnATXoOAmy2VpKcYhNgEjInIGwmBAyf59KPlwN2Tu7gj743T49u5j9QdK7haLnYioCdRdzoMqJRnaK7/Ap0dPhDw7AYrffDTcnljsRER3waTTofjD3Sjdvw9yX1+Ez3kevo92lzQTi52I6A7V/Hgeqk0boVcVwu/xaCifGQO5960PwLAXFjsRkZVMdbXQpO9A+SeHoQgORsSLC+D9UGfLT7QTFjsRkRWqvz0L1ZYUGEpL0ap/HIJHjISLu7vUsRpgsRMRNYKxqgrqD7ah8vgxuIW3RttFS+DZ4Q9Sx7opFjsR0W0IIVCVcwrq1K0w1lQjcMgwBMYPhYurq9TRbonFTkR0C4ayUqhSt6D666/g3q492ry4AO5t20odyyIWOxHR7wghUHH0M2jS3ocwGBA8ajQCYgdAJpdLHa1RWOxERL+h06ih3pyCmtzv4flAR4ROmgK30DCpY1mFxU5EhF837Tp0EEUZ6ZC5uCBkQhL8o5+EzKVRR0M7FBY7EbV42oJ8qFKSUXfpIry7PIKQiZPgGhgkdaw7xmInohZLGAwo2fcRSj76EDIPD4RNmwHfx6LsvmlXU2OxE1GLVJf3MwpTkqG7egW+vR6Dcux4KPz8pI7VJFjsRNSimLRaFGdlovRANuT+/mg990/w6RYpdawm1ahinzhxIkpKSqBQ1D98xYoV6Nq1q3n85ZdfRk5ODjw9PQEAc+fORWxsrA3iEhHduZrzP9Rv2qVWwT86BsGjxkDu5SV1rCZnsdiFEMjLy8Mnn3xiLvbfO3fuHLZu3Wr1IdZERPZgqKmBassmlB/5BK5KJdq8tBBenR6SOpbNWCz2S5cuAQCmTp2KsrIyjB49GhMmTDCP19bWoqCgAIsXL4ZKpUJsbCzmzp0Ll2b4ESEicj5VZ79BXuoW6EpKEBA7AEHDEx1u066mZrHYKyoqEBUVhVdeeQV6vR5JSUm499570bdvXwBAUVERevfujeXLl8PX1xczZ87Ezp07MXr06EaHuN3ZfZYolb53/FxbctRcgONmYy7rMNft6Ssq8PN/NkJz5DN43dMWD/7lz/Dt+IDUsW5gi+tl9WHWKSkp5lfoN3Pw4EFkZmbi3XffbfS/k4dZ24+jZmMu6zDXrQkhUHnqS2i2pcJYW4PAwUPQcdKzKC6rkzTXzUh2mPXp06eh1+sRFRUFoP6i/Xat/fz588jLy8OAAQNuOk5EZC/60lKot25C9Zlv4N7+XrSZPBXubdr+uhOj4xW7rVhcCK+srMTKlSuh1WpRVVWFjIyMBp94EULg9ddfR3l5OfR6PT744AN+IoaI7EoIgbLPPsXlZYtRk/s9lKPH4p7Fr8C9jePvxGgLFl9a9+vXD2fOnMHw4cNhMpkwbtw4REZGIiEhAe+99x4efPBBzJgxA88++ywMBgPi4uIwZMgQe2QnIoJOrYZq80bU/pALzwc7ITRpCtxa+Cf0rF5jtwWusduPo2ZjLusw16+bdn18AEWZuyCTyxH8zBj4PxFz0+0AnO163fUaOxGRo9HmX63ftOvnS/Du2g0hEybBNSBA6lgOg8VORM2GMBhQ/NGHKNm7B3JPL4TNmAXfno81+027mhqLnYiahdpLl6BK2QBdQT58H4tCyNhxkPs6xmfmHQ2LnYgcmkmrRXHmLpR+fACKVgFoPe8F+DzSTepYDo3FTkQOq+aHXKg2JUOv0cA/ph+CR42G/NfNBunWWOxE5HCMNTUo2vkByj87AteQULRZsAheHR+UOlazwWInIodS9c3XUG3dBGN5OQIGDELQsOFOv2lXU2OxE5FDMFRUQLN9KypPnYRbRBtEzP0TPNrfK3WsZonFTkSSEkKg8svjUL+/DabaWgQljEDgoHjIuOfUHeOVIyLJ6EuKod66GdVnz8Djvg4InTwV7q0jpI7V7LHYicjuhMmE8s8+RdHONAiTCcqx49Dqqf6Q8YCeJsFiJyK70qkKodq0EbU/nodXp4cQkjQZbsqWvWlXU2OxE5FdCKMRpQf2ozgrAzKFAqGTp8Kv7xPcDsAGWOxEZHPaK7+gMCUZ2st58O4WidAJSVC04qZdtsJiJyKbMen1KPkoCyX79kLu5Y3wWXPg070nX6XbGIudiGyi9uJPUKUkQ3etAH5RfaEc8yzkPnd+cD01XqOKfeLEiSgpKTGfZbpixQp07drVPH7s2DH8/e9/h1arxaBBgzB//nzbpCUih2fSalGUsRNlhz6GIiAAEX96Ed5dHpE6VotisdiFEMjLy8Mnn3xy00Oq6+rqsHjxYmzZsgXh4eGYOXMmjhw5gpiYGJsEJiLHVfbNGeT9ax0MRUXw7/c0lCNHwcWDm3bZm8Viv3TpEgBg6tSpKCsrw+jRozFhwgTz+NmzZ9GuXTu0bVt/aOzQoUORnZ3NYidqQYzV1dDseB8VRz+Ha2go2ix8GV4PdJQ6VotlsdgrKioQFRWFV155BXq9HklJSbj33nvRt29fAIBarYZSqTQ/PiQkBCqVyqoQtzu7zxKl0jE32nfUXIDjZmMu6zhKruLjX+Ln9e9BX16BiJEj0HbMM5A74KZdjnK9fs8WuSwWe2RkJCIjI823R40ahSNHjpiL3WQyNXiHWwhh9TvePMzafhw1G3NZxxFyGcrLod6+FVWnT8G97T24Z+4LaNujy6+5dJJm+z1HuF43I9lh1qdPn4Zer0dUVBSA+uL+7Vp7WFgYNBqN+bZGo0FICL9FRuSshBCoPH4M6ve3Qei0CBoxEoEDBnHTLgdicWOGyspKrFy5ElqtFlVVVcjIyEBsbKx5vGvXrvj5559x+fJlGI1G7NmzB9HR0TYNTUTS0BcXI/+dt1CY/H9wCw9Hu+UrEBQ/lKXuYCz+1+jXrx/OnDmD4cOHw2QyYdy4cYiMjERCQgLee+89hIaG4o033sDzzz8PrVaLmJgYDBw40B7ZichOhMmE8k8PQ5O+E4CA8tnxaNXvaW7a5aBkQgjrF7ebGNfY7cdRszGXdeyZS1d4rX7Trgs/wuuhzghNmgzXYOVNH8vrZR3J1tiJqGUSBgNKD2SjOCsTMjc3hE75I/z6PM7tAJoBFjsR3aDul8tQpSRD+8tl+HTvgZBxE6DwbyV1LGokFjsRmZn0OpR8mIWS7L2Q+/ggfPZz8O3eU+pYZCUWOxEBAGovXEDhpg3QFxbCr8/jUI4ey027mikWO1ELZ6qrRdGunSj75DAUgYGImP9neHd+WOpYdBdY7EQtWPW5b6HanAJDaQla9XsawYmj4OLhIXUsukssdqIWyFhVBU3adlQc+wKuYWFou3AxPO+/X+pY1ERY7EQtTGXOKahTt8BYVYXAwUMQOHQYXFzdpI5FTYjFTtRCGMrKoN62BVVf5cD9nnaIeOEleNzTTupYZAMsdiInJ4RAxbGj0HywHUKnQ/DIZxAQNxAyuVzqaGQjLHYiJ6Yv0kC1OQU1338Hz/sfQOikKXALC5c6FtkYi53ICQmTCWWHD6EoYycAGULGT4R/TD9u2tVCsNiJnIy2oACqTcmou/gTvB7ugtCJk+AaFCx1LLIjFjuRkxAGA0qy96JkTxZk7u4I++N0+Pbuw027WiAWO5ETqMvLg2rTBmivXIFPj14IeXY8FP7+UsciiVhV7P/4xz9QWlqKN954o8H9GRkZ+Oc//4mgoCAAwJNPPon58+c3XUoiuimTTofirEyUHsiG3NcXrZ97Hj6R3aWORRJrdLEfP34cGRkZePLJJ28YO3fuHBYtWoQhQ4Y0ZTYiuo2aH89DtSkZepUKfo9HQ/nMGMi9vaWORQ6gUcVeVlaG1atXY9asWfjhhx9uGP/222+Rl5eH9evXo2PHjnjllVfgz18DiWzCWFuLi/9+H4X7sqEIDkbEiwvg/VBnqWORA2nUZ5+WLVuG+fPnw8/P76bjSqUSc+bMQVZWFsLDw7FixYomDUlE9aq/PYvLy5egMHs/WvWPQ/tX/8ZSpxtYPPN0x44d+Omnn/Dyyy9j165dOHny5A1r7L9VXl6O2NhYnDx5ssnDErVU+opK/LxhIzSfHoFnmzb4w/Nz4PdgR6ljkYOyuBSzd+9eaDQaJCQkoLy8HDU1NXj99dexePFiAEBlZSXS09MxefJkAPVfX5Zb+VVlHmZtP46ajbluTgiBqtOnoN62BcaaGgQOGYbA+KHwax3I62UFZ8t114dZb9y40fzP11+xXy91APDy8sJ//vMfREZGomvXrti6dStiY2OtDkpEDRnKSqHauhnV33wN93bt0ebFhXBv21bqWNQM3PHn2JcsWYKnnnoKTz/9NN5++2389a9/RV1dHdq3b4+VK1c2ZUaiFkUIgYqjn0GT9j6EwYDgUaMREDuAm3ZRo1lcY7cHLsXYj6NmY656Oo0aqk0bUftDLjwf6Fi/aVdomOS5Gou5rCPZUgwR2Z4wmVB26CCKMtIhc3FByIQk+Ec/yU276I6w2Ikkps3Ph2rTBtRdugTvLo8gZOIkuAYGSR2LmjEWO5FEhMGAkn0foXhPFlw8PRE2fSZ8e/Xmpl1011jsRBKo+/kSClOSocu/Ct9ej0H57HgofG/+BUAia7HYiezIpNWiOCsDpQf2Q+7vj9Zz/wSfbpFSxyInw2InspOaH3Kh2pwCvVoF/+gYBI8aA7mXl9SxyAmx2IlszFhTg6L0NJQf+RSuSiXavLQQXp0ekjoWOTEWO5ENVZ35Buqtm2AoK0NA7AAEDU+Ei7u71LHIybHYiWzAUFkBzfZtqDx5Am6tI9B29lx43tdB6ljUQrDYiZqQEAKVJ7+EZnsqjLU1CBo2HIGDh0Cm4B81sh/ONqImoi8pgXrrJlSfPQOPe+9Dm8lT4R7RRupY1AKx2InukjCZUP75Zyja+QGE0Qjl6LFo1T+O2wGQZFjsRHdBp1JBtXkjas//AM8HOyE0aQrcQkKkjkUtHIud6A4IoxGlHx9A8e4MyORyhCRNhv8TMdwOgBwCi53IStqrV1CYkgxt3s/w7toNIRMmwTUgQOpYRGYsdqJGMun1KNm7ByV790Du5YXwGbPh07MXX6WTw2l0sf/jH/9AaWnpDQdZFxQUYMGCBSguLsa9996LVatWwdvbu8mDEkmp9tJFqFKSoSvIh+9jUQgZOw5yX1+pYxHdVKOK/fjx48jIyMCTTz55w9irr76KcePGIT4+Hu+++y7WrVuHBQsWNHVOIruqOHEMRbvS8WNpMWTuHhB1dVAEBKL1vBfg80g3qeMR3ZbFz2OVlZVh9erVmDVr1g1jer0ep06dwoABAwAAiYmJyM7ObvqURHZUceIYVJtTYCgpBgQg6uoAFxcEDk1gqVOzYLHYly1bhvnz58PP78a9oktLS+Hj4wPFr9+qUyqVUKlUTZ+SyI6K0ndA6HQN7zSZULInS5pARFa67VLMjh07EB4ejqioKOzateuGcSHEDW8c3ckbSbc7lNUSpdIx1zkdNRfguNkcIVfxlydhKC296ZihtMQhMl7nSFl+i7msY4tcty32vXv3QqPRICEhAeXl5aipqcHrr7+OxYsXAwACAwNRWVkJo9EIuVwOjUaDkDv4ckZxcRVMJmH185zt5HF7cNRsUucyVFRAs30rKk+dBORywGi84TGKgECHuXZSX69bYS7r3GkuFxfZbV8Q37bYN27caP7nXbt24eTJk+ZSBwBXV1f06NEDe/fuxdChQ5GZmYno6GirQxJJRQiByhPHoX4/FUKrRdDwRCgCAqFO3dxgOUbm5obgxJESJiVqvDv6HPuSJUvw1FNP4emnn8by5cuxaNEi/O///i/Cw8Px1ltvNXVGIpvQlxRDvWUTqr89C4/7OiB08lS4t44AAMjkLijalQ5DaQkUAYEIThwJv959JE5M1DgyIYT1ayBNjEsx9uOo2eyZS5hMKD/yKYrS0yBMJgQnjkKrp/rfdNMuXi/rMJd1JFmKIXI2usLC+k27fjwPr04PISRpMtyU3LSLnAuLnVoEYTSi9MB+FGdlQKZQIHTyVPj1fYLbAZBTYrGT09Ne+QWFGzdA+8tleEc+itDxE6FoxU27yHmx2MlpmfR6lHyUhZJ9eyH38kb4rDnw6d6Tr9LJ6bHYySnVXvypftOuawXwi+oL5ZhnIfe58y/CETUnLHZyKqa6OhRlpqPs0MdQBAQg4k8vwrvLI1LHIrIrFjs5jervzkG1JQWGoiL493saypGj4OLhKXUsIrtjsVOzZ6yuhibtfVR88TlcQ8PQZuHL8Hqgo9SxiCTDYqdmrfKrHKhTN8NYWYmAQfEIGpYAF1c3qWMRSYrFTs2SobwM6m1bUZVzGu5t70HEvPnwaNde6lhEDoHFTs2KEAKVx49B/f42CJ0WQSNGInDAIMgUnMpE1/FPAzUb+uIiqDanoOa7c/Do8AeETZ4Kt/DWUscicjgsdnJ4wmRC+aeHoUnfCUBA+ex4tOr39E037SIiFjs5OF3hNRSmJKPupwvw6vwwQidOgmuwUupYRA6NxU4OSRgMKD2QjeKsTMjc3BE6ZRr8+vTldgBEjcBiJ4dT98tlqFKSof3lMny690DIuAlQ+LeSOhZRs9GoYn/nnXewf/9+yGQyjBo1ClOmTGkwvnbtWqSnp8PPzw8AMHr0aIwfP77p05JTM+l1KPkwCyXZeyH38UH47Ofg272n1LGImh2LxX7y5EmcOHECWVlZMBgMGDx4MGJiYnDfffeZH3Pu3Dm89dZbiIyMtGlYcl4VuT/g8tv/gr6wEH59Hody9Fhu2kV0hywWe69evbB582YoFAqoVCoYjUZ4eXk1eMy5c+ewfv165Ofno2fPnvjLX/4Cd3d3m4Um52Gqq0XRrp0o++QwFIGBiJj/Z3h3fljqWETNWqPPPF2zZg2Sk5MxcOBA/P3vfze/iVVdXY0XXngBixYtQrt27bBo0SJERERg/vz5Ng1OzV/pV1/j4rp/Q1tUjPD4QWg3YRzknty0i+huWXWYdW1tLWbNmoXBgwdjzJgxN33M999/j8WLFyMzM7PRIXiYtf04QjZjVRU0H2xHxfEv4BYWjtBJU3FPn0clz3UzjnC9boa5rONsue76MOuLFy9Cp9OhU6dO8PT0RFxcHM6fP28eLygowLFjxzBq1CgA9V/5VvDr3XQLladPQZ26BcbqKgTGD0XgkKHctIuoiVn86t7Vq1exdOlS6HQ66HQ6HDp0CN27dzePe3h44M0338SVK1cghEBqaipiY2NtGpqaH0NZGQrW/QvX/v0uFAEBuGfpcgSPGMlSJ7IBiy+tY2JicPbsWQwfPhxyuRxxcXGIj4/H9OnTMW/ePHTp0gUrVqzA7Nmzodfr8eijj97wcUhquYQQqPjiKDRp2yF0OgSPfAYBcQMhk8uljkbktKxaY7cVrrHbjz2z6TUaqLakoOb77+B5/wMInTQFbmHhkueyBnNZh7msI9kaO5G1hMmEssOHULRrByBzQcj4ifCP6cdNu4jshMVOTUpbUADVpmTUXfwJXg93QejEyXANCpI6FlGLwmKnJiEMBpRk70XJnizI3N0R9sfp8O3dh5t2EUmAxU53rS4vD4UpG6C7egU+PXoh5NnxUPj7Sx2LqMVisdMdM+l0KM7KROmBbMh9fdH6uefhE9nd8hOJyKZY7HRHan48D9WmZOhVKvg9Hg3lM2Mg9/aWOhYRgcVOVjLW1qIofQfKPz0M12AlIl5cAO+HOksdi4h+g8VOjVZ19gzUWzfBUFqKVv3j6r85yl08iRwOi50sMlZWQv3BNlSeOA631q3RdtESeHb4g9SxiOgWWOx0S0IIVJ0+BfW2LTDW1CBwyDAExg+Fi6ur1NGI6DZY7HRThrJSqLZuRvU3X8O9XXu0eXEh3Nu2lToWETUCi50aEEKg4vPPoNnxPoTBgOBRoxEQO4CbdhE1Iyx2MtNp1FBt2ojaH3Lh+UDH+k27QsOkjkVEVmKxU/2mXR8fRFFmOmQuLgiZOAn+T8Rw0y6iZorF3sJp8/Oh2rQBdZcuwfuRrgiZMAmugYFSxyKiu9CoYn/nnXewf/9+yGQyjBo16oaDNHJzc7FkyRJUV1ejR48eePXVV3k8noMTBgNK9n2E4j1ZcPH0RNj0mfDt1ZubdhE5AYu/a588eRInTpxAVlYW0tPTsWXLFly6dKnBYxYsWIBly5Zh//79EEIgLS3NZoHp7tX9fAmXX/srindnwLd7D7R/7XX4PRbFUidyEhaLvVevXti8eTMUCgWKi4thNBrh5eVlHs/Pz0ddXR26desGAEhMTER2drbtEtMdM2m1+HnjJvzy+mswVleh9dw/IXzGbCh8/aSORkRNqFHrJa6urlizZg2Sk5MxcOBAhIaGmsfUajWUSqX5tlKphEqlavqkdFdqfsiFatNG6DVq+Ec/ieBRoyH/zV/QROQ8Gr0QPm/ePEyfPh2zZs1CWloaxowZAwAwmUwNfoUXQlj9K/3tzu6zRKn0vePn2pKj5DJUVyNv0xao9h+ER1gYHnjtr2j1SBepY92Uo1yz32Mu6zCXdWyRy2KxX7x4ETqdDp06dYKnpyfi4uJw/vx583hYWBg0Go35dlFREUJCQqwKwcOsbaPqzDf1m3aVlSEgbiCCEkagVZtgh8j2e45yzX6PuazDXNax1WHWFtfYr169iqVLl0Kn00Gn0+HQoUPo3v2/hylERETA3d0dOTk5AIDdu3cjOjra6qDUdAyVFbj23r9R8K+34eLljbYvvwLl6LHciZGohbD4ij0mJgZnz57F8OHDIZfLERcXh/j4eEyfPh3z5s1Dly5dsGrVKixduhRVVVXo3LkzkpKS7JGdfkcIgcqTX0KzPRXG2hoEDRuOwMFDIONHT4laFJkQwvo1kCbGpZi7py8pgXrrJlSfPQOPe+9D6OSpcI9o4xDZGoO5rMNc1nG2XJaWYvhSrpkTJhPKP/8MRTs/gDAaoRw9Fq36x3E7AKIWjMXejOlUKqg2b0Tt+R/g+WAnhCZNgZuVb1wTkfNhsTdDwmhE6ccHUJy5CzKFAqFJU+D3RDS/OUpEAFjszY726hUUpiRDm/czvLt2q9+0KyBA6lhE5EBY7M2ESa9Hyd49KNm7B3IvL4TPmA2fnr34Kp2IbsBibwZqL12EKiUZuoJ8+D4WhZCx4yD3dcxv0RGR9FjsDsyk1aIocxfKPj4ARasAtJ73Anwe6SZ1LCJycCx2B1WT+339pl1FGvg/+RSCRz4Duaen1LGIqBlgsTsYY001NDs+QMXnn8E1JBRtFiyCV8cHpY5FRM0Ii92BVH39FVRbN8NYUY6AgYMRNGw4XNzcpI5FRM0Mi90BGCoqoNm+FZWnTsItog0inv8TPNrfK3UsImqmWOwSEkKg8sRxqN9PhdBqETQ8EYEDB3PTLiK6K2wQiehLiqHesgnV356Fx30d6jftah0hdSwicgIsdjsTJhPKj3wKzc40QJigHDsOrZ7qz027iKjJsNjtSFdYCNWmZNRe+BFenTojNGkyXH9zXiwRUVNgsduBMBpRemA/irMy6jftmjwVfn2f4HYARGQTjSr2tWvXYt++fQDqT1RauHDhDePp6enw8/MDAIwePRrjx49v4qjNk/bKLyjcuAHaXy7DO/JRhI6fCEUrbtpFRLZjsdiPHTuGo0ePIiMjAzKZDNOmTcPBgwcRGxtrfsy5c+fw1ltvITIy0qZhmxOTXo+Sj7JQsm8v5F7eCJ81Bz7de/JVOhHZnMViVyqVWLRoEdx+/aJMhw4dUFBQ0OAx586dw/r165Gfn4+ePXviL3/5C9xb8MHJFbk/4GM/SowAAAxmSURBVJd33oXuWgH8ovpCOeZZyH1ufYwVEVFTsvhRjPvvvx/dutVvPJWXl4d9+/YhJibGPF5dXY1OnTphwYIFyMjIQEVFBdatW2e7xA7MVFcH9fZUfPvyUpi0WkS88CLC/jidpU5EdtXow6wvXLiAmTNn4vnnn8eIESNu+bjvv/8eixcvRmZmZpOFbA7KvjmDn979N7RqNcIGD0S7iROg8OKmXURkf4168zQnJwfz5s3D4sWLER8f32CsoKAAx44dw6hRowDUf5tSYeU3J4uLq2AyNervlwYc4eRxY3U1NGnvo+KLz+EaGoY2C19Gu7496nNVO8+p6LbGXNZhLus4Wy4XFxmCgm69EmCxga9du4bnnnsOq1evRlRU1A3jHh4eePPNN/HYY4+hTZs2SE1NbfDGqjOr/CoH6tTNMFZWImBQPIKGJcDFlZt2EZG0LBb7hg0boNVq8cYbb5jvGzt2LA4fPox58+ahS5cuWLFiBWbPng29Xo9HH30UU6ZMsWloqRnKy6DethVVOafh3vYeRMybD4927aWORUQEwIo1dltqLksxQghUHPsCmg+2Q+i0CBo2HAFxA2/YtMtRf+0DHDcbc1mHuazjbLnueimG6umLi6DanIKa787Bo8MfEDZ5KtzCW0sdi4joBix2C4TJhLJPD6MofQcAQDluAlo9+RQ37SIih8Vivw1d4TUUpiSj7qcL8Or8MEInToJrMDftIiLHxmK/CWEwoPRANoqzMiFzc0folGnw69OX2wEQUbPAYv+dul8uQ5WSDO0vl+HTvQdCxk2Awr+V1LGIiBqNxf4rk16Hkg+zUJK9F3IfH4TPfg6+3XtKHYuIyGosdgC1F35EYUoy9KpC+PV9AsrRYyH39pY6FhHRHWnRxW6qq4UmfSfKPzkERVAQIub/Gd6dH5Y6FhHRXWmxxV597luoNqfAUFqCVk/HInjESLh4eEgdi4jorrW4YjdWVUHzwXZUHP8CbmHhaLtwMTzvv1/qWERETaZFFXvl6VNQp26BsaYagfFDEThkKDftIiKn0yKK3VBWBnXqFlR9nQP3e9ohYv5L8LinndSxiIhswqmLXQiBii+OQpO2HUKnQ/DIZ+o37ZLLpY5GRGQzTlvseo2mftOu3O/gef8DCJ00FW5hYVLHIiKyOacrdmEyoezwIRTt2gHIXBAyfiL8Y/px0y4iajGcqti1BQVQbUpG3cWf4PVwF4ROnAzXoCCpYxER2VWjin3t2rXYt28fACAmJgYLFy5sMJ6bm4slS5aguroaPXr0wKuvvmr1uafWqDhxDEW70vFjaQkUAYEIShgBQ2kJSvZkQebujrA/zoBv7yhu2kVELZLF9Yljx47h6NGjyMjIQGZmJr777jscPHiwwWMWLFiAZcuWYf/+/RBCIC0tzWaBK04cq/9iUUkxIAQMJcVQpWxAceYueHd7FO1f+zv8ovqw1ImoxbJY7EqlEosWLYKbmxtcXV3RoUMHFBQUmMfz8/NRV1eHbt26AQASExORnZ1ts8BFu9IhdLqGdwoBuY8vWs+aA4Wfn81+NhFRc2BxveT+33wrMy8vD/v27cP27dvN96nVaiiV/z18QqlUQqVSWRXidmf3/d6PpSU3vd9YXQWl0teqn2tLjpTl9xw1G3NZh7ms05JyNXoh/MKFC5g5cyYWLlyI9u3bm+83mUwNlj2EEFYvg1hzmLUiILB+GeYm9zvKYbWOenAu4LjZmMs6zGUdZ8tl6TDrRn0GMCcnB5MnT8ZLL72EESNGNBgLCwuDRqMx3y4qKkJISIjVQRsrOHEkZG4NtwGQubkhOHGkzX4mEVFzYrHYr127hueeew6rVq1CfHz8DeMRERFwd3dHTk4OAGD37t2Ijo5u+qS/8uvdB6FJk6EIDAJkMigCgxCaNBl+vfvY7GcSETUnFpdiNmzYAK1WizfeeMN839ixY3H48GHMmzcPXbp0wapVq7B06VJUVVWhc+fOSEpKsmlov9594Ne7j8P+ekVEJCWZEKJxi9s2ZM0a+285arE7ai7AcbMxl3WYyzrOlqtJ1tiJiKj5YLETETkZFjsRkZNxiE3AXFzu/Ov/d/NcW3LUXIDjZmMu6zCXdZwpl6XnOMSbp0RE1HS4FENE5GRY7ERETobFTkTkZFjsREROhsVORORkWOxERE6GxU5E5GRY7ERETobFTkTkZBy62KuqqjBkyBBcvXr1hrHc3FwkJiZiwIABWLJkCQwGAwCgoKAA48ePx8CBAzF79mxUV1fbNdfHH3+MhIQEDBs2DHPmzEF5eTkAICMjA48//jgSEhKQkJCA1atX2zXX2rVr0a9fP/PPT01NBXDr62iPXLm5ueY8CQkJeOKJJzBkyBAAtr9ea9euRXx8POLj47Fy5cobxqWaX5ZySTW/LOWSan7dLpeU8wsA3nnnHQwePBjx8fHYuHHjDeM2nWPCQX3zzTdiyJAhonPnzuLKlSs3jMfHx4uvv/5aCCHEyy+/LFJTU4UQQsyYMUPs2bNHCCHE2rVrxcqVK+2Wq7KyUvTt21cUFhYKIYR4++23xWuvvSaEEGLFihXiww8/bNIsjc0lhBAzZ84UX3311Q333+o62ivXdTU1NSI+Pl6cOnVKCGHb6/XFF1+IMWPGCK1WK3Q6nUhKShIHDhxo8Bgp5pelXFLNr8ZcLynmV2NyXWfP+SWEEF9++aUYO3as0Ov1ora2VvTr109cvHixwWNsOccc9hV7Wloali9fftPzU/Pz81FXV4du3boBABITE5GdnQ29Xo9Tp05hwIABDe63Vy69Xo/ly5cjNDQUANCxY0dcu3YNAPDtt98iIyMDQ4cOxZ///GfzKy175AKAc+fOYf369Rg6dChWrFgBrVZ7y+toz1zXrV+/Hj179kSPHj0A2PZ6KZVKLFq0CG5ubnB1dUWHDh1QUFBgHpdqflnKJdX8spQLkGZ+NSbXdfacXwDQq1cvbN68GQqFAsXFxTAajfDy8jKP23qOOWyx/+1vfzP/R/g9tVoNpVJpvq1UKqFSqVBaWgofHx8oFIoG99srV0BAAGJjYwEAdXV1eO+999C/f39zljlz5iArKwvh4eFYsWKF3XJVV1ejU6dOWLBgATIyMlBRUYF169bd8jraK9d1lZWVSEtLw9y5cxtksdX1uv/++81/oPLy8rBv3z7ExMSYx6WaX5ZySTW/LOWSan5ZynWdvefXda6urlizZg3i4+MRFRVl/gsZsP0cc9hivx2TyQSZ7L/bVgohIJPJzP//W7+/bQ+VlZWYMWMGHnzwQYwYMQIA8O6776J79+6QyWSYNm0aPv/8c7vl8fb2xv/93/+hQ4cOUCgUmDp1Ko4cOXLL62hvWVlZ6N+/P4KCgsz32eN6XbhwAVOnTsXChQvRvn178/1Sz69b5bpOqvl1q1xSzy9L10uq+QUA8+bNw/Hjx3Ht2jWkpaWZ77f1HGuWxR4WFgaNRmO+XVRUhJCQEAQGBqKyshJGoxEAoNFoLC4BNDW1Wo1x48ahY8eO+Nvf/gag/g9iSkqK+TFCCMjlcrtlKigowM6dOxv8fIVCccvraG8ff/wxBg8ebL5tj+uVk5ODyZMn46WXXjKX43VSzq/b5QKkm1+3yyXl/LJ0vQBp5tfFixeRm5sLAPD09ERcXBzOnz9vHrf1HGuWxR4REQF3d3fk5OQAAHbv3o3o6Gi4urqiR48e2Lt3LwAgMzMT0dHRdstlNBoxa9YsDBo0CEuWLDH/Tevl5YX//Oc/OHPmDABg69at5l+p7cHDwwNvvvkmrly5AiEEUlNTERsbe8vraE9CCHz33XeIjIw032fr63Xt2jU899xzWLVqFeLj428Yl2p+Wcol1fyylEuq+WUpFyDN/AKAq1evYunSpdDpdNDpdDh06BC6d+9uHrf5HLP67VY769evn/nTFNOmTRNnz54VQgiRm5srRo4cKQYMGCBefPFFodVqhRBCXL16VUyYMEEMGjRITJ06VZSVldkt14EDB0THjh3FsGHDzP9bvHixEEKIU6dOieHDh4uBAweKWbNmiYqKCrvlEkKI7OxsER8fL+Li4sSiRYvM1+tW19FeuYqKikSfPn1ueLwtr9drr70munXr1uC/07Zt2ySfX5ZySTW/GnO9pJhfjcklxfy6bs2aNWLQoEFiyJAhYs2aNUII+3UYT1AiInIyzXIphoiIbo3FTkTkZFjsREROhsVORORkWOxERE6GxU5E5GRY7ERETobFTkTkZP4fKeIiHEH7IRcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "sns.set()\n",
    "\n",
    "x_data = [1.0, 2.0, 3.0]\n",
    "y_data = [2.0, 4.0, 6.0]\n",
    "\n",
    "plt.plot(x_data, y_data, 'r-o');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-13T03:38:14.210339Z",
     "start_time": "2020-08-13T03:38:14.207126Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# our model for the forward pass\n",
    "def forward(x):\n",
    "    return x * w\n",
    "\n",
    "# Loss function\n",
    "def loss(y_pred, y_val):\n",
    "    return (y_pred - y_val) ** 2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-13T03:38:14.860718Z",
     "start_time": "2020-08-13T03:38:14.855815Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# List of weights/Mean square Error (Mse) for each input\n",
    "w_list = []\n",
    "mse_list = []\n",
    "\n",
    "for w in np.arange(0.0, 4.1, 0.1):\n",
    "    # Print the weights and initialize the lost\n",
    "    #print(\"w=\", w)\n",
    "    l_sum = 0\n",
    "    for x_val, y_val in zip(x_data, y_data):\n",
    "        # For each input and output, calculate y_hat\n",
    "        # Compute the total loss and add to the total error\n",
    "        y_pred = forward(x_val)\n",
    "        l = loss(y_pred, y_val)\n",
    "        l_sum += l\n",
    "        #print(\"\\t\", x_val, y_val, y_pred_val, l)\n",
    "    # Now compute the Mean squared error (mse) of each\n",
    "    # Aggregate the weight/mse from this run\n",
    "    #print(\"MSE=\", l_sum / 3)\n",
    "    w_list.append(w)\n",
    "    mse_list.append(l_sum / 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-13T03:38:15.764701Z",
     "start_time": "2020-08-13T03:38:15.525889Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEJCAYAAACKWmBmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deVhTV94H8G9Cwr5D2PdFQFkVd4WisiggrbZWa6vWse10bJ1xZuxYnakd39r2rba2vrZTp+3beTvahWoRpUpVtG64orIooiJbACHsBAiE5L5/WDNSNBAkuTfh93mePg/JyTVfDpRf7rnnnsNjGIYBIYQQ8hB8tgMQQgjhNioUhBBC1KJCQQghRC0qFIQQQtSiQkEIIUQtKhSEEELUokJBCCFELQHbAbShubkDSqXmt4c4OFiisVGqhUSPhqu5AO5mo1yaoVyaMbRcfD4PdnYWD203yEKhVDJDKhT3juUiruYCuJuNcmmGcmlmJOWioSdCCCFqUaEghBCiFhUKQgghalGhIIQQohYVCkIIIWpRoSCEEKIWFYpfiCVSvPj2ETS0dLEdhRBCNNKrUGLjvy7g5JVqrfz7VCh+YWEqRF1TB47n17AdhRBCNHLlZgPK77TDzEQ7t8ZRofiFnZUJxoU441RBLXoVSrbjEELIoJ3Ir4G9tQmigpy08u9TobhP4kRvtHb0oKC0ke0ohBAyKA0tXbha1oRpYa4w4vO08h5UKO4THeIMW0tjnKDhJ0KInjhRUAsAmB7uprX3oEJxHyMjPqaFu6LwdiOa2mRsxyGEELUUSiVOFdQg1M8BDjamWnsfKhS/Mj3cDQwDnPylShNCCFcVljahRdqDmAjtnU0AVCj6EdmaYYyPHU4W1HB2dUhCCAHuXsS2tjBGRICDVt+HCsUDxES6o6mtG0VldFGbEMJNze3dyC9twLQwVwiMtPunXOv7UUilUixcuBCffvopSktL8cEHH6ja6urqEBERgR07dvQ5JiMjA++//z4cHO5WycceewyrV6/WdlSVqEBHWJkLcfxKDcL9HXX2voQQMlinCmrAMEBMhKvW30urhSI/Px9//etfUV5eDgCIjY1FbGwsAEAikWDRokV4/fXX+x1XVFSEtWvXIiUlRZvxHkpgxMfUMFccOl+FFmk3bC1NWMlBCCEPomQYnCyoRYi3HZzszLX+flo9X0lPT8eGDRvg5NT/JpD33nsPCxcuhI+PT7+2wsJCZGRkIDU1FX/+85/R2tqqzZgPFBPhBiXD4HQhXdQmhHDLtfImNLTKEBup3YvY92i1UGzatAnR0dH9ni8vL8f58+exZMmSBx4nEonwu9/9Dvv27YOrqys2btyozZgP5GJvjiBPW5zIr4GSoYvahBDuOHGlBpZmQkQFinTyfqzsmf3dd9/hmWeegbGx8QPbP/74Y9XXK1asQHx8vEb/voOD5ZCziURWqq9Tpvvh/a8v4U5LNyJG6eYH8jD35+IarmajXJqhXJphK1dzuwyXbzYgdbof3Fxt+rVrIxcrhSInJwdffPHFA9va29uxZ88eLFu2DADAMAyMjIw0+vcbG6VDmtoqEllBImlXPR7lZgULUwH2nbgFNzvt3cyiaS4u4Wo2yqUZyqUZNnMdPFsBhZJBdKBjvwxDzcXn89R+wNb59NimpibIZDJ4eno+sN3c3Byff/458vPzAQA7d+7U+IxiuAgFRpg8xgWXbkjQ3tnDSgZCCLmHYRicyK9BoIcN3BwtdPa+Oi8UYrEYLi4u/Z5fv349cnJyYGRkhA8//BBvvvkmZs+ejatXr2LNmjW6jqkSE+mGXgWD3KI7rGUghBAAKKlsQV1zl9bvxP41nQw9HT16VPV1eHg40tPT+71m06ZNqq+jo6ORkZGhi2gD8hBZwt/NGifya5Aw3hM8nnZWZySEkIGcyK+BmYkA0cHaWU78YejO7EGIiXBDbWMnbop1P02XEEIAQNolx8USCSaPcYaJULPrto+KCsUgTAhxhqmxEY5foeXHCSHsOFN0B70Kpc6HnQAqFINiYnz3ovaF6/WQdsnZjkMIGWEYhsGxy9XwdbWGl7Pup+VSoRikuLHu6FUocbKAzioIIbpVXNGMO02dmDHWnZX3p0IxSB4iS4zytMWxS9V0pzYhRKeOXaqGpZkQE0J0exH7HioUGpgx1h0NrTIU3ablxwkhutHUdvdO7OnhrhAKdHsR+x4qFBoYO0oEGwtjHL1UzXYUQsgIcfxKDRiGwWNR7Aw7AVQoNCIw4iMmwg2FpY2QtHSxHYcQYuB6FUocz69BmL8DRLZmrOWgQqGh2Eg38Hg8/HyZzioIIdp16YYEbR09mDHWg9UcVCg0ZG9tiqhAR5wsqIW8V8F2HEKIATuaJ4bI1hShfvas5qBCMQQzxrpD2iXH+eJ6tqMQQgyUuF6KG+JWxEV5gM/y0kFUKIYg2NsOrg7mdFGbEKI1Ry9XQyjgY1q49vfEHggViiHg8XiIi3JHWW0bymrb2I5DCDEwXd29OFN0BxNCnGBpJmQ7DhWKoZoS6goToRGO0VkFIWSY5RbdQbdcwfpF7HuoUAyRuakAk8c441xxHa3/RAgZNgzD4OglMXxdreDras12HABUKB5J3FgPyHuVOFVQy3YUQoiBuF7ZgtrGTsRFceNsAqBC8Ug8nSwR6GGDny/T+k+EkOFx7JIYFqYC1tZ1ehAqFI8obqw76lu6cLWsie0ohBA919zejUs3GjA93A3GOt6cSB2tFwqpVIqUlBSIxWIAwOuvv46EhASkpaUhLS0Nhw8f7ndMcXEx5s2bh8TERKxfvx69vb3ajjlk0UFOsDYX0kVtQsgjO36l+pd1nXS/OZE6Wi0U+fn5WLRoEcrLy1XPFRUVYefOncjMzERmZibi4+P7HbdmzRq88cYb+Omnn8AwzAP32OYKgREfMZFuyL/VgAZa/4kQMkT31nUK9XOAk50523H60GqhSE9Px4YNG+DkdHesraurCzU1NVi3bh1SU1Oxbds2KJXKPsdUV1dDJpMhMjISADBv3jxkZ2drM+YjeyzSHTwej27AI4QMWV6JBK3SHtY2J1JHq4Vi06ZNiI6OVj1uaGjApEmT8PbbbyM9PR0XL17E7t27+xxTX18PkUikeiwSiVBXV6fNmI/M3toU44JEOJ5fA1kPd4fJCCHcxDAMDl2ogrOdGcL8HdiO049Al2/m6emJjz/+WPX4ueeew969e7FgwQLVc0qlErz71jVhGKbP48FwcLAcckaRaGj70S6ID8KF/zmJ/LJmpEzzG/L7P8xQc+kCV7NRLs1QLs0MZ67isiaU1bbht0+Ewdnp0e6d0EZ/6bRQlJSUoLy8HImJiQDuFgGBoG8EFxcXSCQS1eOGhgbV0NVgNTZKoVRqPl1VJLKCRNKu8XEA4GAhhJ+bNTJ+voXxoxyHdRGvR8mlbVzNRrk0Q7k0M9y50o+UwNxEgHBfu0f6d4eai8/nqf2ArdPpsQzD4O2330Zrayvkcjm+++67fhez3d3dYWJigry8PABAZmYmYmJidBlzyBLGe6K+uQsFt2irVELI4DS0diGvpB6xkW4wNdbpZ/dB02mhCA4OxosvvohFixYhOTkZISEhSElJAQC88MILKCwsBABs2bIF77zzDpKSktDZ2YklS5boMuaQjR0lgp2VCQ5dqGQ7CiFET+TkicEDDzPHcedO7F/TSfk6evSo6uvFixdj8eLF/V7z2Wefqb4ODg7ud5FbHwiM+Jg1zgPf/1yKyrp2eDlzc2yVEMINXd29OJFfi+hgEeytTdmO81B0Z/Ywi4l0g7GQj8MXq9iOQgjhuNOFtejq7kX8eE+2o6hFhWKYWZgKMTXMFeeu1aFV2s12HEIIRymVDI5cFMPfzRr+bjZsx1GLCoUWxEd7olfB4NhlugGPEPJg+aUNqG/p4vzZBECFQitc7M0R4e+AY5erIe9VsB2HEMJBhy9Uwd7aBOOCRAO/mGVUKLQkfrwn2jvlOHuV23eVE0J0r7KuHdcrWzBznAeM+Nz/M8z9hHoqxNsOHiJLHL5YBYb2qiCE3OfwhSqYCI0QG8GtVWIfhgqFlvB4PMSP94BY0oHiima24xBCOKJV2o1zxXWYFuYKc1Mh23EGhQqFFk0a7QxrcyEOXaCpsoSQu45eqoZCwWBWNHdvsPs1KhRaJBQY4bEodxSUNuJOUyfbcQghLJP3KnDscjUiAhzhbM+tPSfUoUKhZXFjPSAw4tENeIQQnLlaB2mXXC+mxN6PCoWW2VgYY+JoZ5wurIW0S852HEIIS5QMg8MXquAhskSwly3bcTRChUIHkiZ4oUeuxNE8MdtRCCEsKShtRHVDB2ZP9NJ4jx22UaHQAXeRJSIDHHEkT4zuHroBj5CR6MDZCjhYm2J8iGb763ABFQodmT3JC9IuOU4W1LAdhRCiYzeqWnBL3IrECZ4QGOnfn139S6ynAj1sEehhg5/OV6JXoWQ7DiFEhw6erYClmRDT9eQGu1+jQqFDsyd5o7GtGxeK69mOQgjREbFEivzSRswa5wEToRHbcYaECoUOhfs7wN3RAgfOVdCyHoSMEAfPVsJYyMcMDu9gNxCtFwqpVIqUlBSIxXdn/Hz33XdISUlBamoqXn/9dfT09PQ7JiMjA9OmTUNaWhrS0tKwdetWbcfUCT6Ph9mTvFAt6UBBKe2rTYiha2yV4XxxHWIj3GFpph/LdTyIVgtFfn4+Fi1ahPLycgBAWVkZvvjiC3z77bfYt28flEolvv76637HFRUVYe3atcjMzERmZiZWr16tzZg6NSHEGQ7WJjh4toLtKIQQLfvpQiUAIHGCft1g92taLRTp6enYsGEDnJzuTgczNjbGhg0bYGlpCR6Ph1GjRqGmpv8soMLCQmRkZCA1NRV//vOf0draqs2YOiUw4iNhghduiFtxS2w43xchpC9plxwn8mswabQzp/fDHgytFopNmzYhOjpa9djd3R1Tp04FADQ1NWHXrl2YOXNmv+NEIhF+97vfYd++fXB1dcXGjRu1GVPnYsLdYGkmxAE6qyDEYOXkidEjVyJpkjfbUR6ZgI03raurw4oVKzB//nxMnDixX/vHH3+s+nrFihWIj4/X6N93cLAccjaRyGrIx2pi7nQ/fH2oBJ0KBt4u1gO+Xle5hoKr2SiXZiiXZtTlknX34uglMSaOcUFkiIsOU2mnv3ReKEpLS7FixQo899xzWL58eb/29vZ27NmzB8uWLQMAMAwDIyPNppQ1NkqhVGo+q0gksoJE0q7xcUMxKcQJu4/dxDcHi/GblNGcyaUprmajXJqhXJoZKNfhi1Vo75RjZpS7TvMPtb/4fJ7aD9g6nR4rlUrxm9/8Br///e8fWCQAwNzcHJ9//jny8/MBADt37tT4jEIfWJoJERPhhrPX6tDYKmM7DiFkmPQqlDh0vhKjPGwQ4GHDdpxhodNCsXv3bjQ0NODLL79UTX396KOPAADr169HTk4OjIyM8OGHH+LNN9/E7NmzcfXqVaxZs0aXMXUmcbwXANDGRoQYkPPFdWhs68ZsA7g2cQ+PMcA7v/Rh6Omez7Ou4WJJPbb8bupD51lz9fQb4G42yqUZyqWZh+VSMgw2fHEe4AEbl0/Q+SqxBjH0RPqbPfHuEuQ5tAQ5IXqv4NbdpcTnTPTWu6XE1aFCwTJ3kSWiAh1x+EIVOmW9bMchhAwRwzDYd7oMjjb6uZS4OlQoOGDuVF90dvciJ4+uVRCirwpvN6L8TjtSpvjo5VLi6hjWd6OnvF2sEBngiEMXqtDVTWcVhOgbhmGQeaocjjammBKq2/smdIEKBUfMneaDDlkvXasgRA8VlTWhrLYNyZO9De5sAqBCwRk+LtYI93fAT+cr6ayCED3CMAz2nSqDg7UJpoa5sh1HK6hQcEjaNF90yO7e+k8I0Q9Xy5tQWtOG5MmGd23iHsP8rvSUr6s1wvwc8NP5Ksh66KyCEK67ezZRDntrE0wLN8yzCYAKBefMneYDaZccxy5Vsx2FEDKAaxXNuFXdiuRJhnlt4h7D/c70lL+bDUL97HHwXCWdVRDCYXdnOpXBzsoE08Ld2I6jVVQoOChtqu/ds4rLdFZBCFcVVzTjlrgVyZO9IRQY9p9Sw/7u9JS/uw3G+Noj+1wlunsUbMchhPzKvZlOdlYmmG7gZxMAFQrOSpvqi/ZOOqsghIsKSxtwQ9yKOZMM/2wCoELBWQEeNhjtY4fscxV0rYIQjvnmUAlsLY0RE2G4M53uR4WCw+ZO9UVbpxzZZ2hvbUK4oqSyGUWljZg9yRtCgWa7b+orKhQcNsrTFiHedvjh2E30yOlaBSFccG+mU2yE4V+buIcKBcelTfNFc3s3jtJ9FYSw7lp5E65XtuDJGYEwFo6MswmACgXnjfK0xbhgJ/x4phydMjnbcQgZsRiGwZ7jpXCwNkHSZB+24+iUVguFVCpFSkoKxOK7axfl5uYiNTUVCQkJ2Lp16wOPqampweLFi5GUlISXX34ZHR0d2oyoF5bMGY0OWS+yz1eyHYWQEevSDQnKatuRNs1vRJ1NAIMsFA0NDcjJyQEAbN68GUuXLsX169fVHpOfn49FixahvLwcACCTybBu3Tp88sknOHDgAIqKinD8+PF+x/3973/HM888g+zsbISGhuKTTz7R8FsyPH7uNpg42hmHLlShVdrNdhxCRhyFUokfTtyGm6OFQe43MZBBFYq1a9eiqqoKZ86cwcmTJ5GWloa33npL7THp6enYsGEDnJzubglYUFAAb29veHp6QiAQIDU1FdnZ2X2OkcvluHDhAhITEwEA8+bN6/eakerx6b5QKBjszy1nOwohI05u4R3UNnZiXowf+HzD2Qt7sAZVKFpaWrBs2TKcOHECKSkpmDdvHrq6utQes2nTJkRHR6se19fXQyQSqR47OTmhrq6uzzHNzc2wtLSEQCAAAIhEon6vGamc7cwRE+GG41dqUN+ivu8JIcNH3qvA3lNl8HOzRlSgI9txWCEYzIvkcjnkcjlOnjyJd999F11dXejs7NTojZRKJXi8/1RihmH6PH7Yc79+PBgODpYaH3OPSGQ15GO1SSSywrK5oThddAfZ56vwp8Xj2I6kwuU+4yLKpRm2c+09fgvN7d1Y82w0nJysVc+znethtJFrUIVi5syZmDx5MkJCQhAaGoqUlBSkpKRo9EYuLi6QSCSqxxKJRDUsdY+9vT3a29uhUChgZGT0wNcMRmOjFEolo/FxIpEVJJJ2jY/TtvtzxUd74MCZCjwW4QovZ/Z/UfWhz7iEcmmG7Vydsl58d/gGQn3t4WJjosrCdq6HGWouPp+n9gP2oIaeVq1ahaysLHz11VcAgC1btmDlypUaBYmIiEBZWRkqKiqgUCiQlZWFmJiYPq8RCoWIjo7GgQMHAAB79+7t95qRLmmiF8xMBPjhxG22oxBi8H46XwlplxzzY/3ZjsKqQc96unr1Kng8HjZv3ox33nlnwFlPv2ZiYoJ3330Xr776KubMmQM/Pz8kJSUBANavX6+aVbVhwwakp6djzpw5uHjxIv7whz9o+C0ZNgtTIeZM9kZBaSNuVLWwHYcQg9Xa0YNDF6owPtgJ3i7sn72ziccwzIBjNCtWrMC0adMQFBSEd955B8uWLcMPP/yAnTt36iKjxgx56AkAuuUKrN1xBiJbM7y+eOyQruNoKxtXUC7NUK7+dh2+gWOXqvHWCxPhYm/OmVzqsDr0NJRZT0R7TIRGSJvqi1viVuSXNrIdhxCDI2npws+XqzE9wrVfkRiJBlUo7p/1NGXKlCHNeiLDa1q4K5zszLDneOmQzp4IIQ+392QZ+Hwe5k71ZTsKJwyqUNyb9WRnZ4fQ0FA89dRTGs96IsNLYMTHvBg/VEs6cO4a3WtCyHAR10tx9uodzBrnATsrE7bjcMKgpseuWrUKCxYsgIvL3VvXt2zZguDgYK0GIwOLDnaC19kK/HDiNqKDRSNmbXxCtGn38VKYmggwe5I321E4Y1BnFEqlEvv378dzzz2HRYsW4ciRI+jtpV3X2Mbn8fB0XAAa22Q4dKGK7TiE6L3C240oKG1E6hQfWJoJ2Y7DGYMqFO+//z7Onj2LpUuX4vnnn8fly5fx3nvvaTsbGYQQH3tEBToi60wFWmjBQEKGTKFU4tucm3CyNcPMcR5sx+GUQRWKkydP4tNPP8WsWbOQkJCAf/zjHzhx4oS2s5FBWhAXgN5eJd2ER8gj+PlyDWobO7FgRgCEAtqq536D6g2GYSAU/uc0zNjYuM9jwi5ne3PMivbA6YJaVNzh3txuQriuQybH3pO3EexlO2IX/lNnUIUiODgYb7/9NiorK1FVVYV33nkHo0aN0nY2ooHUKT6wMBPim5ybGMQ9lISQ++w7VY5OWS8Wzgxk9QZWrhpUodiwYQPa2tqwcOFCLFiwAI2NjVi0aJG2sxENmJsK8USMH25UtSCvRDLwAYQQAEBtYweOXhJjeoQbJxba5KJBTY+1tLTEu+++2+e5sWPH4tKlS1oJRYYmJsIVRy+JkX7sFiICHGi6LCGDkH70FoQCPp6I8WM7CmcN+YoNDW9wjxGfj4UzAtHQKsPhi2K24xDCeUVljcj/ZTqsjYUx23E4a8iFgsbxuGmMrz0iAxyRlVtO+2sTooZCqcR3ObcgsjXFrGhPtuNwGs0BM0ALZgRA3qtExkmaLkvIw5y4UoPqhg4siKPpsANRe40iKirqgWcODMNAJpNpLRR5NC725pg5zgOHL1RhxlgPukBHyK90yuTIOFmGIE9bjB0lYjsO56ktFFlZWbrKQYZZ6lQf5BbdwTdHbuK1Zx5c8AkZqfadLkdHl5ymww6S2kLh7u6uqxxkmFmYCvH4dF/sPHQDeSUSRAdrvvc4IYaotrEDOXliTAt3HfE71w3WoKbHDqfvv/++z854YrEYaWlpeOONN1TPbd++HXv27IG1tTUAYMGCBVi8eLGuo+q92Eg3/Hy5Bt/k3ESonz1MjXX+4yaEUxiGwc5DN2AsNMI8mg47aDr/y/HUU0/hqaeeAgDcvHkTK1euxCuvvNLnNUVFRfjggw8QFRWl63gGxYjPx5KkILz97zxknirD0zMC2Y5ECKvOXatDcUUznksYBRtL2mtisFi91P/mm29i9erVsLe37/N8UVERduzYgdTUVGzcuBHd3TTNc6gC3G0QE+GGwxfEqKyjdaDIyNUhk+PbnJvwdbVCbCQNq2uCtUKRm5sLmUyG2bNn93m+o6MDISEhWLNmDTIyMtDW1oZPPvmEpZSG4cnH/GFuKsC/D5VASTdKkhHqh+O30d4lx5LEYPD5dAFbEzyGpVusV61ahYSEhAG3VL127RrWrVuHvXv36iiZYTp6sRJbv7mMV56KQOIkH7bjEKJTNyqb8edtJ5A6zQ8vPB7Gdhy9w8rVzZ6eHly4cKHf+lEAUFNTg9zcXDz55JMA7l58Egg0i9nYKIVSqXn9E4msIJFwb3hmOHKFetkiyNMWX+6/igAXK1gP03IFhtxn2kC5NDMcuRRKJT769hJsLIyRGO0xLN+nofUXn8+Dg4Plw9sfJdRQlZSUwMfHB+bm5v3aTE1NsXnzZlRVVYFhGOzatQvx8fEspDQsPB4PzyUGQdajQPqxW2zHIURnjuZVo7JOikWzRsHMhGb+DQUrhaKqqgouLi59nnvhhRdQWFgIe3t7bNy4ES+//DKSkpLAMAyef/55NmIaHDdHCyRN9EJu0R1cr2hmOw4hWtfc3o2Mk7cR6meP6CC6A3uoWCmvc+bMwZw5c/o899lnn6m+TkxMRGJioq5jjQgpU3xw7lod/n2oBH9fPgECI1rjhhiub3JuQqFk8Gz8KLoD+xHQX4kRxkRohGcTRqG2sRPZ5yrZjkOI1hTebsTF6/VImewNJ7v+w9xk8KhQjEDh/o4YFyTC/txy1Ld0sR2HkGHXI1dg56ESuNibI2miN9tx9B4VihFq0cxA8Pk87Dp0gzahIgYn60wFJC0yPJcwipYQHwbUgyOUvbUpnpjuh8LbjTh7tY7tOIQMm8q6dhw8W4HJY5wR4mM/8AFkQFQoRrBZ4zwQ4G6Dr4/cQAvthkcMQK9CiS9+LIaFmRCLZo1iO47BoEIxgvH5PCxPDkFPrxJfZZfQEBTRe1m55aiql2JpYhAszYRsxzEYVChGOBd7c8yP8cOVWw3ILbrDdhxChqziTjt+PHN3yCmKdq0bVlQoCGZFeyLAwwbfHLmJ5nYagiL6596QkyUNOWkFFQoCPp+H38wJQa9Cif/Lvk5DUETv7D9dDrFEiqVJwTTkpAVUKAgAwNneHPNj/VFQ2ojThTQERfTHvSGnKaEuiAx0ZDuOQaJCQVRmRntglIcNvsm5iaY2GdtxCBmQvFeJz3+8BmsLIRbNoh0ctYUKBVHh83h4PjkECqUS/6IhKKIH9ueWoVrSgaVJwbAwpSEnbaFCQfpwtjPHk7H+KLrdhFMFtWzHIeShymrbcOBMJaaGuSAigIactIkKBelnxjgPBHna4tujNARFuEneq8T//lgMG0tjLJpJQ07aRoWC9HNvCEqpBL74sXhIuwUSok0ZJ26juuHukJM5DTlpHRUK8kBOtmZYNCsQxRXNOHiugu04hKgU3m5E9vlKxEW5I9zfge04IwIVCvJQ08NdMSHECRknynBL3Mp2HELQIu3G51nX4CGywNMzAtiOM2JQoSAPxePxsCQxGPbWJtixrwgdMjnbkcgIplQy+Gz/NXTLFfhtWiiMhUZsRxoxWCkUzz33HJKTk5GWloa0tDTk5+f3ac/NzUVqaioSEhKwdetWNiKSX5ibCvDbtFC0SHvwr4M0ZZaw58DZChRXNGPxrFFwc7RgO86IovM9sxmGQXl5OY4dOwaBoP/by2QyrFu3Dv/+97/h6uqKl156CcePH0dsbKyuo5Jf+LlZY16sH74/Voqfr9QgLsqd7UhkhLklbsXek2WYEOKEaeGubMcZcXR+RnH79m0AwPLlyzF37lzs3LmzT3tBQQG8vb3h6ekJgUCA1NRUZGdn6zom+ZXECV4I9bXHN0duQlwvZTsOGUE6ZHLs2FcEBxsTLEkMBo/HYzvSiKPzM4q2tjZMnjwZf/vb3yCXy7FkyRL4+vpi6tSpAID6+nqIRP9ZIiCllX8AABY4SURBVNjJyQl1dZrtwObgYDnkfCKR1ZCP1SYu5PrL0glY9f4xfPbjNXzw+1iYmtz99eFCtgehXJrhYi6GYfB1zi20SHvw3qvT4e1px3YkFS72F6CdXDovFFFRUYiKilI9fvLJJ3H8+HFVoVAqlX0+MTAMo/EniMZG6ZDm/otEVpBI2jU+Ttu4lGt5cgg++PYK/ue7S1g2O4RT2e5HuTTD1VwXbjbgTGEtFsQFwM5MwJmMXO2voebi83lqP2DrfOjp4sWLOHPmjOoxwzB9rlW4uLhAIpGoHkskEjg5Oek0I3m4MT72mDPZGyfya3G+mPbaJtpTVS/F55lFCPWzR8IET7bjjGg6LxTt7e1477330N3dDalUioyMDMTHx6vaIyIiUFZWhoqKCigUCmRlZSEmJkbXMYkaadN84e9ujf/Lvo6aBrpeQYZfV3cvPs0sgqWZECuSR4NP1yVYpfNCERcXh9jYWDz++OOYP38+5s+fj6ioKKSlpaGurg4mJiZ499138eqrr2LOnDnw8/NDUlKSrmMSNQRGfLw0dwz4PB7e+t/z6OruZTsSMSBKhsEXPxbjTlMn/rR4HKwtjNmONOLxGAOcGE/XKHTjWnkTPkjPR4S/A1bOC+PUpz6u9hnlGljmqTJknirDwpmBWDxnNGdy3Y9L/XU/g7lGQQzHaB97/CZ1DC7fbMC+U2VsxyEG4NINCTJPlWFqqAvioz3YjkN+QYWCPJLU6X6YGuaCfafLkVdSz3YcoseqJVJ8lnUNvq5WWJIURPdLcAgVCvJI7q4HFQRfV2t8nlVMN+ORIZF2yfE/ewphKjTCK/PCIRTQOk5cQoWCPDKhwAivzAuDqYkR/ueHAki7aPFAMngKpRI7MovQ1C7DynlhsLMyYTsS+RUqFGRY2FmZ4JUnwtDc3o1PM4ugUCrZjkT0xO6fS3G1vBnPJgQhwN2G7TjkAahQkGHj726D5xKCcK28Gd8fK2U7DtEDuUW1+Ol8FWaMdUdMhBvbcchD6HwJD2LYpke4obJeikMXquDpZImpYbTSJ3mwsto2/OtgCYI8bbGQ9r3mNDqjIMPu6RkBCPayxb8OXse18ia24xAOqm/pwke7C2BjYYyXnwiFwIj+FHEZ/XTIsBMY8bFyXhhc7M2x/YdCVNzh3o1JhD2tHT344NsrUCiUWL0gAtbmdOc111GhIFphYSrEH5+OhIWpAFvTr6C+uZPtSIQDurp78WF6Plqk3fj9UxG0U52eoEJBtMbOygR/fDoSCiWDD77LR2tHD9uRCIt6FUp8nFGIqnopXn48lGY46REqFESrXB0s8IcFEWjp6MbW9Cu0gOAIpWQYfJ51DdfKm/H8nGBEBDiyHYlogAoF0Tp/Nxv87vEwiOs7sP2HQsh76R6LkYRhGHx75CbOF9fjqcf8aSacHqJCQXQi3N8Bz88JRnFFMz7Pugal4S1aTB7iwNkKHMkTI2G8J5ImerEdhwwB3UdBdGZqmCvaO+VIP3YL1hbGeGZWIC38ZuBOFtRgz/HbmDTaGQtmBNDPW09RoSA6lTTRC60d3fjpfBXMTQR4fLov/fEwUBev1+P/DpZgjK89lieHcGq/EqIZKhRE556KC0CnrBf7c8uhUDKYH+tHxcLAnL16B59lXUOAuw1W0g11eo+VQrF9+3YcPHgQABAbG4vXXnutX/uePXtgbW0NAFiwYAEWL16s85xEO/g8HpbODobAiI8DZyvQq1DiaRqWMBinC2vxvz8WI8jLFqueDIepMX0e1Xc6/wnm5ubi1KlTyMjIAI/Hw4oVK3D48GHEx8erXlNUVIQPPvgAUVFRuo5HdITP4+HZhFEwMuLh0IUq9CqUeCZ+FA1P6LnjV6rxVXYJRvvY4ZX54TAR0r4ShkDnhUIkEmHt2rUwNr57276/vz9qamr6vKaoqAg7duxAdXU1xo8fj7/85S8wMaE16g0Nj8fDopmBEBjxkX2uEr0KJZYkBVOx0FM5eWLsOnwD4f4OWPlEKG0+ZEB0PnAYGBiIyMhIAEB5eTkOHjyI2NhYVXtHRwdCQkKwZs0aZGRkoK2tDZ988omuYxId4fF4eOoxf6RM8cGJ/Fp8+WMxlEqaOqtvfjpfiV2HbyAq0BErnwijImFgeAzDzoT2mzdv4qWXXsKrr76KJ5544qGvu3btGtatW4e9e/fqMB1hw7eHS7Ar+zpiotzxx0VjYUQXQPXC9zk38NWBYkyNcMOfF4+jC9cGiJWrTHl5eVi1ahXWrVuH5OTkPm01NTXIzc3Fk08+CeDuXZ0CgWYxGxulQ/pUKhJZQSLh3kqnXM0FDG+2mZFu6JbJsfvnUnR09ODFuaOH/MmUq31mSLkYhkHmqTLsO12OSaOdsSxxFJqbOljPpQuGlovP58HBwfLh7Y8Saihqa2uxcuVKbNmypV+RAABTU1Ns3rwZVVVVYBgGu3bt6nOhmxi2OZO8sXBmIPJuSLD5mytoo4UEOUneq8TnWcXYd7oc08JcsSJlNIz4dCZhqHR+RvHFF1+gu7sb7777ruq5hQsX4ujRo1i1ahXCwsKwceNGvPzyy5DL5Rg7diyef/55XcckLEoY7wl7KxN8lnUNb311EaueDIeH6OGfdohutXX2YPsPhbglbsXj032ROsWHpjYbONauUWgTDT3pjjazldW2YdvuAnTLFXj58VCE+TlwItej0Pdc1Q0d+Oj7u0vG/yY5BBNCnDmRS9cMLRfnhp4IGSxfV2v8bWk0RLZm+PD7fOTkidmONKIVlTXi7X9fRE+vEq89E6X1IkG4gwoF4TR7a1O8/uxYRPg7YtfhG9h5qAQKJS1TrmtHL4nxYXoBHKzN8Lcl0fB3o02HRhK6t55wnqmxAK/MC8Pun0uRfb4Sdc1deDktFOam9OurbQqlEt/m3EJOnhgR/g54ce4YmJlQv480dEZB9AKfz8OCGQFYNjsY1yuasfFfF3C7po3tWAatsVWGzd9cQc4ve0m8Oj+cisQIRT91oldiItzgYm+Of+6/ind25iFtmi/mTPIGn0+zbobT+eI6fJVdAgXD4DfJIbQr3QhHhYLonVGetvj78gn4KrsEP5y4jatlTXghdTTsrU3Zjqb3urp78fWRGzhdeAe+rtZ4ce5oONuZsx2LsIwKBdFLFqZC/DZtDML8HLDr8A288cV5LJ0djPHBTmxH01u3a9rwz31XIWnpQsoUb8yd6kvLcRAAVCiIHuPxeJgW7opATxv8c99V/GNvEQrDXPFMfCDb0fSKUskg/cgNfP3TddhYGuO1Z6IQ5GXHdizCIVQoiN5ztjPH68+Ow77TZfgxtwI3xC1Y9XQU3GxpKGogNQ0d+OqnEtyoasH4YCcsSQqChamQ7ViEY6hQEIMgMOJjXow/xvjY44sfi/HXT3MxLkiEp+MC4GhrxnY8zumUyZF5qhxHL4lhLDTCHxZGIczblpbiIA9EhYIYlCAvO7y1YiJOXa1D+pEbKChtxOyJXpg9yZt2WwOgZBicKqjFnuOlkHbKERPphidi/ODv7cDJJSkIN1ChIAbHWGiEp+ODEOlnj/Rjt7DvdDlOFdbi6RmBiA4SjdhPzbeqW7Hr8A1U3GlHgIcN/rhgFLxdrNiORfQAFQpisOytTfHbtFDERTXj6yM38Y+9RQj2ssXCmYHwch45fyCb27ux++dSnLl6B3ZWJngxdTQmjnYesQWTaI4KBTF4QV522LBsPI7n1yDjxG28+eUFhPs7IHGCF4K9DHdcvloixU/nq3Dm6h3weEDyZG8kT/aGqTH9b080Q78xZETg83mIi3LHhBAnHM0TIydPjM3fXIa3ixWSJnghOlhkEBvvMAyD65UtyD5XicLbjTAW8vFYpDsSJnhCRBf1yRBRoSAjioWpEKlTfZE00Qu5RXeQfb4KO/Zdxe6fTZEw3hPTI1z18hO3QqnExesSZJ+rREVdO6zNhXhiui/ixnrA0oymu5JHo3//RxAyDIQCI8RGumN6hBvybzUg+1wlvsm5icxTZZg42hnjgkQI8rLl9FkGwzCoqGtHXokEZ6/WobFNBhd7cyxNCsKUUJch7zdOyK9RoSAjGp/HQ1SgCFGBIpRWt+LwxSqcLqrFscvVsDQTIjLQEdFBIoz2sefEchZKhsHtmjbkldQjr0SChlYZ+Dwegr1t8Ux8ICICHME30GsuhD2sFIr9+/fjH//4B3p7e7F06VIsXry4T3txcTHWr1+Pjo4OREdH4+9//zsEAqppRLv83W3g726DbrkCRbcbkVciwcXr9ThVUAszEyNEBDhibKAIfm7WsLMy0dlFcGmXHOW1bci/1Yi8G/VokfbAiM/DGF97pE7xQWSgI6zMjXWShYxMOv/rW1dXh61bt+KHH36AsbExFi5ciIkTJyIgIED1mjVr1uCtt95CZGQk1q1bh/T0dDzzzDO6jkpGKBOhEcYFOWFckBPkvUpcK29CXokEl2/eHeIBACtzIbydreDtYgUvZyt4O1tCZGv2yMWjRdqNijvtqKhrR8WddlTWSdHYJgMACAV8hPraIzrYCRH+jrRxE9EZnf+m5ebmYtKkSbC1tQUAJCYmIjs7G6+88goAoLq6GjKZDJGRkQCAefPmYdu2bVQoCCuEAj4iAhwREeCIXkUQymvv/yPejuxzlVAoGQCAmYkATrZmMDcVwNxUAAtTAcxNhTA3ufu1qYkAxiaNqG+QokPWi06ZHJ3dvaqvm9q60drRo3pvZ3tz+LtbY8Y4d3g5W8HfzVovL7QT/afz37r6+nqIRCLVYycnJxQUFDy0XSQSoa6uTqP3cHCwHHI+kYibN2JxNRfA3WzayOXqYoPJ9z3ukStQeacdpdUtuCVuRUNLF6SdPahv6YK0U46OLjl6evvv8W3E58HSXAhLMyEszISwszaDr7vt3eEvD1v4ulnDXMeL842kn+NwGEm5dF4olEpln9NzhmH6PB6ofTAaG6VQ/vIpTxMikRUn17vhai6Au9l0mcvG1Ahj/R0w1t/hge3yXgU6Zb3o7O6Fm4sNZB3dMBby1f5ed7TL0NEu01bkfujnqBlDy8Xn89R+wNb5NA4XFxdIJBLVY4lEAicnp4e2NzQ09GknRN8IBUawsTSBq4MFHG3NYGJsZLB3gxPDpPNCMWXKFJw5cwZNTU3o6urCoUOHEBMTo2p3d3eHiYkJ8vLyAACZmZl92gkhhOiWzguFs7MzVq9ejSVLluDxxx9HSkoKwsPD8cILL6CwsBAAsGXLFrzzzjtISkpCZ2cnlixZouuYhBBCfsFjGEbzwXyOo2sUusPVbJRLM5RLM4aWi3PXKAghhOgXKhSEEELUokJBCCFELYO8zZPPH/rUw0c5Vpu4mgvgbjbKpRnKpRlDyjXQMQZ5MZsQQsjwoaEnQgghalGhIIQQohYVCkIIIWpRoSCEEKIWFQpCCCFqUaEghBCiFhUKQgghalGhIIQQohYVCkIIIWqNyEKxf/9+zJkzBwkJCdi1a1e/9uLiYsybNw+JiYlYv349ent7OZFr+/btiIuLQ1paGtLS0h74Gm2RSqVISUmBWCzu18ZWfw2Ui63+2r59O5KTk5GcnIz33nuvXztb/TVQLrb666OPPsKcOXOQnJyML7/8sl87m79fA2Vj8//J//7v/8batWv7PV9TU4PFixcjKSkJL7/8Mjo6Oh79zZgR5s6dO0xcXBzT3NzMdHR0MKmpqczNmzf7vCY5OZm5fPkywzAM8/rrrzO7du3iRK6XXnqJuXTpktaz/NqVK1eYlJQUZsyYMUxVVVW/djb6azC52Oiv06dPM08//TTT3d3N9PT0MEuWLGEOHTrU5zVs9NdgcrHRX+fOnWMWLlzIyOVypquri4mLi2NKS0v7vIat36/BZGPr/8nc3Fxm4sSJzF/+8pd+bS+++CKTlZXFMAzDbN++nXnvvfce+f1G3BlFbm4uJk2aBFtbW5ibmyMxMRHZ2dmq9urqashkMkRGRgIA5s2b16edrVwAUFRUhB07diA1NRUbN25Ed3e31nMBQHp6OjZs2PDAvcvZ6q+BcgHs9JdIJMLatWthbGwMoVAIf39/1NTUqNrZ6q+BcgHs9NeECRPw1VdfQSAQoLGxEQqFAubm5qp2Nn+/BsoGsNNnLS0t2Lp1K37729/2a5PL5bhw4QISExMBDF9/jbhCUV9fD5FIpHrs5OSEurq6h7aLRKI+7Wzl6ujoQEhICNasWYOMjAy0tbXhk08+0XouANi0aROio6Mf2MZWfw2Ui63+CgwMVP1RKy8vx8GDBxEbG6tqZ6u/BsrF5u+XUCjEtm3bkJycjMmTJ8PZ2VnVxubv10DZ2OqzN954A6tXr4a1tXW/tubmZlhaWkIguLsw+HD114grFEqlEjzef5bUZRimz+OB2tnKZWFhgc8++wz+/v4QCARYvnw5jh8/rvVcA2GrvwbCdn/dvHkTy5cvx2uvvQYfHx/V82z318Nysd1fq1atwpkzZ1BbW4v09HTV82z3l7psbPTZ999/D1dXV0yePPmB7Q/qn+HorxFXKFxcXCCRSFSPJRJJn6GLX7c3NDQ8dGhDl7lqamqwe/du1WOGYVSfGtjEVn8NhM3+ysvLw7Jly/CnP/0JTzzxRJ82NvtLXS62+qu0tBTFxcUAADMzMyQkJKCkpETVzmZ/DZSNjT47cOAATp8+jbS0NGzbtg1Hjx7F22+/rWq3t7dHe3s7FAoFgP5/R4ZqxBWKKVOm4MyZM2hqakJXVxcOHTqEmJgYVbu7uztMTEyQl5cHAMjMzOzTzlYuU1NTbN68GVVVVWAYBrt27UJ8fLzWcw2Erf4aCFv9VVtbi5UrV2LLli1ITk7u185Wfw2Ui63+EovF+Otf/4qenh709PQgJycH48aNU7Wz+fs1UDY2+uzLL79EVlYWMjMzsWrVKsyYMQPr1q1TtQuFQkRHR+PAgQMAgL179w5Pfz3y5XA9tG/fPiY5OZlJSEhg/vnPfzIMwzArVqxgCgoKGIZhmOLiYmb+/PlMYmIi88c//pHp7u7mRK7s7GxV+9q1a3WW6564uDjV7CIu9NdAudjor//6r/9iIiMjmblz56r++/rrr1nvr8HkYuv3a9u2bczs2bOZlJQUZtu2bQzDcOf3a6BsbP4/uWfPHtWsp3Xr1jFHjhxhGIZhxGIx8+yzzzKzZ89mli9fzrS0tDzye9EOd4QQQtQacUNPhBBCNEOFghBCiFpUKAghhKhFhYIQQohaVCgIIYSoRYWCEEKIWlQoCCGEqEWFghAtSUtLw5kzZwAAWVlZCAsLg0wmAwCsX78eX3/9NZvxCBk0KhSEaEl8fDxOnDgBADh58iRsbGxw8eJFMAyD48ePc2IJFkIGgwoFIVpyr1AwDIOLFy9i2bJlOH36NK5cuQIvL68+y2cTwmXsLz9KiIEKCgqCXC5HTk4OfHx8EBcXh9WrV0MgEKg2liFEH9AZBSFaNGvWLLz//vuYOnUq/P39IZVKsX//fiQkJLAdjZBBo0JBiBbFx8fj9u3bmDJlCoC7y8mLRCK4urqynIyQwaPVYwkhhKhFZxSEEELUokJBCCFELSoUhBBC1KJCQQghRC0qFIQQQtSiQkEIIUQtKhSEEELUokJBCCFErf8HjyuX/tITVBgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot it all\n",
    "plt.plot(w_list, mse_list)\n",
    "plt.ylabel('Loss')\n",
    "plt.xlabel('w')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-13T03:40:03.659167Z",
     "start_time": "2020-08-13T03:40:03.608503Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 w= 2.0 loss= 0.0\r",
      "Epoch: 1 w= 2.0 loss= 0.0\r",
      "Epoch: 2 w= 2.0 loss= 0.0\r",
      "Epoch: 3 w= 2.0 loss= 0.0\r",
      "Epoch: 4 w= 2.0 loss= 0.0\r",
      "Epoch: 5 w= 2.0 loss= 0.0\r",
      "Epoch: 6 w= 2.0 loss= 0.0\r",
      "Epoch: 7 w= 2.0 loss= 0.0\r",
      "Epoch: 8 w= 2.0 loss= 0.0\r",
      "Epoch: 9 w= 2.0 loss= 0.0\r",
      "Epoch: 10 w= 2.0 loss= 0.0\r",
      "Epoch: 11 w= 2.0 loss= 0.0\r",
      "Epoch: 12 w= 2.0 loss= 0.0\r",
      "Epoch: 13 w= 2.0 loss= 0.0\r",
      "Epoch: 14 w= 2.0 loss= 0.0\r",
      "Epoch: 15 w= 2.0 loss= 0.0\r",
      "Epoch: 16 w= 2.0 loss= 0.0\r",
      "Epoch: 17 w= 2.0 loss= 0.0\r",
      "Epoch: 18 w= 2.0 loss= 0.0\r",
      "Epoch: 19 w= 2.0 loss= 0.0\r",
      "Epoch: 20 w= 2.0 loss= 0.0\r",
      "Epoch: 21 w= 2.0 loss= 0.0\r",
      "Epoch: 22 w= 2.0 loss= 0.0\r",
      "Epoch: 23 w= 2.0 loss= 0.0\r",
      "Epoch: 24 w= 2.0 loss= 0.0\r",
      "Epoch: 25 w= 2.0 loss= 0.0\r",
      "Epoch: 26 w= 2.0 loss= 0.0\r",
      "Epoch: 27 w= 2.0 loss= 0.0\r",
      "Epoch: 28 w= 2.0 loss= 0.0\r",
      "Epoch: 29 w= 2.0 loss= 0.0\r",
      "Epoch: 30 w= 2.0 loss= 0.0\r",
      "Epoch: 31 w= 2.0 loss= 0.0\r",
      "Epoch: 32 w= 2.0 loss= 0.0\r",
      "Epoch: 33 w= 2.0 loss= 0.0\r",
      "Epoch: 34 w= 2.0 loss= 0.0\r",
      "Epoch: 35 w= 2.0 loss= 0.0\r",
      "Epoch: 36 w= 2.0 loss= 0.0\r",
      "Epoch: 37 w= 2.0 loss= 0.0\r",
      "Epoch: 38 w= 2.0 loss= 0.0\r",
      "Epoch: 39 w= 2.0 loss= 0.0\r",
      "Epoch: 40 w= 2.0 loss= 0.0\r",
      "Epoch: 41 w= 2.0 loss= 0.0\r",
      "Epoch: 42 w= 2.0 loss= 0.0\r",
      "Epoch: 43 w= 2.0 loss= 0.0\r",
      "Epoch: 44 w= 2.0 loss= 0.0\r",
      "Epoch: 45 w= 2.0 loss= 0.0\r",
      "Epoch: 46 w= 2.0 loss= 0.0\r",
      "Epoch: 47 w= 2.0 loss= 0.0\r",
      "Epoch: 48 w= 2.0 loss= 0.0\r",
      "Epoch: 49 w= 2.0 loss= 0.0\r",
      "Epoch: 50 w= 2.0 loss= 0.0\r",
      "Epoch: 51 w= 2.0 loss= 0.0\r",
      "Epoch: 52 w= 2.0 loss= 0.0\r",
      "Epoch: 53 w= 2.0 loss= 0.0\r",
      "Epoch: 54 w= 2.0 loss= 0.0\r",
      "Epoch: 55 w= 2.0 loss= 0.0\r",
      "Epoch: 56 w= 2.0 loss= 0.0\r",
      "Epoch: 57 w= 2.0 loss= 0.0\r",
      "Epoch: 58 w= 2.0 loss= 0.0\r",
      "Epoch: 59 w= 2.0 loss= 0.0\r",
      "Epoch: 60 w= 2.0 loss= 0.0\r",
      "Epoch: 61 w= 2.0 loss= 0.0\r",
      "Epoch: 62 w= 2.0 loss= 0.0\r",
      "Epoch: 63 w= 2.0 loss= 0.0\r",
      "Epoch: 64 w= 2.0 loss= 0.0\r",
      "Epoch: 65 w= 2.0 loss= 0.0\r",
      "Epoch: 66 w= 2.0 loss= 0.0\r",
      "Epoch: 67 w= 2.0 loss= 0.0\r",
      "Epoch: 68 w= 2.0 loss= 0.0\r",
      "Epoch: 69 w= 2.0 loss= 0.0\r",
      "Epoch: 70 w= 2.0 loss= 0.0\r",
      "Epoch: 71 w= 2.0 loss= 0.0\r",
      "Epoch: 72 w= 2.0 loss= 0.0\r",
      "Epoch: 73 w= 2.0 loss= 0.0\r",
      "Epoch: 74 w= 2.0 loss= 0.0\r",
      "Epoch: 75 w= 2.0 loss= 0.0\r",
      "Epoch: 76 w= 2.0 loss= 0.0\r",
      "Epoch: 77 w= 2.0 loss= 0.0\r",
      "Epoch: 78 w= 2.0 loss= 0.0\r",
      "Epoch: 79 w= 2.0 loss= 0.0\r",
      "Epoch: 80 w= 2.0 loss= 0.0\r",
      "Epoch: 81 w= 2.0 loss= 0.0\r",
      "Epoch: 82 w= 2.0 loss= 0.0\r",
      "Epoch: 83 w= 2.0 loss= 0.0\r",
      "Epoch: 84 w= 2.0 loss= 0.0\r",
      "Epoch: 85 w= 2.0 loss= 0.0\r",
      "Epoch: 86 w= 2.0 loss= 0.0\r",
      "Epoch: 87 w= 2.0 loss= 0.0\r",
      "Epoch: 88 w= 2.0 loss= 0.0\r",
      "Epoch: 89 w= 2.0 loss= 0.0\r",
      "Epoch: 90 w= 2.0 loss= 0.0\r",
      "Epoch: 91 w= 2.0 loss= 0.0\r",
      "Epoch: 92 w= 2.0 loss= 0.0\r",
      "Epoch: 93 w= 2.0 loss= 0.0\r",
      "Epoch: 94 w= 2.0 loss= 0.0\r",
      "Epoch: 95 w= 2.0 loss= 0.0\r",
      "Epoch: 96 w= 2.0 loss= 0.0\r",
      "Epoch: 97 w= 2.0 loss= 0.0\r",
      "Epoch: 98 w= 2.0 loss= 0.0\r",
      "Epoch: 99 w= 2.0 loss= 0.0\r"
     ]
    }
   ],
   "source": [
    "# compute gradient\n",
    "def gradient(x, y):  # d_loss/d_w\n",
    "    return 2 * x * (x * w - y)\n",
    "\n",
    "# Training loop\n",
    "for epoch in range(100):\n",
    "    for x_val, y_val in zip(x_data, y_data):\n",
    "        # Compute derivative w.r.t to the learned weights\n",
    "        # Update the weights\n",
    "        # Compute the loss and print progress\n",
    "        grad = gradient(x_val, y_val)\n",
    "        w = w - 0.01 * grad\n",
    "        #print(\"\\tgrad: \", x_val, y_val, round(grad, 2))\n",
    "        y_pred = forward(x_val)\n",
    "        l = loss(y_pred, y_val)\n",
    "    print(\"Epoch:\", epoch, \"w=\", round(w, 2), \"loss=\", round(l, 2),  end='\\r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Auto Gradient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-13T03:41:27.541561Z",
     "start_time": "2020-08-13T03:41:26.706805Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "w = torch.tensor([1.0], requires_grad=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-13T03:44:23.779127Z",
     "start_time": "2020-08-13T03:44:23.716863Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 99 | Loss: 9.094947017729282e-13\n"
     ]
    }
   ],
   "source": [
    "# Training loop\n",
    "for epoch in range(100):\n",
    "    for x_val, y_val in zip(x_data, y_data):\n",
    "        y_pred = forward(x_val) # 1) Forward pass\n",
    "        l = loss(y_pred, y_val) # 2) Compute loss\n",
    "        l.backward() # 3) Back propagation to update weights\n",
    "        #print(\"\\tgrad: \", x_val, y_val, w.grad.item())\n",
    "        w.data = w.data - 0.01 * w.grad.item()\n",
    "        # Manually zero the gradients after updating weights\n",
    "        w.grad.data.zero_()\n",
    "\n",
    "print(f\"Epoch: {epoch} | Loss: {l.item()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-13T03:44:38.300392Z",
     "start_time": "2020-08-13T03:44:38.272470Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2.0000])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w.data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Back Propagation in Complicated network\n",
    "\n",
    "\n",
    "![image.png](./images/nn10.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](./images/nn11.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](./images/nn12.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](./images/nn13.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-13T03:53:35.218082Z",
     "start_time": "2020-08-13T03:53:35.214175Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from torch import nn\n",
    "import torch\n",
    "from torch import tensor\n",
    "from torch import sigmoid\n",
    "\n",
    "x_data = tensor([[1.0], [2.0], [3.0]])\n",
    "y_data = tensor([[2.0], [4.0], [6.0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-13T03:56:49.423764Z",
     "start_time": "2020-08-13T03:56:49.419364Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class Model(nn.Module):\n",
    "    def __init__(self):\n",
    "        \"\"\"\n",
    "        In the constructor we instantiate two nn.Linear module\n",
    "        \"\"\"\n",
    "        super(Model, self).__init__()\n",
    "        self.linear = torch.nn.Linear(1, 1)  # One in and one out\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        In the forward function we accept a Variable of input data and we must return\n",
    "        a Variable of output data. We can use Modules defined in the constructor as\n",
    "        well as arbitrary operators on Variables.\n",
    "        \"\"\"\n",
    "        y_pred = self.linear(x)\n",
    "        return y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-13T03:59:36.441381Z",
     "start_time": "2020-08-13T03:59:36.435510Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# our model\n",
    "model = Model()\n",
    "# Construct our loss function and an Optimizer. The call to model.parameters()\n",
    "# in the SGD constructor will contain the learnable parameters of the two\n",
    "# nn.Linear modules which are members of the model.\n",
    "criterion = torch.nn.MSELoss(reduction='sum')\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=0.01)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-13T04:00:31.825433Z",
     "start_time": "2020-08-13T04:00:31.721167Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 | Loss: 80.97074890136719 \n",
      "Epoch: 100 | Loss: 0.2520463764667511 \n",
      "Epoch: 200 | Loss: 0.059265460819005966 \n",
      "Epoch: 300 | Loss: 0.013935551047325134 \n",
      "Epoch: 400 | Loss: 0.003276776522397995 \n"
     ]
    }
   ],
   "source": [
    "# Training loop\n",
    "for k, epoch in enumerate(range(500)):\n",
    "    # 1) Forward pass: Compute predicted y by passing x to the model\n",
    "    y_pred = model(x_data)\n",
    "\n",
    "    # 2) Compute and print loss\n",
    "    loss = criterion(y_pred, y_data)\n",
    "    if k%100==0:\n",
    "        print(f'Epoch: {epoch} | Loss: {loss.item()} ')\n",
    "\n",
    "    # Zero gradients, perform a backward pass, and update the weights.\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-13T04:01:38.716592Z",
     "start_time": "2020-08-13T04:01:38.711952Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction (after training) 4 7.967859268188477\n"
     ]
    }
   ],
   "source": [
    "# After training\n",
    "hour_var = tensor([[4.0]])\n",
    "y_pred = model(hour_var)\n",
    "print(\"Prediction (after training)\",  4, model(hour_var).data[0][0].item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Pytorch  Rhythm\n",
    "\n",
    "![image.png](./images/nn14.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Regression\n",
    "\n",
    "Let’s start with a simple example of House Price. \n",
    "- Say you’re helping a friend who wants to buy a house.\n",
    "\n",
    "- She was quoted $400,000 for a 2000 sq ft house (185 meters). \n",
    "\n",
    "Is this a good price or not?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "So you ask your friends who have bought houses in that same neighborhoods, and you end up with three data points:\n",
    "\n",
    "\n",
    "\n",
    "| Area (sq ft) (x)  |  Price (y)   |  \n",
    "| -------------|:-------------:|\n",
    "|2,104|399,900|\n",
    "|1,600|329,900|\n",
    "|2,400|369,000|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "$$y = f(X) = W X$$\n",
    "\n",
    "- Calculating the prediction is simple multiplication.\n",
    "- But before that, we need to think about the weight we’ll be multiplying by. \n",
    "- “training” a neural network just means finding the weights we use to calculate the prediction.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A simple predictive model (“regression model”)\n",
    "- takes an input, \n",
    "- does a calculation, \n",
    "- and gives an output \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src= 'img/neuralnetwork/data_points_graph_animated.gif' width= \"700px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-07T03:15:14.317623Z",
     "start_time": "2019-04-07T03:15:14.313438Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Model Evaluation\n",
    "- If we apply our model to the three data points we have, how good of a job would it do?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "<img src= 'img/neuralnetwork/data_points_error_animated.gif' width= \"700px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "<img src= 'img/neuralnetwork/model_evaluation.png' width= \"500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Loss Function**\n",
    "\n",
    "how bad our prediction is\n",
    "\n",
    "- For each point, the error is measured by the difference between the **actual value** and the **predicted value**, raised to the power of 2. \n",
    "- This is called **Mean Square Error**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "<img src= 'img/neuralnetwork/lines_and_errors_animated.gif' width= \"700px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- We can't improve much on the model by varying the weight any more. \n",
    "- But if we add a bias (intercept) we can find values that improve the model.\n",
    "\n",
    "<img src= 'img/neuralnetwork/NNs_bias_2.png' width= \"500px\">\n",
    "\n",
    "$$y = 0.1 X + 150$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Gradient Descent**\n",
    "\n",
    "- Automatically get the correct weight and bias values \n",
    "- minimize the loss function.\n",
    "\n",
    "<img src= 'img/neuralnetwork/gd.png' width= \"700px\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Regression\n",
    "\n",
    "<img src= 'img/neuralnetwork/NNs_2_variables.png' width= \"500px\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T07:13:36.927633Z",
     "start_time": "2020-05-24T07:13:36.922702Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "from torch import nn, optim\n",
    "from torch.autograd import Variable\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T07:17:03.772993Z",
     "start_time": "2020-05-24T07:17:03.518421Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD7CAYAAACPDORaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAYPUlEQVR4nO3dfZCU5Z3u8e8MIEwCmGQYAgZRkfU30YTFinFNfEPztqm1CrMFYRNW426QGANbm8jheOpoBJMT1+jiHreKMosYrIOJZo3ZCgG2dgsTohJMRNGTEK6jBEnQSYWa3RwEURm6zx/PM6adM0w/Dd10c3t9/tG++366r5oerr7nfvqlrVwuY2ZmaWpvdgAzM2scl7yZWcJc8mZmCXPJm5klzCVvZpaw4c0OUGEk8H6gBzjU5CxmZseLYcBE4GfAqwOvbKWSfz/wSLNDmJkdpy4EHh042Eol3wPwn/+5n1Kp9tfud3aOprd3X91DHa1WzQWtm825auNctUktV3t7G29/+1sh79CBWqnkDwGUSuUjKvn+Y1tRq+aC1s3mXLVxrtokmmvQbW6feDUzS5hL3swsYS55M7OEFd6Tj4jbgXGSroqI6cDdwFjgx8A1kvoiYjKwGhgPCJgrqfXOcJiZvUkUWslHxIeAz1QMrQYWSDoDaAOuzseXA8sldQNPADfWMauZmdWoaslHxDuA/wF8Lb98CtAhaXM+ZRUwOyJGABcBD1aO1zmvmQ2wd7v4j3U/4MCO55odxVpQke2abwD/HTg5v3wSb3w9Zg8wCRgH7JXUN2C8Jp2do2s95HVdXWOO+NhGatVc0LrZnKuYvdvFL25cQqmvj/bhwznrK0sY2x3NjvW6Vvt59Xsz5Rqy5CNiHvAbSRsi4qp8uB2ofDFnG1AaZJx8vCa9vfuO6LWiXV1j2LPnpZqPa7RWzQWtm825ivuPx5+k1NcHpRKlvj56Hn+SVztPanYsoDV/XpBervb2tiEXx9VW8nOAiRGxFXgHMJqsyCdWzJkAvAj8DjgxIoZJOpTPebHmxGZWWEd00z58OKW+PtqGDacjupsdyVrMkHvykj4i6T2SpgNfBr4v6a+AVyLi/HzaFcB6SQfJPntmTj5+JbC+QbnNDOg4fSpnfWUJ4y7/cyYtWkzH6VObHclazJF+rMFcYEVEjAWeBO7Mx68F7o2IG4BfA586+ohmNpSx3dEyWzTWegqXvKRVZK+YQdLTwLmDzNkFzKhPNDMzO1p+x6uZWcJc8mZmCXPJm5klzCVvZpYwl7yZWcJc8mZmCXPJm5klzCVvZpYwl7yZWcJc8mZmCXPJm5klzCVvZpYwl7yZWcJc8mZmCXPJm5klzCVvZpYwl7yZWcJc8mZmCXPJm5klrNB3vEbEzcAsoAyslLQsIq4CFgOHgIeB6yT1RcRkYDUwHhAwV9K+RoQ3M7OhVV3JR8TFwKXANOAcYGFEBPBV4EOS3guMAP4mP2Q5sFxSN/AEcGMjgpuZWXVVS17SRuASSX1kq/PhwLnATyT15NN+AFweESOAi4AH8/FVwOx6hzYzs2IK7clLOhgRS4FtwAbgp8B5EXFyRAwj28qZAIwD9uZPCAA9wKT6xzYzsyLayuVy4ckR8RZgDfAAsB9YBBwAvgPMAz4GbJZ0cj5/OLBP0qgCN38qsLOW8GZm9rrTgOcHDlY98RoR3cAoSVslvRwRD5Ft19wq6ex8zmxgB/A74MSIGCbpEDAReLGWlL29+yiVij/x9OvqGsOePS/VfFyjtWouaN1szlUb56pNarna29vo7Bx9+OsL3MYUYEVEjIyIE4CZwA+BDRExJh9bCDwg6SDwCDAnP/ZKYH3Nqc3MrC6KnHhdB6wFngK2AJsk3QcsBTYDPwcelvSt/JBrgfkRsQ24ELihEcHNzKy6Qq+Tl7QEWDJgbCWwcpC5u4AZRx/NzMyOlt/xamaWMJe8mVnCXPJmZglzyZuZJcwlb2aWMJe8mVnCXPJmZglzyZuZJcwlb2aWMJe8mVnCXPJmZglzyZuZJcwlb2aWMJe8mVnCXPJmZglzyZuZJcwlb2aWMJe8mVnCXPJmZgkr9B2vEXEzMAsoAyslLYuIjwK3AcOAJ4F5kl6LiMnAamA8IGCupH0NSW9mZkOqupKPiIuBS4FpwDnAwogIsi/x/gtJ7wHeAlyZH7IcWC6pG3gCuLERwc3MrLqqJS9pI3CJpD6y1flwYD/ZCn5sRAwDRgEHImIEcBHwYH74KmB2A3KbmVkBhfbkJR2MiKXANmAD8AJwLfAj4EVgHFmxjwP25k8IAD3ApDpnNjOzgtrK5XLhyRHxFmANsBH4NPBnwE5gGTAC+BqwWdLJ+fzhwD5Jowrc/Kn5bZmZWe1OA54fOFj1xGtEdAOjJG2V9HJEPES2iv+5pB35nBXAd4C/BU6MiGGSDgETyVb6hfX27qNUKv7E06+rawx79rxU83GN1qq5oHWzOVdtnKs2qeVqb2+js3P04a8vcBtTgBURMTIiTgBmkr165tyIeGc+ZybwM0kHgUeAOfn4lcD6mlObmVldFDnxug5YCzwFbAE2SbqF7FUzP4yIZ8hedbMoP+RaYH5EbAMuBG5oRHAzM6uu0OvkJS0BlgwYuxe4d5C5u4AZRx/NzMyOlt/xamaWMJe8mVnCXPJmZglzyZuZJcwlb2aWMJe8mVnCXPJmZglzyZuZJcwlb2aWMJe8mVnCXPJmZglzyZuZJcwlb2aWMJe8mVnCXPJmZglzyZuZJcwlb2aWMJe8mVnCXPJmZglzyZuZJazQF3lHxM3ALKAMrAS2A1+rmPIu4HFJl0XEdOBuYCzwY+AaSX11TW1mZoVUXclHxMXApcA04BxgIbBD0nRJ04E/BfYCX8wPWQ0skHQG0AZc3YjgZmZWXdWSl7QRuCRfjY8nW/3vr5hyG3CXpGcj4hSgQ9Lm/LpVwOz6RjYzs6IKbddIOhgRS4FFwD8DLwBExB8BM4B5+dSTgJ6KQ3uASbUE6uwcXcv0N+jqGnPExzZSq+aC1s3mXLVxrtq8mXIVKnkASTdFxK3AGrItmH8C5gPLJb2aT2sn27fv1waUagnU27uPUqlcfeIAXV1j2LPnpZqPa7RWzQWtm825auNctWnFXAd2PEf77p2UJp1Gx+lTazq2vb1tyMVx1ZKPiG5glKStkl6OiIfI9ucBLgc+WjF9NzCx4vIE4MWaEpuZvYkc2PEcu2//OuVDfbQNG86kRYtrLvqhFHkJ5RRgRUSMjIgTgJnAoxExjmz/fWf/REm7gFci4vx86Apgfd3Smpkl5oC2U+47CKUS5UN9HND2ut5+kROv64C1wFPAFmCTpPvJyn/3IIfMBe6IiO3AaODO+sU1M0tLR3TTNnwEtLfTNmw4HdFd19sveuJ1CbBkwNhPgfMGmfs0cG4dspmZJa/j9KlMWrT4iPfkqyl84tXMzBqj4/SpdJ13dkNOCPtjDczMEuaSNzNLmEvezCxhLnkzs4S55M3MEuaSNzNLmEvezCxhLnkzs4S55M3MEuaSNzNLmEvezCxhLnkzs4S55M3MEuaSNzNLmEvezCxhLnkzs4S55M3MEuaSNzNLWKGv/4uIm4FZQBlYKWlZRHwAuAMYAzwDfEbSaxExHbgbGAv8GLhGUl9D0puZ2ZCqruQj4mLgUmAacA6wMCL+GHgImC/prHzqZ/P/rgYWSDoDaAOurntqMzMrpGrJS9oIXJKvxseTrf6nAz+R9Ew+bSHwvYg4BeiQtDkfXwXMrntqMzMrpK1cLheaGBFLgUXAPwO/BM4CTgC6gceA64CzgdskXZAfMxVYl6/qqzkV2FljfjMzy5wGPD9wsNCePICkmyLiVmANWRl/DDgP+DWwErge+Heyfft+bUCplpS9vfsolYo98VTq6hrDnj0v1Xxco7VqLmjdbM5VG+eqTWq52tvb6Owcffjrq91ARHTnJ1OR9DLZXvz1wGZJOyUdAr4DnAvsBiZWHD4BeLHm1GZmVhdFXkI5BVgRESMj4gRgJjAfeF9EnJzPuQzYImkX8EpEnJ+PXwGsr3doMzMrpsiJ13XAWuApYAuwSdL/Aj4HrImI7cA7gFvyQ+YCd+Tjo4E7GxHczMyqK7QnL2kJsGTA2Fqy8h8492myrRszM2syv+PVzCxhLnkzs4S55M3MEuaSNzNLmEvezCxhLnkzs4S55M3MEuaSNzNLmEvezCxhLnkzs4S55M3MEuaSNzNLmEvezCxhLnkzs4S55M3MEuaSNzNLmEvezCxhLnkzs4S55M3MElboO14j4mZgFlAGVkpaFhHfBC4A9ufTlkr6XkR8GFgGdAAPSLqhAbnNzKyAqiUfERcDlwLTgBHAtohYC5wDXCSpp2JuB3APcDHwG2BtRHxc0vpGhDczs6FV3a6RtBG4RFIfMJ7sieEAMBm4JyKeiYilEdEOnAs8K2lnPn81MLtx8c3MbCiF9uQlHYyIpcA2YAPZiv5h4K+B84ALgc8CJwE9FYf2AJPqGdjMzIortCcPIOmmiLgVWAN8SNIn+q+LiH8ErgQeJNu379cGlGoJ1Nk5upbpb9DVNeaIj22kVs0FrZvNuWrjXLV5M+UqsiffDYyStFXSyxHxEDAnInolfTef1gYcBHYDEysOnwC8WEug3t59lErl6hMH6Ooaw549L9V8XKO1ai5o3WzOVRvnqk1qudrb24ZcHBdZyU8BlkbEBWSr9JnARuAfIuJhYB8wH7gXeByIiJgK7AQ+TXYi1szMmqDIidd1wFrgKWALsEnSzcAtwGNk+/RbJX1b0ivAVcB38/HtZFs4ZmbWBIX25CUtAZYMGFsOLB9k7gbgj+uQzczMjpLf8WpmljCXvJlZwlzyZmYJc8mbmSXMJW9mljCXvJlZwlzyZmYJc8mbmSXMJW9mljCXvJlZwlzyZmYJc8mbmSXMJW9mljCXvJlZwlzyZmYJc8mbmSXMJW9mljCXvJlZwlzyZmYJK/QdrxFxMzALKAMrJS2ruG4BMEvSjPzyZGA1MB4QMFfSvjrnNjOzAqqu5CPiYuBSYBpwDrAwIiK/7kzg+gGHLAeWS+oGngBurGtiMzMrrGrJS9oIXCKpj2x1PhzYHxEjgW8AX+6fGxEjgIuAB/OhVcDsOmc2M7OCCu3JSzoYEUuBbcAG4AXgFuAe4FcVU8cBe/MnBIAeYFL94pqZWS0K7ckDSLopIm4F1gDzgcmSvhQRMyqmtZPt21cq1RKos3N0LdPfoKtrzBEf20itmgtaN5tz1ca5avNmytVWLg/s5DeKiG5glKSt+eUvAO8DPgC8CowGJgBrgb8EeoG3SzoUEScDGyVNKZDlVGBnb+8+SqWhMw2mq2sMe/a8VPNxjdaquaB1szlXbZyrNqnlam9v618cnwY8//9dX+A2pgArImJkRJwAzAT+TdK7JU0H5gFPSJoj6SDwCDAnP/ZKYH3Nqc3MrC6KnHhdR7ZKfwrYAmySdP8Qh1wLzI+IbcCFwA31CGpmZrUrtCcvaQmw5DDX/QiYUXF5V+VlMzNrHr/j1cwsYS55M7OEueTNzBLmkjczS5hL3swsYS55M7OEueTNzBLmkjczS5hL3swsYS55M7OEueTNzBLmkjczS5hL3swsYS55M7OEueTNzBLmkjczS5hL3swsYS55M7OEueTNzBJW6DteI+JmYBZQBlZKWhYRnwcWAG1kX/S9WFI5IqYDdwNjgR8D10jqa0h6MzMbUtWVfERcDFwKTAPOARZGRABfAs4F3gt8EPhIfshqYIGkM8ieAK5uQG4zMyugaslL2ghckq/Gx5Ot/vcDZ0raD7wNOBH4fUScAnRI2pwfvgqY3YjglQ7seI7dDz7EgR3PNfquzMyOK4X25CUdjIilwDZgA/BCPnY18CugB9gKnJT/f78eYFJ9I7/RgR3Psfv2r7Prvm+z+/avu+jNzCoU2pMHkHRTRNwKrCHbgvknSSsi4pvAN4ElZHvz5YrD2oBSLYE6O0fXMp3dG3dSPtQHpRJl+mjfvZOu886u6TYaratrTLMjHFarZnOu2jhXbd5MuaqWfER0A6MkbZX0ckQ8BPxJRPxC0mOS+iLifuDzwDeAiRWHTwBerCVQb+8+SqVy9Ym50qTTaBs2nDJ9tA0bTmnSaezZ81Itd9lQXV1jWipPpVbN5ly1ca7apJarvb1tyMVxkZX8FGBpRFxAtkqfSfaqmfvyV9L8X7JX3jwqaVdEvBIR50t6DLgCWF9z6hp0nD6VSYsW0757J6VJp9Fx+tRG3p2Z2XGlyInXdWTbME8BW4BNkr4K3AJsAp4GXgb+Pj9kLnBHRGwHRgN3NiD3G3ScPpVJs/7cBW9mNkChPXlJS8j23CvHvkG2PTNw7tNkL600M7Mm8ztezcwS5pI3M0uYS97MLGEueTOzhBV+M9QxMAyy13weqaM5tpFaNRe0bjbnqo1z1SalXBXHDBvs+rZyufgbjxrsAuCRZocwMztOXQg8OnCwlUp+JPB+ss+7OdTkLGZmx4thZJ808DPg1YFXtlLJm5lZnfnEq5lZwlzyZmYJc8mbmSXMJW9mljCXvJlZwlzyZmYJc8mbmSWslT7WYFARMZbsy0kuk/R8RHwAuAMYAzwDfEbSa/m3VN0NjCX75qpr8q8mnAysBsYDAuZK2lfPXMCZwNcqrn4X8Liky5qZK/95fRS4jewNE08C8/Kf16D3HxFvA+4j+0awPcAnJf22AbmuAhaTvfHtYeC6oX4uDcx1E/DJ/OJaSYsj4sPAMqADeEDSDfncY/ZYDpYrHx8B/CvwFUk/aoVcETEf+Buyb457Avjcsf43eZhcnwcWkH3X9FpgsaRys3NVXLcAmCVpRn65Ib/7Lb2Sj4g/IXub7hn55bHAQ8B8SWfl0z6b/3c1sEDSGWQP6tX5+HJguaRusl/AG+udS9I6SdMlTQf+FNgLfLHZuXIrgb+Q9B7gLcCVVe7/q8Ajkt4NrAD+Z71zRUTk9/MhSe8FRpCVxLHO9WHgo8DZwHTgfRHxKeAesq+5fDfw/oj4eH7IMXksD5PrE/nP7UfABwcc0sxc/xX4L3mmaWSd8oUWyPVF4EtkX2D03jzfR1og1yfy684Erh9wSEN+91u65Ml++F/gD18G/hHgJ5KeyS8vBL4XEacAHZI25+OrgNn5quci4MHK8QbkqnQbcJekZ1sk1zBgbEQMA0YBB6rc/5+RrRoAvg18PJ9fz1zTyB7HnvzyD4DLm5Crh+wviNckHQR+SfZE9KyknZL6yAph9jF+LAfLNZlsQXMb8Hj/xBbINQq4VtJeSWXgfwOTWyBXCThT0n7gbcCJwO9bINfkiBhJ9q16X+6f2Mjf/ZberpE0DyBbwAAwFdgXEfcD3cBjwHVkz5Q9FYf2AJOAccDe/B9r5Xi9c5Ff/iNgBjAvHzqpBXJdS7b62wvsJPslGur+X8+c/wm7F+hi8Ce0I831NLAsIk7Ob3cWMKEJuX7R///5Y/dJ4B8Z/DE7Zo/lYXKdL+nZfOxvK6a3Uq4usu2Rq1ok18GIuBq4HfgpsBV4X7NzkX0/9j1k/x77Nex3v9VX8gMNBz4G/DeyB+utZH/ytJPtB/ZrI3smHzhOPt4o88n+3Or/kKCm5oqICcDfAe8h+wCjzWR7zUPd/8DPOm2jztkk/R+yx+37ZJ88+gzwWrNyRcRZwL+TbTv8imKPWcMfy8pc/UU6iJbIFRHvAjYAK/NzBS2RS9IKoBP4Ldn3VDc1F3AqMFnSNwdMa9jv/vFW8r8FNud/Sh8CvkO257abrMT6TSB7lvsdcGK+VUE+54hXfgVcDtxfcbnZuS4Efi5ph6QS2X7ejCr3/0Kek4gYTnaCu7eeoSJiFPBTSWdL+mB+nzuakSsizicrp+sl3cvhH7Nj+lgOkutwmp4rIrrJTqrfK+krrZArIk7Ox8hXx/eTbRM2++f1KeCsiNhKdvL3nIh4oMr9H9Xv/vFW8v9GdvLi5PzyZcAWSbuAV/ofVOAKYH2+D/YIMCcfvxJY34hgETGObK/v9T/BWiDXz4FzI+Kd+eWZwM+q3P86/nBydg7ZCZ+Ddc71VmBDRIyJiBPIzq08cKxz5b9H/wJ8WlL/k/Pj2VUxNf8H92myx+yYPZaHyTWoZueKiDFk/y5vkPT3rZKLbA/+voh4W0S0kW0JPtrsXJL+WtK78xdpzAOekDSnkb/7Lb0nP5Ck30TE54A1+WpwK7Aov3ousCJ/Bc6TwJ35+LXAvRFxA/BrsmfSRphCtkoYqGm5JP0yIm4EfhgRfcBzZFtKQ93/jcCqiPgF8Ps8f71z9UbEUrLtoxHAtyR9qwm5FpGdOFxWcb7gLrI95e/m163jDyfDjtVjOWguSXcdZn4zcz0AvBO4LiKuy8e+L+nLTc51F9ne9yagj6xA+5+EWvVxbMjvvj9P3swsYcfbdo2ZmdXAJW9mljCXvJlZwlzyZmYJc8mbmSXMJW9mljCXvJlZwlzyZmYJ+3+pAiziSgrM0QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_train = np.array([[2104],[1600],[2400]], dtype=np.float32)\n",
    "y_train = np.array([[399.900], [329.900], [369.000]], dtype=np.float32)\n",
    "\n",
    "plt.plot(x_train, y_train, 'r.')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T07:17:05.263887Z",
     "start_time": "2020-05-24T07:17:05.258498Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "x_train = torch.from_numpy(x_train)\n",
    "y_train = torch.from_numpy(y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-07T04:47:17.479477Z",
     "start_time": "2019-04-07T04:47:17.468259Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**nn.Linear**\n",
    "\n",
    "> help(nn.Linear)\n",
    "\n",
    "Applies a linear transformation to the incoming data: $y = xA^T + b$\n",
    "\n",
    "- **in_features**: size of each input sample\n",
    "- **out_features**: size of each output sample\n",
    "- **bias**: If set to False, the layer will not learn an additive bias. Default: ``True``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T07:17:06.464409Z",
     "start_time": "2020-05-24T07:17:06.458690Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Linear Regression Model\n",
    "class LinearRegression(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(LinearRegression, self).__init__()\n",
    "        self.linear = nn.Linear(1, 1)  # input and output is 1 dimension\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = self.linear(x)\n",
    "        return out\n",
    "\n",
    "model = LinearRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T07:17:07.149424Z",
     "start_time": "2020-05-24T07:17:07.146247Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Define Loss and Optimizatioin function\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=1e-9)#1e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-20T13:07:42.012941Z",
     "start_time": "2019-06-20T13:07:42.003484Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "> help(nn.MSELoss)\n",
    "\n",
    "To measures the **mean squared error** (squared L2 norm) between each element in the input `x` and target `y`.\n",
    "\n",
    "> help(optim.SGD)\n",
    "\n",
    "Implements **stochastic gradient descent** (optionally with momentum).\n",
    "\n",
    "Momentum is a variation on stochastic gradient descent that takes previous updates into account as well and generally leads to faster training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T07:17:08.856804Z",
     "start_time": "2020-05-24T07:17:08.642890Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch[50/1000], loss: 524703.812500\n",
      "Epoch[100/1000], loss: 224658.125000\n",
      "Epoch[150/1000], loss: 96851.820312\n",
      "Epoch[200/1000], loss: 42411.964844\n",
      "Epoch[250/1000], loss: 19222.966797\n",
      "Epoch[300/1000], loss: 9345.485352\n",
      "Epoch[350/1000], loss: 5138.111816\n",
      "Epoch[400/1000], loss: 3345.956299\n",
      "Epoch[450/1000], loss: 2582.575439\n",
      "Epoch[500/1000], loss: 2257.412354\n",
      "Epoch[550/1000], loss: 2118.905518\n",
      "Epoch[600/1000], loss: 2059.905518\n",
      "Epoch[650/1000], loss: 2034.777222\n",
      "Epoch[700/1000], loss: 2024.072266\n",
      "Epoch[750/1000], loss: 2019.512207\n",
      "Epoch[800/1000], loss: 2017.569946\n",
      "Epoch[850/1000], loss: 2016.742554\n",
      "Epoch[900/1000], loss: 2016.390259\n",
      "Epoch[950/1000], loss: 2016.241577\n",
      "Epoch[1000/1000], loss: 2016.177734\n"
     ]
    }
   ],
   "source": [
    "num_epochs = 1000\n",
    "for epoch in range(num_epochs):\n",
    "    inputs = Variable(x_train)\n",
    "    target = Variable(y_train) \n",
    "    # forward\n",
    "    out = model(inputs)\n",
    "    loss = criterion(out, target)\n",
    "    # backward\n",
    "    optimizer.zero_grad() # Clears the gradients of all optimized\n",
    "    loss.backward()\n",
    "    optimizer.step() # Performs a single optimization step.\n",
    "\n",
    "    if (epoch+1) % 50 == 0:\n",
    "        print('Epoch[{}/{}], loss: {:.6f}'\n",
    "              .format(epoch+1, num_epochs, loss.data.item()))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T07:17:10.284215Z",
     "start_time": "2020-05-24T07:17:10.002466Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAESCAYAAADTx4MfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3gUZdfH8W96CCmQAqGFIuQO0qtIFUSxgIUmgiJIURGVR3wVFRWsWB4bivIAUgQEESwoKEgv0olIuykCoZcECAmk7r5/zGZJIIHskuxskvO5Li+Z2ZmdX7KbPTvt3B5WqxUhhBDCUZ5mBxBCCFE0SQERQgjhFCkgQgghnCIFRAghhFOkgAghhHCKt9kBXMQPaAYcBzJNziKEEEWFF1AB2AikXvlgSSkgzYBVZocQQogiqg2w+sqZJaWAHAc4ezYZi8Xx+17CwgKJj08q8FA3yl1zgftmk1yOkVyOKW65PD09KFu2NNg+Q69UUgpIJoDFYnWqgGSt647cNRe4bzbJ5RjJ5ZhimivXQ/9yEl0IIYRTpIAIIYRwihQQIYQQTpECIoQQwikl5SS6EEKUKMPGriYxOe2q+cGlffn0mdYFsg3ZAxFCiGIot+JxrfnOkAIihBDCKVJAhBCimDmfdFXXkUIh50CEEKKYsFqtrPnnBLOX7nXJ9tyigCilPgLCtdb9lFL3A6MBD+AA0F9rfVYpFQVMB8oBGuijtXa/ngFCCGGCM+cuMfUPzY4DCdSqHMLeI+cLfZumH8JSSt0OPGb7dzDwFXCv1roBsA0YZVt0HDBOax0DbAJec31aIYRwLxarlT83Hea1SRvYd/Q8j9wZzUt9GhNc2jfX5fOa7wxT90CUUqHAO8C7QAPAB3haa33Utsg2oI9SygdoCzxgmz8FWAG85NLAQgjhRo6dSWbKwt3sO3qeujVC6dtJER5SCiDHpboREUGcPn2hwLdv9iGs8cCrQBUArXU88COAUqoUMAIYC4QDiVrrDNt6x4HKLk8rhBBuICPTwu/r4/hlzQH8fLwY2Lk2t9aJxMPDw6U5TCsgSqmBwGGt9RKlVL8rHgvBKCR/a62nKqUqAVe2krQ4us2wsEBn4xIREeT0uoXJXXOB+2aTXI6RXI4p7Fz7jpxj7OxY/j12nlYNKvLEg/UoG+RvSi4z90AeAioopWKBUCBQKfUJ8AHwB7AU+I9t2VNAiFLKS2udiTFC1jFHNxgfn+RUS+PC2v27Ue6aC9w3m+RyjORyTGHmSkvP5Jc1B/l9fRxBAT48/WA9mqgIMlLSOZ2SXii5PD09rvnF27QCorW+I+vftj2Q24AXgPXA91rrt7Mtm66UWoVRdGYCfYGFrswrhBBm2XP4HJMX7uZkwkXa1K9Azw41Ke3vY3Ys08+BXOk+oDHgrZTqbpu3SWs9EBgCTFVKjQTigIdNyiiEEC5xKTWDH1bsZ9mWo4SH+DO8V0PqVAs1O5adWxQQrfUUjCurII9Li7XWhzD2UoQQotjbtj+eaX/s5mxiKnc0rULXtjXw8/UyO1YOblFAhBBCGJIupfPdn3v5a8cJKoQF8PKjTahZKcTsWLmSAiKEEG7AarWySZ9mxiJNckoGXVpWo3PLavh4m36/d56kgAghhMnOJaXy7R+arXvPUDUyiOG9alOlnPO3HbiKFBAhhDCJ1Wpl9bbjzFq6j4xMCz3a38Sdzarg5em+ex3ZSQERQggTnDp3iakLd7Pr0Fmiq5Sh/90xlA8NMDuWQ6SACCGEC1ksVv7cfIR5K/fj6eHBo50U7RpWxNPFbUgKghQQIYRwkaNnkpmyYBf7jyVS/6Yw+nZShAZfvw2Ju5ICIoQQhSwj08KCdYf4de1B/H29GdTlZlrcXN7lzQ8LmhQQIYQoRAeOJzJ5wW6OnE6iee1y9O4YXaBjcphJCogQQhSC1PRMfl59gD82xBFS2pdnutWjUa0Is2MVKCkgQghRwHTcWSYv3M2ps5do26AiPdvXJMC/+H3cFr+fSAghTHIxJZ1pf2iWbz1KRBl//q9XQ2q7UfPDgiYFRAghCsDf+84wY/Ee4hNTuLNZFR5sWwM/H/dqfljQpIAIIcQNuHAxje+W7GXdjpNERQbxxP11uKmiezY/LGhSQIQQwglWq5UNu04xY/EeLqVmcH/r6jzWpS7nziabHc1lpIAIIYSDzl4wmh/G7jtD9QrB9L8nhsoRgW7dObcwSAERQoh8slqtrPz7GN8v20dmppWHOtTkjqZV8PQs2jcEOksKiBBC5MOpsxeZsnA3u+POERNVhn53x1CubNFqfljQpIAIIcQ1WCxWFm86zI8r/8XLy4PH7lK0bVCxyLchKQhuUUCUUh8B4VrrfkqphsBEIBhYCTyptc5QSkUB04FygAb6aK2TTAsthCj2jpxOYvKC3Rw4nkjDmuE82klRNsjP7Fhuw/QzPkqp24HHss2aDgzVWkcDHsAg2/xxwDitdQywCXjNpUGFKGES161l08An2DOwH/++OJzEdWvNjuQyGZkWfl59gNGTN3L63CWeuK8Oz3SrJ8XjCqbugSilQoF3gHeBBkqpqkAprfU62yJTgNFKqYlAW+CBbPNXAC+5NLAQJUTiurWcnDYFa1oaABkJ8ZycNgWA4BYtTUxW+P49lsjkhbs4ejqZFnXK8/DttQgKKB7NDwua2YewxgOvAlVs0xWB49kePw5UBsKBRK11xhXzhRCF4My8ufbikcWalsaZeXOLbQFJTc/kx5X/snjTYcoE+vFc9/o0qBludiy3ZloBUUoNBA5rrZcopfrZZnsC1myLeQCWXOZjm++QsDDnB6mPiAhyet3C5K65wH2zSa7r23M2Idf5GWcT3CZnQebYtu80Y7+P5UT8Re6+tRr9Ot9MgL+P6bkKUmHkMnMP5CGgglIqFggFAjGKRIVsy0QCx4BTQIhSyktrnWlb5pijG4yPT8JiubIOXV9ERBCnT19weL3C5q65wH2zSa788S4bSkZCfK7z3SFnQf2+LqZk8P2yfaz8+xjlypbipd6NUFFlSb6QQvKFFNNyFTRnc3l6elzzi7dpJ9G11ndoretqrRsCrwO/aK37AylKqVa2xR4FFmqt04FVGEUHoC+w0OWhhSghwrt2w8M353F/D19fwrt2MylRwYvde4aRE9exatsx7rolitGPN0dFlTU7VpFi9jmQ3PQBJiilgoEtwOe2+UOAqUqpkUAc8LBJ+YQo9rLOc5z9eR6pp8/gHRpGeNduxeL8R2JyGjP/3MOGXaeoHFGaZ7rVp3qFYLNjFUluUUC01lMwrqxCa/030DyXZQ4Bt7kylxAlWXCLltzUpZNbHpJxhtVqZd3Ok3z3515S0jJ4sE117m5RFW8v0+9mKLLcooAIIURhSkhMYdofmm3747mpYjD97qlNpfDSZscq8qSACCGKLYvVyorYY8xZtg+L1crDt9fi9iaVS2zzw4ImBUQIUSydTDCaH+rD56hdtSyP3R1DuTKlzI5VrEgBEUIUK5kWC4s2HuanVQfw9vKk/90xtK5fQZofFgIpIEKIYiPu5AUmL9zNoRMXaFQrnEfulOaHhUkKiBCiyEvPsDB/7UEWrjtEaX9vnnqgLk1VhOx1FDIpIEKIIm3f0fNMXrCL4/EXaVk3kl631yKwlHNtSIRjpIAIIYqk1LRM5q7cz5JNRygb7MewHg2of1OY2bFKFCkgQogiJ3bPKT6btZUz51Po0LgS3drdRCk/+ThzNfmNCyGKjIsp6cxauo/V245TvmwpRvRpTHSVMmbHKrGkgAghioQte07z7SLNheR0uneoRcdGFfH18TI7VokmBUQI4dbOJ6cxY/EeNu0+RVS5QIZ1b0DTehWLTY+uokwKiBDCLVmtVtZuP8GsJXtJTc+ka9sa3HVLlDQ/dCNSQIQQbif+fApT/9jN9n8TqFkphP73xFAhTJofuhspIEIIt2GxWlm25Sg/rNgPVujdsRYdmlTGU24IdEtSQIQQbuF4fDJTFu5m75Hz1KlWlsfuiiFcmh+6NSkgQghTZWRa+GNDHD+vPoifjycD7q1Ny7qR0oakCJACIoQwzaETF5i8cBdxJ5NooiJ45I5oQgKl+WFRIQVECOFy6RmZ/LLmIAvXxREY4MOQB+rSNKac2bGEg0wvIEqpN4HugBWYpLX+WCl1J/Ah4AVsAQZqrdOUUlHAdKAcoIE+Wuskk6ILIZyw98g5Ji/YzYmEi7SqF8lDHaT5YVFl6gXVSql2QAegPtAUeEYppYBJQC+tdV0gAOhrW2UcME5rHQNsAl5zfWohhDNS0jKYsXgPY6ZvIT3DwvMPNWDAvTdL8SjCTN0D0VqvUEq111pnKKUq2fIkY+x5BCulvAB/4JJSygdoCzxgW30KsAJ4yfXJhRCO2H4gnqkLNQmJKXRoUplu7Wrg72v6ARBxg0x/BbXW6Uqp0cALwBzgKDAEWA4kAgeAH4BwIFFrnWFb9ThQ2eWBhRD5lnQpndlL97LmnxNEhgYw4pHG1KoszQ+LCw+r1Wp2BgCUUgHAfIy9it7AvRjF42PAB3gXWKe1rmJb3htI0lr75+Ppq9meSwjhImu2HePredtITE6je4daPNQxWpofFl3VgYNXzjR1D0QpFQP4a61jtdYXlVLzMPY+tmut99uWmQB8DwwDQpRSXlrrTKACcMyR7cXHJ2GxOF4wIyKC3LJxm7vmAvfNJrkc40yuc0mpzFi0h817ThNVPpBh3esTVT6I8+cumprLFYpbLk9PD8LCAvN83OxDWDWA0Uqp1hhXYd2PcZXVU0qp8lrrk7Z5G22HulYBDwEzMU6sLzQptxDiClarlTX/GM0P0zIsdL/tJjo1r4KXpzQ/LK7MPom+QCnVHNgKZAJztdbvKaWOAcuUUhnAPmCwbZUhwFSl1EggDnjYjNxCiJzOnLvE1N93s+PgWWpVDqHf3dL8sCQwew8ErfUoYNQV86YCU3NZ9hBwmytyCSGuz2K1snTzEeau+Bc84JE7o7mtUSVpflhCmF5AhBBF07EzRvPDfUfPU7dGKH07KcJDpPlhSSIFRAjhkIxMCwvXxzF/zQH8fLwY2Lk2t9aR5oclkRQQIUS+HTpxgW8W7OLwqSSaxZSj9x3RhJT2NTuWMIkUECHEVYaNXU1iclquj4WU9mVo13o0jo5wcSrhbqSACCGuklfxAHh70C2U9pf+VcLkZopCiKJHiofIIgVECCGEU+QQlhDC7kTCRV753zqzY4gCkrhuLWfmzWXP2QS8y4YS3rUbwS1aFtjzSwERQmC1Wvnqp+1s0qfNjiIKSOK6tZycNgVrmnE+KyMhnpPTpgAUWBGRQ1hClHAHTyQy4P1l9uIxqPPNBOdxaW5e84X7OTNvrr14ZLGmpXFm3twC24bsgQhRQlmsVsZM38K+o+cBCA7w4cMhrfDx9uTWupGA+3aXFdeXkRDv0HxnSAERogTaeTCBj2bF2qeH9WhA/ZvCTEwkCpp3aFiuxcI7tOBeZykgQpQgGZkWXh6/jvjEFACiygXyer9meHpKG5LiJrxrtxznQAA8fH0J79qtwLYhBUSIEmLj7lN89dN2+/QrjzahZqUQExOJwpR1ovzMvLlkyFVYQghnpKZlMvTTlWTaRuOsf1MYz3WvL80PS4DgFi0JbtGy0M5lSQERohhbtuUI3y7aY59+a+AtVAqXgZ5EwZACIkQxlHQpnWc/W2WfbtugAv3urm1iIlEcSQERopj5efUBfl59wD794VMtCQvxNzGRKK6kgAhRTCQkpvDCuLX26S4tq/Fg2xomJhLFnekFRCn1JtAdsAKTtNYfK6VuBT4BgoBtwGNa6zSlVENgIhAMrASe1FpnmBRdCLfx7R+aZVuP2qc/e7Y1QQFy17goXKa2MlFKtQM6APWBpsAzSqkGwDxgsNa6jm3RAbb/TweGaq2jAQ9gkIsjC+FWjp1J5vExS+3Fo88d0XwzooMUD+ESphYQrfUKoL1tL6Icxh5RQ+AvrfU222LPAD8qpaoCpbTWWa1CpwA9XBxZCLdgtVoZO3cbIyeuB4xvU+Oeb8vtTSqbG0yUKKYfwtJapyulRgMvAHOASCBJKTULiAHWAMOBRsDxbKseB+SvRZQ4/x5L5O1pm+zTT9xXh1tuLm9iIlFSeVitVrMzAKCUCgDmY5zbGAK0AOKAScBBYDEwRmvdxrZ8LWC+1jomH09fDThwvYWEcGeZFisvfLaCfUeM5ofhIf7875U78PGWptqi0FXH+BzOwdQ9EKVUDOCvtY7VWl9USs0DPgIWaa0P2Jb5HhgKTAYqZFs9EjjmyPbi45OwWBwvmO7akdRdc4H7Ziuqubb/G8/H3/9tn37+oQbUrR7GubPJpuYyi+RyjLO5PD09CAsLzPvxGwlVAGoAE5RSfkopX+B+YDDQRClVxbZMZ2Cz1voQkKKUamWb/yiw0OWJhXChjEwL//litb14VK8QzMSX2lO3unTOFeYzdQ9Ea71AKdUc2ApkAnO11t8qpRKA+UopfyAW4/wIQB+MghMMbAE+NyO3EK6wfudJxv+ywz49sm9TalQMNjGREDm5w0n0UcCoK+b9BvyWy7J/A81dEkwIk6SkZTDk45X26cbRETz9YF1pfijcjkMFRCk1DZgGLNFau8fZdyGKkSWbjzBj8eXmh+8MuoUKYdL8ULgnR/dAHsE4jHRCKTUDmJ7tfg0hhJPOJ6Xy+Jil9un2jSvx6J3KxERCXJ+jBaQaRhHpjXFeYrhSajswFfhOa338GusKIXIxb+V+fl17yD790ZCWhAZL80Ph/hwqIFrrOOBd4F1bX6pHgF4Yl96+r5RainGI60et9cWCDitEcXLm/CVe/Oov+/QDratzX+vqJiYSwjFOn0TXWscCsUqp/wNaY1xu2wujgFxUSs0FJmit1xRIUiGKkckLdrFq2+Ud9plv3U1KcqqJiYRwXEFchVUDaIVRRCphtOWJA3oCjyqllgB9tNanC2BbQhRpR08n8dqkDfbpRzsp2jeqRFCArxQQUeQ4VUCUUpHAQxjnQppiFI1jwMfAVK31Dtu9GiNs/00D7i6QxEIUQVarlU/nbOOff+MB8PbyZOxzbfDz9TI5mRDOc/Qy3scxikY7wAu4BMzCOIn+p9bakrWs1joReEUp1RNoU2CJhShi9h05z7vTN9unhzxQl6Yx5UxMJETBcHQPZCLGwE+rMIrGHK110nXWOQb840Q2IYo0i8XKqMkbOXLa+BMJD/Hn3cEt8PYyu4OQEAXD0QLyOvCtrS9Vvmit2zq4DSGKvG37z/DpnMu3SP1fr4bUrhZqYiIhCp6jl/G+XVhBhCgO0jMsDP9yDUmX0gGoVTmEl/o0xlPakIhiyPReWEIUF2u3H2fir7vs06/3a0q1SGl+KIovKSBC3KBLqRk8/cnl5ofNYsrx5P11pPmhKPakgAhxA/7YEMfspfvs0+8NbkH50AATEwnhOlJAhHDC+eQ0/jN2tX26Y9PK9O4YbWIiIVxPCogQDpqzbB8L18fZp//7dCvKBvmZmEgIc0gBESKfTp+7xEtfX25+2K1dDe69tZp5gYQwmRQQIfJh4q87Wbv9hH36i2FtCPD3MTGREOaTAiLENRw+lcQb31xuftjv7hjaNqhoYiIh3IfpBUQp9SbQHaNFyiSt9cfZHhsKdNda32abjgKmA+UAjdHl93qtVIRwmNVq5aNZsew6dBYAP18vPnumNb4+0vxQiCymNuVRSrUDOgD1Mbr6PqOUUrbHbsbo5JvdOGCc1joG2AS85sK4ooTYc/gcA95fZi8eQ7vW46vn20nxEOIKphYQrfUKoL3WOgNjr8IbSFZK+QHjMXpvAaCU8gHaAj/YZk0Berg0sCjWMi0WXp2wjjEztgAQGRrAhBdvo3F0hMnJhHBPph/C0lqnK6VGY4yxPgc4CvwX+AY4kG3RcCDRVmwAjgOVXZlVFF9b955m7NzLTaNf6t0IFVXWxERCuD/TCwiA1voNpdT7wHxgMBCltX5eKXVbtsU8Mc6TZGfBAWFhgU5njIgIcnrdwuSuucB9s2XPlZqeyWOjfic5xfheUr9mOG8/2dKUNiRF4fflTiSXYwojl4fVeuVnsusopWIAf9v46iilngaaALcCqUAgEAn8BjwCxANltdaZSqkqwAqtdY18bKoacCA+PgmLxfGfNyIiiNOnLzi8XmFz11zgvtmy51q17RiTF+y2PzaqfzOiypvzx18Ufl/uRHI5xtlcnp4eWV+8qwMHr3zc7D2QGsBopVRrjL2L+4FvtNaPA9j2QEZprR+yTa/CGEp3JtAXWGhGaFG0XUxJZ+inq+zTLW4uz+D76piYSIiiydQCorVeoJRqDmwFMoG5WutZ11hlCDBVKTUSiAMedkFMUYzMXbqXKb/ttE+PeaIF5cpK80MhnGH2Hgha61HAqDweWw7clm36UPZpIfLrXFIqz3+xxj7dqXkVHupQy8REQhR9phcQIQrbrCV7WbTxsH36k6GtCAmU5odC3CgpIKLYOnn2Ii+PX2ef7tH+Jvp2ruuWJzmFKIqkgIhiafwvO1i/86R9+othbQnwl7e7EAVJ/qJEsXLoxAVGT9lon378ntq0rl/BxERCFF9SQESxYLFa+WDmVvYcPgdAaX9vPh7aCh9v6V8lRGGRAiKKvN2HzvLBd1vt0892r0/DmuEmJhKiZJACcg2J69ZyZt5c9pxNwLtsKOFduxHcoqXZsYRNRqaFkRPXc+rsJQAqRZRmVP9meHma2iNUiBJDCkgeEtet5eS0KVjT0gDISIjn5LQpAFJE3MBmfZovf7zc/HBEn8ZEVyljYiIhSh4pIHk4M2+uvXhksaalcWbeXCkgJkpNz+S5z1aRlmH00axTPZTnezYwpfmhECWdFJA8ZCTEOzRfFL7lsUeZ9ru2T7/5eHMql3O+w7IQ4sZIAcmDd2hYrsXCOzTMhDQlW3JKOs9ka37Yqm4kAzrfXOjbzchIJzk5kdTUS1gsmYW6rVOnPLFYHBqdwCUkl2PcPZenpxd+fqUoXToYb2+fG35eKSB5CO/aLcc5EAAPX1/Cu3YzMVXJM3/tQX5c+a99+oMnbyW8TKlC325GRjoJCScJCAgiNDQSLy+vQj1M5u3tSUaG+33wSC7HuHOu9PRMMjMzSUlJJiHhJKGh5W+4iEgByUPWeY4z8+aSIVdhudzZC6kM//Jy88N7b61Kt3Y3uWz7ycmJBAQEERgY4rJtClGYPDw88Pb2tr+nk5MTCQm5sSMqUkCuIbhFS4JbtHTbQWKKqxmL97Bk8xH79KfPtCa4tK9LM6SmXiI0NNKl2xTCVfz9S5OQcOKGn0cKiHAbx+OTeXXCevt0r9trcWezKqZksVgy8fKSu9hF8eTl5VUg5/WkgAjTWa1Wxv20nc36tH3el/9pSyk/c9+ecmmwKK4K6r0tBUSY6sDxRN6ausk+PajzzdxaVw4dCVEUSAERprBYrbw3fTP7jyYCEFzalw+faomPt7QhEaKokAIiXG7HwQT+OyvWPv2fng2oV0PurxGiqDG9gCil3gS6A1Zgktb6Y6XUYOBZ27xNwBNa6zSlVENgIhAMrASe1FpnmBRdOCgj08LL49cRn5gCQFT5QF5/rBmennKuwd0dPHiAOXO+Y/PmjZw+fYpSpUpRsWJlOnToSOfODxAYmP+OAAsWzOfdd0fz7LPP07Nnb4ezDB06mNjYLSxcuIygoCCH18+v48eP0aPHfbRp04733vuvU8/x2Wf/Zc6c7/j8869p3LipU8+xYcM6goKCqF27jlPrFyZTjxcopdoBHYD6QFPgGaWUAv4PaGmb7wk8bVtlOjBUax0NeACDXB5aOGXDrpMM/nC5vXi8+mgTRvVvLsWjCJg581see6wXv/76M5UqVaFbt57cdltHUlNT+eKLT+nTpzvbt/9z/SeyqVUrmv79B3HzzfWcynPPPV0YMGAwvr6uvbTbDD/++APPPz+U06dPX39hE5i6B6K1XqGUaq+1zlBKVbLlSQGGaK0TAZRS/wBRSqmqQCmtddYg11OA0cBXJkQX+ZSalsnTn6zEYrUCUP+mMJ7rXl+ucCoifvhhNuPGfUa1ajV4660xVK9eI8fjixf/znvvvcWzzz7JxInTqFHj+jd71qqlqFVLOZ3pnnu6uO0d3wUtwc1775l+xlJrna6UGg3sBJYAcVrrxQBKqQhgKPAzUBE4nm3V40BlF8cVDli25QhPfbzCXjzeGngLw3pI59yi4syZM3z++SeEhITw6adfXlU8AO644y5eeulV0tJSee+90SakFGYy/RwIgNb6DaXU+8B8jMNS/7PtkSzEOC+yXCnVCuOcSBYPwKGvIGFhzndujYgovGOtN8IdcyUmp9Fl+M/26U4tqjK0R0MTE+WUn9/ZqVOeeLvgirBza9dyau4cMuLj8Q4Lo1y3HpRp6R7tcubPn0daWhqPPPIYkZHl81zu3ns7891337Jr10727dtNTMzNHDt2jK5dO/P444O4cOEC8+f/hJ+fPy+++DKXLl3i7bdHMWzYcHr16mN/nk2bNjBp0gT27NH4+Hhz220d6NGjF3369GTAgMEMGvQkAE89NYitWzezePEKgoKC2Lx5E08/PZiRI0dhtVqYNWsmhw/HERJShjvu6MTgwU/i73+5f1pGRjpz585h0aLfOXjwIKmpKYSHh3Prra0YPHgIZcuWBcDLy3j9jRYg138vzJ//E7Nnf8fhw4eJiIigT59HyRrbzMsr5/tp9eqV/PDD9+zevZMLF5IICgqkfv2GDBz4BNHRKsfPCfDKKy8AsG7dFgAuXrzIrFkzWLZsCUePHiEjI4Ny5crTrl17Bg58glKlru4Xd+XP4OnpecOfH6YWEKVUDOCvtY7VWl9USs0D6tvm/wF8rrXOOnt1BKiQbfVI4Jgj24uPT8JisV5/wSu4aysTd8z18+oD/Lz6gH36w6daEhbi7zY58/s7s1gshX6I5KpBy+LjOT7lGywWi1v0XNuwwegK0Lz5rdf9XbRpcxv79u1l2bJl1KwZQ2amsfxPP80D4IEHuhMXd5CYmDps3rwRAIvFan/eFSuW8vrrL1OqVADt2rXH39+fP/9cZM+QfVmrbY82I8N4jbK2NWfObPbv30u7dh245ZZbWbFiGTNnfto7PH8AAB9hSURBVEtiYiIjRrxmzzpy5MssX76U+vUbct99D5KWlsqGDev48ce57Nq1i4kTpwHYn9dqtV73558w4SumTp1EZGQFOne+j/j4eD766H3KlCljf66s55g7dzaffPIhlSpVpmPHTvj4+LJr1w5WrlzOpk0bmTlzLuHh4dx9d2esViuxsVu4/fY7iIqqZvuZMxg69El27dpB8+a30qxZCy5evMiaNSuZMWMaR48e4e23P8iRL7dDfhaL5bp/C56eHtf84m32HkgNYLRSqjXG3sX9wLfAIuBVrfW3WQtqrQ8ppVKUUq201muARzH2UIQbSEhM4YVxa+3Tve5Q3NmkkomJ3J+7D1p2+PAhAKKiql532WrVqgNw7NjRHPPPnk3gm29mUKtWdJ7rpqSk8N//vk+pUgFMmDCVKlWiAOjd+zEGDOiT53pX2rdvD19+OYG6desD0LfvAHr1epBFi37nuedeoFSpUmzf/g/Lly/lzjvv5vXX37Kvm5GRwYABj7B7907i4g7l62fOcvhwHNOnT6FWrWjGjZtAqVKlAVizZhUjRjyfY9m0tDT+979xVKkSxTffzMixp/DRR2P46acfWLNmJfff35V77unC8ePHbAWkE23b3gbA8uVL2LlzO337Ps7gwUPs6w8Z8gy9enVl1aoVpKSk4O/vn++fwVmmngPRWi8AfgO2ApuBtUA4UB4YrpSKtf33pm2VPsAnSqndQCDwuQmxxRWm/b47R/H47NnW9LkrxsRERYO7D1qWlJQEQEBA6esuGxwcDMD58+dyzK9cuco1iwfA+vVrSUiIp3v3h+zFAyAyMpKHHsp/AWnYsLG9eAAEBgZSr1590tJSOXXqJADlypXj1VdHMWDAEznW9fb2pl494zDr2bNn871NgGXL/iQzM5O+fR/PcVlxq1ZtaN781hzLWiwWXnxxJCNGvHbVYaZGjZrYtp9wze1FR8cwYsRIHnoo5yXQAQGliY5WZGZmkph43qGfwVlm74GgtR4FjLpi9id5LPs30LyQI4l8OnYmmZETLzc/7HNHNLc3kesa8svdBy0LCgrm7NkE0tJS8fa+9kfFpUvG5dllypTNMb9ChYrX3c6uXTsBcr3PoV69BvmNm6P4ZCld2jj8kmbb0ytXrjx3392ZjIwMtN5NXNxBjh49wt69mk2bNgA43GRw3769AMTEXD3IWb169Vm//vKXK39/f26//Q4A4uIOcfDgAY4ePcKBA/uzbf/ah8uioqoSFVWV1NRUduzYzuHDhzh69Aha77KfM3HVoFamFxBR9FitVsbO/YfYfWcA42qGL59vi7+vvJ0c4e6DllWqVJmzZxOIi4sjJqb2NZc9eNAY9Kt8+Zx9zPz8/K67nay9lrCwqwtneHhEfuPi43P1fSGXr/i7fO7zp5/mMmXKRM6cMe6tCAwMok6delStWp2dO7fbz7Hk14ULRjuegICAqx4LCgq+al5s7BY+//xj9uzZDYCvrx81a9ZCqdqcOnXyutu3WCx8++1kZs2aYd922bKh1K1bnwoVKnLw4AGHfwZnyV+8cMj+Y+d5Z9pm+/ST99ehee28r9ARecsxaFlCPN6hYW41aFmbNu3Yvn0bq1Ytv24BWb16JYD9OL0jsg6RJScnX/VYbvNuxNKlf/LRR+9x0021GD78JaKjY+xF76OP3mPnzu0OP2dWkUhKSiIsLDTHY5cuXcwxfeLEcV544Vl8ff148cVXqV+/IVWqROHl5cWSJYtYtWr5dbc3a9Z0Jkz4ikaNmtCnz2PUqhVNWFg4AMOHP8vBgweu8wwFRwqIyBeLxcpb0zZx6IRx1UZosB9jnrgVby/TbyUq0rIGLXPHG+Puuutevv12MnPnzqZLlweIjKyQ63IrVixl587tREcrp9ptKGUUp507d9CkSbMcjznzgX4tixf/DsAbb7x91U2Pzn7wKhXD0qWL+eefv6laNedhtN27d+WYXrlyOSkpKTz11LPcd9+D191+bvdMLV78O15eXowZ81/7ITowjgzExR20/9sV5K9fXNf2f+MZ+MEye/EY/lBDPhrSSopHMRcWFs6wYcNJSkpi2LAh/Pvv/quWWbFiGW+//Qa+vr68+qpzNxK2adOO4OAQfvhhVo6ruE6dOsnMmdOczp+brPYnV56oXrjwV2JjjXssMjIca693++134uvrx5Qpk4iPP2Of//ffW1m9ekW+tr9v317mzJl11fazzj2lp6dnew4/MjMzOXcu5wULU6dO4vjxY079DM6SPRCRp/QMCy9+tZbzycYx+hoVg3nl0SZ4yp3kJUbnzveTnp7Jf/87hv79e9Os2S3cdFMt0tPT+PvvWLTeRVhYGG+88Q433VTTqW2UKlWK559/kdGjRzJgwKO0a9ceT09PVq5cZl+moEaH7NTpHpYsWcQrr7xAx46dKF26NDt37iA2dgtly4Zy9mzCVVeSXU9kZAWefvo5PvnkA/r27U3btu1JTk5i2bI/KV8+kqNHLw/P3KpVG77++gu+/XYyhw4dpFKlyhw5EsfatavtexPZtx8RUQ6AadMmsXevpn//Qdx5593s2PEPTz01gA4dOuLt7cOWLZvYs2e3/WcoMVdhCfe0bscJ/jd/p316ZN+m1Kh49QlBUfx16fIAjRs3Ze7c2WzatIG//47Fx8eHSpUqM2TIc3TufB/BwSE3tI2OHTvh71+KadO+YfHi321XK91JgwaNeeONl/HzK5h7Glq2bM3o0e8yY8ZUFi1aiJ+fPxUrVuL551+ibt16PP74I6xbt4Y77rjLoeft1q0nERHlmD59CgsW/EJISBkGDHgCHx8fxo69fFFpREQ5Pv10HOPHf8HmzRvZsOEvIiMr0L37QzzySH8efvhB1q//C6vVioeHBx063MFff61h7dpV/PjjHO65pzNdu/YArPz441zmz/+JwMAgqlSJYtSod/Dz8+Pll1/gr7/W5LikubB4uOpYmcmqAQfkTvTru5SawdOfrLRPN46O4OkH6zrcv6qo/85OnDhEZGT+bya7Ue54DgRckys5OYmLFy8SHh5x1fvst99+4b333mT06Pfsl7+6KpczilKu/LzHs92JXh04eNXjBRdRFHWLNx3OUTzeGXQLQ7vWk+aHolAdPhzHgw/ew3vvvZljfmpqCvPmzcHLy4sGDdynl5q4TA5hCRIvpjHs89X26faNK/Honc632xbCEdHRMdSuXYcFC+Zz/PgxateuQ2pqCmvXrub48WMMHjzEoftBhOtIASnh5q3cz69rD9mnPxrSktDgwu+hI0QWT09PPvnkS2bPnsGyZX8yb973eHv7ULNmLYYMeZb27TuaHVHkQQpICXXm/CVe/Oov+/QDbapzX6vqJiYSJVlgYCADBjxxVY8q4d6kgJRA3yzYxeptl8fm+vy5NgSW8jExkRCiKJICUoIcOZ3E65M22Kf7dlLc1khargshnCMFpASwWq18Mudvtv9r3P3q7eXJ2Ofa4OdbMDdnCSFKJikgxdy+I+d5d/rl5odDHqhL05hyJiYSQhQXUkCKKYvFyqjJGzhy2uhmGlHGn3cGtZD+VUKIAiMFpBj6e98ZPvthm336/3o1pHa10GusIYQQjpMCUoykZ2Qy/Mu1JF0yOnfWqhzCS30aS/NDIUShkAJSTKz55ziTfrs89sDr/ZpSLVKaHwohCo/pBUQp9SbQHWPMyUla64+VUh2Bj4FSwGyt9Ujbsg2BiUAwsBJ4Umvtmsb3burK5ofNa5fjifvqSP8qIUShM/WMqlKqHdABqA80BZ5RSjUAvgHuB2oDzZRSd9tWmQ4M1VpHYwzFPcj1qd3HTyv25Sge7w1uwZP3O945V4jcLFgwnxYtGtO6ddNr/rdy5XL78q1bN+X772fmeJ7Fi3/PMSYGwJkzZ/jtt19yzOvevQt33XVbYf5I1zRp0nhat27KggXz871s1s9eUpm6B6K1XqGUaq+1zlBKVbLlKQPs1VofAFBKTQd6KKV2AqW01utsq08BRgNfmRDdVOeT0/jP2MvNDzs2rUzvjtEmJhLFWcOGjWnUqEmej1etWg2AWrWi6d9/EDffXM/+2LhxnzNz5jQmT55hn3f2bAK9e3ejSZOm3Hvvffb5PXs+TGpqWsH/AIUg6/eR9bOXVKYfwtJapyulRgMvAHOAisDxbIscBypfY36+2fraOyUiIsjpdQvSlF93MHfZPvv01Dc6uW3zQ3f5nV0pP7lOnfLE29u1O+iu3t71eHoae7JNmjRl0KAnr7t87dq1qV27do55586dBcDL6/LvMz09jYsXk/Hw8MjxM/fu/YhD+Qr695X183p6elz3uZs3b07z5s1dkqugXJnL09Pzhv9GTS8gAFrrN5RS7wPzgWiM8yFZPAALxuG23ObnW1EeUOrUuUuM+Ppy88Nu7WrQ7756nD59gdOn06+xpjnc4XeWm/zmslgsLh0YyB0HIsr6W7FYrE5nyxqwLjPz8u8zM9Nif8zZ5y2M31dB/Lzu+DpC7rksFst1/xayDSiV+/MWSDonKaViAH+tdazW+qJSah7GCfXMbItFAseAI0CFXOYXexPm7+SvHSfs018Ma0OAvzQ/LOqGjV1NYvLVh2yCS/vy6TOtTUh0YxYsmM+7747m2Wefp2fP3nTv3oUTJ4yDBv379yEysgKPPz6Yd98dDcCqVSto3bopr7zyBvfc04Xu3buQlHSB339fnuP5Pv10HHv2aH75ZR6nTp0kPDyCLl3up3fvx3KMlZ6RkcF3333Lb7/N59Spk1SsWJFevR7hzJnTTJz4NXPm/EKFChUL5GedNGk8kydP4N13P6Jt29sAaN26Kffc04UuXR5g/Pgv0XoXXl7eNG/egqeeeuaqbR85cphvvvkfGzeuJynpAhUrVuKuu+7l4YcfxdvbLb7bX5fZKWsAo5VSrTH2Lu4HxgMfKqVqAgeA3sA3WutDSqkUpVQrrfUa4FFgoVnBXSHu5AVGTd5on+53dwxtGxTMH4AwX27F41rzi5qePR9mwYJf2bdvD/ff35WqVatRq1Y0PXo8zJw53xEVVZXbb7+TWrWuff7uq6/GEhd3kPbtOxIUFMSff/7B+PHjyMy00K/fQPtyr7/+MitXLuOmm2rx4IPdOXbsCGPGvEXFiq5rGKr1LhYv/p369Rvy4IPd2blzB0uXLmbPHs3MmT/g6elpW243zz33JKmpqbRt257IyAps27aV8eO/JDZ2Kx988EmO4uiuzD6JvkAp1RzYirHXMVdrPUspdRqYC/gDC4AfbKv0ASYopYKBLcDnJsQudFarlY9mxbLrkHH82N/Xi0+faY2vj/u/oUqCNf8cz9EOvzC8P2PLDa3fun4FWtWrcP0F82Hr1s1MmjQ+18fuuadLnt/qe/bszd69e9i3bw8PPNCNWrWUbX4Qc+Z8R9Wq1fI1/sfRo4eZPHkmlStXAaB79148/HBXfvnlR3sBWb58CStXLqNNm9t4660x9m/wc+d+zyeffODwz+ys/fv3MWTIs/Tu3Rcw/paHD3+GDRvWsXXrZpo0aYbVauWdd94gLS2dr776hpiYy+eNxo79mNmzZ/Lzz/Po2rWHy3I7y+w9ELTWo4BRV8xbAjTIZdm/gdzPXBUTOu4s78/cap8e2rUejaNlOE9hntjYLcTG5l7QGjVqUmCHhfLSrl0He/EAqFChItWr12Dv3j2kpqbi5+fHwoW/AjB06LAch38efLA7c+fOJi7u0FXPWxj8/Pzp0eNh+7SHhwctWrRkw4Z1HD58iCZNmrFjx3b+/Xc/Xbv2yFE8AAYOfIp58+awYMF8KSAi/zItFl6buIETCRcBiAwN4K2BzfHydM8rOkqyVvUK5tv942OW5vnYS30a3/DzF5T+/QeZOlJglSpRV80LDDRO7Kanp+Pn58fu3TsJCQmhUqWcF2Z6enpSt259lxWQyMhIfHxynp8sXdrImpZmXOyitdEx4ujRo7nu2QUEBLBv3x6sVqvb39MlBcQNbN17mrFz/7FPv9S7ESqqrImJhHAfvr6+ucw1PlizrvI6d+4cUVFVc10/LCy8sKJdJbesl4uAkTUpybjyaf36taxfvzbP57p06SIBAaULPGNBkgJiorT0TIaNXU1KmnHRWUxUGf7v4UZu/61DFIzg0r55XoUlHFO6dCDJycm5PnbxYu7zzVKqVAAAI0a8RufO95uc5sZIATHJqm3HmLxgt316VP9mRJV3zxvvROHIfqmuu94/cKNy+zJUGF+QlIph06YNnDlzhvDwnHscO3duL/Dt3YiaNWsBsHv3rqsKSEZGBl99NZYKFSrQvXsvM+I5RA6wu9jFlHQeH7PUXjxa1CnPNyM6SPEQxZKXl/EdNT398s2uWSe5s8+7Uffc0wWr1cq4cZ+RmXn5NrI//ljArl07C2w7BaFBg0ZUqFCJ3377me3bt+V4bPr0KcyePQOtd+extnuRPRAXWrDuED8s32+fHvPkrZQrU8rEREIUrogI4wrCL774lKZNm/P444MJCSmDr68vW7ZsZuzYj2nbtgMNGjS8oe107NiJP/5YwKJFCzlwYD+NGzflyJHDrF27mjJlynDu3Dn7PRjXM336lDwbKnbr1pP27TveUFYvLy9GjhzNCy88w9NPD6JNm3ZUrFgZrXexefNGKlSoxBNPDL2hbbiKFBAXOJeUyvNfrLFP39U8ip4dapqYSAjX6Nq1J//88zd//x3LwYMH6NXrEQICAnj++ReZOHE8P/74A6VLB95wAfHw8OCddz5gypRJLFq0kHnz5lCpUhVee+1NVq9eydKli/H3z1/PuLi4Q3letZV11/mNatCgIf/731SmTp3E5s0bWbt2NRER5ejevRd9+/YnNDSsQLZT2DyyrmIo5qoBB8zohTVryV4WbTxsn/5kaCtCAv2ceq6CzFXY3DVbfnOdOHGIyMjcr+opDO56DqSo5Dp58gSBgYH2S2azGzp0MLt372Tx4lWFfoFKUfl9Qf7e49l6YVUHDl71eMFFFNmdPHuRx8cstRePHu1v4psRHQqseAghLpsxYyqdOt3G1q2bc8zfvv0ftm2LpVGjJnJ1YyGQQ1iF4Ouft7Nh1yn79BfD2hLgL79qIQrLvffez/z5P/Hii8No164DERHlOHbsKKtWrSAgIICnnx5mdsRiST7VCtChExcYPeVy88MB99YusH5EQoi8KRXD+PGT+fbbKWzZsomzZxMoU6Yst99+B/36DbzqDnVRMKSAFACL1coHM7aw58h5AEr7e/Px0Fb4eEvzQyFcJTo6hrfeGmN2jBJFCsgN2nXoLB9+d7n54bPd69OwputaJwghhFmkgDgpI9PCyAnrOXXuEgCVIkozqn8zaX4ohCgxpIA4YbM+xZc/Xm6PMKJPY6KrlDExkSgMRaEbqhDOKKjbN6SA5CGv4Uazq1s9lP/0bCAfMsWQp6cXmZmZRWZoUSEckZmZiafnjZ+jlb+OPFyveLw5oDmVI/IebF4UbX5+pUhJSSYwMMTsKEIUuJSUZPz8bryNkhQQJ3wzooPZEUQhK106mISEkwD4+5fGy8tL9jRFkWa1WsnMzCQlJZmLFy8QGlr+hp9TCogQufD29iE0tDzJyYkkJJzAYsm8/ko3wNPTE4vF/VpgSC7HuHsuT08v/PxKERpaHm9vn+uveB2mFxCl1BtAT9vkb1rrF5VSdwIfAl7AFmCg1jpNKRUFTAfKARroo7VOMiO3KP68vX0ICXFNU7ui3jvM1SSXYworl6nXnCqlOgJ3Ao2AhkATpdSDwCSgl9a6LhAA9LWtMg4Yp7WOATYBr7k+tRBCCDC/meJxYLjWOk1rnQ7sAqIw9jyClVJegD9wSSnlA7QFfrCtOwXoUVjB8hpWVIYbFUIIg9u0c1dK1QLWAK2AOsB3QCJwAGgHhAIbtdaVbct7Axe11vn5RK9mex4hhBCOy7Wdu+nnQACUUnWA34D/Ay4AY4C6GB/6H9v+exe4sto5dLbKjPFACpO75gL3zSa5HCO5HFPccmUbDyT3x28kVEFQSrUClgAjtNZTgTbAdq31fq21BZgA3AacAkJsh7UAKgDHTIgshBAC80+iVwF+AnprrWfZZm8Hmiulsi5Svh/j0FU6sAp4yDa/L7DQlXmFEEJcZvYhrBcwTpJ/rJTKmvc1xtVVy5RSGcA+YLDtsSHAVKXUSCAOeNi1cYUQQmQxtYBorZ8Dnsvj4am5LH8I43CWo7zAOJ7nrBtZtzC5ay5w32ySyzGSyzHFKVe2dXJtnOU2V2EVstYYh7+EEEI4rg2w+sqZJaWA+AHNMO47KdyeFEIIUXx4YVywtBFIvfLBklJAhBBCFDDTL+MVQghRNEkBEUII4RQpIEIIIZwiBUQIIYRTpIAIIYRwihQQIYQQTpECIoQQwilm98IylVIqGFgLdNZaH1RK3Qp8AgQB24DHbEPpNgQmAsHASuBJrXVGYQ2xmz0XcDNGK/sslYD1WuvOZuay/b4cGnpYKVUGmAHUAE4DPbXWJwohVz/gRYybRpdiDFqW5++lEHPlNlxzR4zhCUoBs7XWI23Luuy1zC2Xbb4P8DvwltZ6uTvkUkoNBp7FGMphE/CEq/8m88j1FDAU8MAYiuJFrbXV7FzZHhsKdNda32abLpT3fondA1FK3YJxa360bToYmAcM1lrXsS02wPb/6cBQrXU0xhtmkG1+gQ+xe2UurfUCrXVDrXVD4C6MQbb+Y3YuG0eHHn4bWKW1ro3Rpv+zgs6ljK6cbwO3a63rAT4YH0CuzpXbcM0PA99gdJiuDTRTSt1tW8Ulr2Vew0jbfm/LgZZXrGJmrpcwxghqCdTH+Lx62g1y/Qd4HmgO1LPlu8MNcj1oe+xmYMQVqxTKe7/EFhCMF/ZpLo8pcgfwl9Z6m236GeBHpVRVoJTWep1t/hSgRyEOsXtlruw+BL7WWu91k1yODj18L8a3HTBGnLzbtnxB5qqP8Toet03/CjxgQq7chmuOBvZqrQ9orTMwPmx6uPi1zGsY6QEY76/1WQu6QS5/YIjWOlFrbQX+AaLcIJcFuFlrnQyUAUKAc26QK0op5QeMB17PWrAw3/sl9hCW1nogQLY28jWBJKXULCAGY3jd4RgV/ni2VY8DlYFwINH2QZB9fkHnwjZdC6MT8UDbrIpukGsIxrfWrKGHf7jO9u2Zbbv1iUAENzAwWC65/sYYHqCK7Xm7A5Em5NqR9W/ba9cTGEvur5nLXss8crXSWu+1zRuWbXF3yhWBccion5vkSldKDQI+AjYAsUATs3MB72Hs5WYfwrvQ3vsleQ/kSt5AJ+BljDdCaYzdQE9yDqXrgfEN5Mr54OAQuw4ajLELmtXQzNRcSqlILg89XAFYh3Fs/1rbv7KftAcFnE1rvQfjdfsFowPzNiDNrFy24ZoXYxyK+Zf8vWaF/lpmz5X1IZ0Lt8illKqEMWrpJNu5GbfIpbWeAIQBJ4BRZucCqgFRWuvJVyxWaO99KSCXnQDW2Q4vZALfYxzjPILxAZklEqM6u3qI3QeAWdmmzc7lzNDDR205UUp5Y1ysEF+QoZRS/sAGrXUjrXVL2zb3m5Erl+Ga83rNXPpa5pIrL6bnUkrFYFwgMVVr/ZY75FJKVbHNw/atfhbGoVOzf18PA3WUUrEYJ/KbKqVmX2f7N/TelwJy2SKME1FVbNOdgc3aGMQqJesNAzwKLNQuHGJXKRWOcWzVvlvqBrmcGXp4AZdPtD+EcfIuvYBzlQaWKKWClFK+GOeyZrs6l8p9uOb1xkOqpu2PuTfGa+ay1zKPXLkyO5dSKgjj73Kk1vq/7pIL45zHDKVUGaWUB8Zh0tVm59JaP661rm274GYgsElr/VBhvvdL7DmQK2mtDyulngDm277FxmIMuQvQB5hgu1JrC/C5bb6rhtitgfHt5kqm5dJa71JKOTr08GvAFKXUDuCcLX9B54pXSo3GOKTmA8zUWs80IVdewzX3A+baHlvA5RObrnotc82ltf46j+XNzDUbKA8MV0oNt837RWv9usm5vsY417AWyMD4cM4qcO76OhbKe1/GAxFCCOEUOYQlhBDCKVJAhBBCOEUKiBBCCKdIARFCCOEUKSBCCCGcIgVECBdRSgUqpfYrpaxKqZ7XWG68bZkrG+IJ4VakgAjhItpo390fo1XEl0qpclcuo5TqinE/zQrgA9cmFMIxUkCEcCGt9UqMltnhwFfZH7PdXTwROAs8amsRI4TbkgIihOu9AuwGuiqlegPYWpvMBMpiDJp02MR8QuSL3IkuhAmUUs0xWmGcA+pgtDgZA0zRWvc3MZoQ+SZ7IEKYQGu9AXgfox34dIx24Psxmj8KUSTIHogQJrF1C96I0Qo8A2ittV5/7bWEcB+yByKESbTWaRgdW8EY9OqMiXGEcJgUECFMopS6D+PcxxkgAKPdtvxNiiJD3qxCmMA2JPBEIAVoC2zFGNN6+LXWE8KdSAERwsVso9hNBiKAV7XWuzD2RNKAt5RSdU2MJ0S+SQERwvWeAe4CVgKfAmittwFvAX7ANKWUj3nxhMgfKSBCuJBSqg7G5bsXgMeuuNt8DLAJaIQx1KgQbk0u4xXCRZRSfsAGjMt2B2qtJ+WyzM0YV2Z5AS211htdm1KI/JM9ECFc512M4vFrbsUDQGu9E3gd8MY4lOXvwnxCOET2QIQQQjhF9kCEEEI4RQqIEEIIp0gBEUII4RQpIEIIIZwiBUQIIYRTpIAIIYRwihQQIYQQTpECIoQQwilSQIQQQjhFCogQQgin/D/AOByavVQM+gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "predict = model(Variable(x_train))\n",
    "predict = predict.data.numpy()\n",
    "plt.plot(x_train.numpy(), y_train.numpy(), 'ro', label='Original data')\n",
    "plt.plot(x_train.numpy(), predict, 'b-s', label='Fitting Line')\n",
    "plt.xlabel('X', fontsize= 20)\n",
    "plt.ylabel('y', fontsize= 20)\n",
    "plt.legend( fontsize= 20)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Have a try"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T07:28:06.556729Z",
     "start_time": "2020-05-24T07:28:06.551348Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "x_train = np.array([[3.3], [4.4], [5.5], [6.71], [6.93], [4.168],\n",
    "                    [9.779], [6.182], [7.59], [2.167], [7.042],\n",
    "                    [10.791], [5.313], [7.997], [3.1]], dtype=np.float32)\n",
    "\n",
    "y_train = np.array([[1.7], [2.76], [2.09], [3.19], [1.694], [1.573],\n",
    "                    [3.366], [2.596], [2.53], [1.221], [2.827],\n",
    "                    [3.465], [1.65], [2.904], [1.3]], dtype=np.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Classification\n",
    "\n",
    "<img src= 'img/neuralnetwork/softmax-regression-scalargraph.png' width= \"500px\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Activation Function\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T12:55:28.328450Z",
     "start_time": "2020-05-24T12:55:28.098083Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAESCAYAAAABl4lHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de1xUdcIG8GeGO3IVZkDxDhbe8JIXQvOShanhPde2VXvZ/HRzeWN37U1tc7PXcnf7xJa5vavV6+etzDRvuRW5XsoKFVETFVCRFFEYhotyG5jL+b1/oJMjCDPAzJkZnu9nWzjn/AYeDzM8wzkzv6MQQggQERFZQSl3ACIich0sDSIishpLg4iIrMbSICIiq7E0iIjIaiwNIiKyGkuDiIis5il3AHurrKyFJNn+VpSwsACUl9fYIVHHcPZ8gPNnZL72Yb72cdZ8SqUCoaFd7rrd7UtDkkSbSuPWbZ2Zs+cDnD8j87UP87WPs+drDg9PERGR1VgaRERkNZYGERFZzSlKo6amBo8++iiKioqabMvNzcWcOXMwZcoUrFy5EkajUYaEREQEOEFpnDp1Co8//jguXbrU7PZly5bhlVdewTfffAMhBLZu3erYgEREZCZ7aWzduhWrVq2CWq1usu3q1auor6/HsGHDAABz5sxBenq6oyMSETk9IQSEEJBu/mcvsr/kds2aNXfdVlpaCpVKZV5WqVTQaDSOiEVEbsgkSajXm1DfYIJOb2z8XG+0XG5o/GgwSTBJAiaTgEn65XNJEjBJAkZJurnt5nbz5wImkwQhAEkINP7+FpDErV/sjR8VSgVMplu/6G/bhltjGm93++//W58LCNz8X7MUCuDZmYMxMrbpk/H2kr00WiJJEhQKhXlZCGGxbI2wsIA2f3+VKrDNt3UEZ88HOH9G5msfZ8hnMknQXtehuKwWxeW1KC6rhaaiDtV1etTVG6FrMEJXb0RdgxF6g8mqr+npoYS3lxIeSiU8PRTw8FDCQ6kwf+6pVMLDQ9E4zlsJT6WXednDQwEPZeN4hQJQKBo/KhUK8+eW6xo/4vYxaPzFD/wy9pZbvwMVN/+vcXTj+FvDlB4KjI7rjrBgvw7Zxxb7psO/YgeKjIyEVqs1L5eVlTV7GKsl5eU1bXoDjUoVCK222ubbOYqz5wOcPyPztY8j8xmMEspu6KCp1EFbqUNppQ6a63XQVupQdqMeptse416eSqhC/NA12BeBvp5QBfvCz9sDvt6e8PX2gK9P40e/mx99vT3gd8c2Tw/7H7m39/6T9MY2fX2lUtHik22nLo2oqCj4+Pjg+PHjuO+++7B7926MHz9e7lhEZEeV1Q3IOleKa2W1KK3UobSyDhVVDRaHYny9PRAR6o+eEYEYGauGOsQP6lA/qEP9ERzgDaVC4fSl66qcsjSWLFmClJQUDBkyBG+++SZefvll1NTUYNCgQVi0aJHc8Yiog+kajDhxXouMMyXIu1wJASDAzwsRoX7o3zME6hA/RIT6QxXaWA6Bfl42H6qmjuE0pXHgwAHz5xs3bjR/Hhsbi88//1yOSERkRyZJQu6lSmScKcGJC1roDRJUIb5IGtsH9w+KRERXf7kjUjOcpjSIyP0JIVCoqcHhsyU4mqPBjVo9uvh6ImFQJO4fHImYqGD+BeHkWBpEZHcVVfU4kqPB4TMluFpWCw+lAnHRYUgYHIm46HB4ecr+ljGyEkuDiOyiufMU0VFBWJh4D0YNiECAn5fcEakNWBpE1KHq6o34dP95HMsthd4oQR3i13ieYnAkIkJ5nsLVsTSIqMNUVNXj79tOobi8Dg8M7Y6EQZGIjgrieQo3wtIgog5RpK1B2tZT0DUY8cL8oRjUp6vckcgOWBpE1G55lyuxbsdpeHsp8dITI9ArQv7pRcg+WBpE1C6ZuRq8/68cqEL8kDp/KMLtMN8ROQ+WBhG12TeZhfjsQD7u6RGMpXPj+IqoToClQUQ2k4TAxt2n8cWhAoy8V4UlSQPh5ekhdyxyAJYGEdnEYDRh454cZJ3T4qGRPbBgcv/Gqb2pU2BpEJHVanQGvLs9G+eLbuC3MwZh7MAIuSORg/G9+0RklbIbOrzx8XEUFFfhmZmDMGtCjNyRSAb8S4OIWlWoqUbatlPQGyT8fv4wxPYOlTsSyYSlQUQtOvtzBdbvPA0/H0+s+M0IRKnafgllcn0sDSK6q4wzxfjfr/LQLcwfqfOHITTQR+5IJDOWBhE1IYTAV0cuY/t3BRjQOxTPzx4Cf1/+uiCWBhHdQQiBj/99HgdPXEX8wAgkTx8ATw++ZoYasTSIyMLxc1ocPHEViaN6Yv6DMXwPBlng0wciMjOaJGw/VIDu4V3w2KRoFgY1wdIgIrPvs4uhqajD3An94KHkrwdqivcKIgIA1OuN2P3Dz+jfIxjDYsLljkNOiqVBRACAvceuoKpWj8cmxvBKe3RXLA0iQlWdHl8fLcSIe1SI6REsdxxyYiwNIsKeHy/BYJAwd0I/uaOQk2NpEHVypZV1+PbkVTwwtBu6hXWROw45OZYGUSe341ABPJQKzBjbV+4o5AJYGkSd2KWSKmTmliJxdE/OK0VWYWkQdVJCCGw7eBEBfl54ZHRvueOQi2BpEHVSZ3+uQO7lSiQl9OFkhGQ1lgZRJyQJgW3fXkR4sC8mDo+SOw65ENlLY8+ePZg2bRoSExPxySefNNl+9uxZzJ07FzNmzMDTTz+NqqoqGVISuZejZzW4UlqDOeP7wctT9l8D5EJkvbdoNBqkpaVh8+bN2LVrFz777DPk5+dbjFmzZg1SUlLwxRdfoG/fvvjggw9kSkvkHgxGCTsOFaBXRABGD4yQOw65GFlLIyMjA/Hx8QgJCYG/vz+mTJmC9PR0izGSJKG2thYAoNPp4OvrK0dUIrdx8EQRyqvq8dhETntOtpP17FdpaSlUKpV5Wa1WIzs722LMSy+9hOTkZLz++uvw8/PD1q1bbfoeYWFtv56xShXY5ts6grPnA5w/Y2fLV6sz4MsjlzHsHhUmdsArpjrb/utozp6vObKWhiRJFhOjCSEsluvr67Fy5Ups2rQJcXFx+N///V/813/9FzZs2GD19ygvr4EkCZuzqVSB0Gqrbb6dozh7PsD5M3bGfNu/u4jqOgNmJvRp99fujPuvIzlrPqVS0eKTbVkPT0VGRkKr1ZqXtVot1Gq1efn8+fPw8fFBXFwcAOBXv/oVMjMzHZ6TyB1UVjfg38euIH5gBHpHut4zXHIOspZGQkICDh8+jIqKCuh0Ouzduxfjx483b+/duzdKSkpQUFAAANi/fz+GDBkiV1wil7b7hwKYJIHZ4zkpIbWdrIenIiIikJqaikWLFsFgMGDevHmIi4vDkiVLkJKSgiFDhuCNN97ACy+8ACEEwsLC8Prrr8sZmcglXSurxffZxZh8Xw+oQvzkjkMuTPa3gSYlJSEpKcli3caNG82fT5gwARMmTHB0LCK3sv27i/Dx8sCjCX3kjkIuju/qIXJzF4qu4+SFMkyN740gf2+545CLY2kQubFbkxIGB3gjcWRPueOQG2BpELmxny6UIf/qDcwc1xc+3h5yxyE3wNIgclMmScLn311EZFd/PBDXTe445CZYGkRu6sfTJSgur8PcCdHwUPKhTh2D9yQiN9RgMGHX9wWIjgrCiHvC5Y5DboSlQeSG9mVdwfUaPR6bGGMxNQ9Re7E0iNxMXb0RXx25jGEx4binZ4jcccjNsDSI3Mzxc6XQNZgwPYHX/aaOx9IgcjNHcjRQh/qhX7cguaOQG2JpELmR6zUNyCusxJgBETyXQXbB0iByI8dySyEEMIaXcSU7YWkQuZEjORr0ighA9/AuckchN8XSIHITmso6/FxchfiBkXJHITfG0iByE5k5GgDA6AHqVkYStR1Lg8gNCCFwJEeDe3qGoGuQr9xxyI2xNIjcwJXSGhSX1yGeJ8DJzlgaRG7gSI4GHkoFRsby0BTZF0uDyMVJQiAzV4NBfbsiwM9L7jjk5lgaRC4uv+gGKqoaeGiKHIKlQeTijuRo4O2lxLD+nAKd7I+lQeTCjCYJx3I1GN5fBV9vT7njUCfQ4r1s+fLlbfqiCoUCr7/+eptuS0TWO/tzBWrrjZw2hBymxdLYuXNnk3W3JkETQjS7TQjB0iBykKO5GnTx9cTgvl3ljkKdRIulsWvXLovl69ev449//CNCQkLw3HPPYcSIEQgODkZdXR1Onz6Nd999F9XV1fjHP/5h19BEBDToTTh5vgzxgyLg6cEjzeQYLZZGbGysxfKKFSvg6emJjz76CKGhoeb1fn5+mDhxIkaOHIlZs2bhnXfewdtvv22fxEQEAPgpvwwNBhNfNUUOZdPTk3379mHy5MkWhXG7gIAATJo0CT/88EOHhCOiuzuao0FooA/685Ku5EA2lYZCoUBVVVWLYzQaDXx8fNoViohaVqMz4HRBOUYPUEPJiy2RA9lUGiNGjEB6ejoyMzOb3b53717s27cPY8eO7ZBwRNS84+dKYZIEp0Enh7Pphd0vvPACjh49iuTkZDzwwAMYNGgQAgICUF1djRMnTuDIkSMICwtDamqqvfISERoPTUV29UeviAC5o1AnY1Np3Hvvvfjkk0+wZs0aHDx4EAcPHjRvUygUeOCBB/CnP/0J3bt3t/pr7tmzB++99x6MRiMWL16MJ554wmJ7QUEBVq1ahRs3bkClUuGtt95CcHCwLbGJ3EpldQPOFV7HzHF9eR1wcjib30I6YMAAfPzxxygtLUVeXh6qqqoQFBSEgQMHIjzctmkMNBoN0tLSsGPHDnh7e2PBggUYM2YMYmJiADS+F+TZZ5/FypUrMX78eLz55pvYsGEDli1bZmtsIreRmauBAK8DTvJo87wDarUaanX7pmHOyMhAfHw8QkIaX/0xZcoUpKenY+nSpQCAs2fPwt/fH+PHjwcAPPPMM62eiCdyd0dyNOgTGYiIrv5yR6FOqMXSWLp0KaZNm4Zp06aZl62hUCiwbt26VseVlpZCpVKZl9VqNbKzs83LhYWFCA8Px4oVK5Cbm4t+/frhT3/6k1UZiNxRcXktLpdUY8GDMXJHoU6qxdLYt2+fxRv89u3bZ9UXtfY4qyRJFmNvTUFyi9FoRGZmJj7++GMMGTIEf//737F27VqsXbvWqq8PAGFhbT9RqFIFtvm2juDs+QDnz+hq+f594ioUCuCRcf0QFuwnU6pfuNr+czbOnq85LZbG/v37ERQUZLHckSIjI5GVlWVe1mq1Foe8VCoVevfujSFDhgAAHn30UaSkpNj0PcrLayBJTefJao1KFQitttrm2zmKs+cDnD+jq+UTQuDAsULE9gqFpDfKnt3V9p+zcdZ8SqWixSfbLZZGVFRUi8vtlZCQgHXr1qGiogJ+fn7Yu3cvXnvtNfP24cOHo6KiAnl5eYiNjcWBAwcwaNCgDs1A5Coua6qhqdRhanxvuaNQJ9amE+FFRUXYtWsXzp07B51Oh5CQEPTv3x/Tpk1Dz549rf46ERERSE1NxaJFi2AwGDBv3jzExcVhyZIlSElJwZAhQ7B+/Xq8/PLL0Ol0iIyMxF//+te2RCZyeUfONl4H/L57Va0PJrIThWhujvMWfPrpp1izZg2MRmOTbV5eXli5ciUWLFjQYQHbi4en5OPsGV0pnyQJ/PEfP6JvtyD8bm6czMkaudL+c0bOmq9dh6fulJGRgdWrVyM8PBzPPPMM7rvvPqjValRVVeHYsWNYv349Vq9ejejoaIwaNard4Ymo0bkr13G9Rs/3ZpDsbCqN999/H4GBgfj000/Ro0cP8/quXbuiT58+iI+Px9y5c/HBBx+wNIg60NEcDXy8PTA0htcBJ3nZNGFhdnY2Jk+ebFEYt+vZsycmT56Mn376qUPCERFgMEo4fq4UI/qHw8fLQ+441MnZVBoGgwH+/i2/C9Xf3x/19fXtCkVEvzjzc/nN64BzRluSn02l0a9fP3z//fd3LQWdTodDhw6hb9++HRKOiBoPTQX4eWFgn+YvfkbkSDaVxmOPPYbCwkKkpKTg6tWrFtvy8/Px3HPPoaioCPPmzevQkESdVb3eiJ8ulGHUADWvA05OwaYT4Y8//jiOHj2Kb775Bg899BAiIiIQGBiI0tJSVFVVQQiBxMTEJtObE1HbnLxQBr1RwpgBfNUUOQebSkOhUODvf/87du/ejZ07dyIvLw9lZWXo0qULRo8ejdmzZ2PWrFn2ykrU6RzN0SAsyAcxPXgNGXIONr8jXKFQYNasWSwHIju7UdOAMwUVmDKmJ68DTk6jzdfT0Ov10Ov1d90eEMDLUBK1x4/Z1yAJXgecnItNpdHQ0IC0tDT861//Qnl5+V3HKRQK5OTktDscUWd26ORVRIV3QQ9VF7mjEJnZVBp/+9vf8PHHHyMgIABxcXHw9va2Vy6iTq38Rj3OFpRj9vh+vA44ORWbSmPv3r3o378/Pv30Ux5+IrKjzFwNAF4HnJyPTS/8rq6uxoQJE1gYRHZ2JEeDe3uHQh0i/9X5iG5nU2mMGDECubm59spCRACultXiSmkNxg/v2IueEXUEm0pj+fLlOH36NP72t7+1eCKciNruaI4GCgXwwDCWBjkfm85p9OvXD4mJifjwww/x4YcfwsfHp9mT4QqFAkePHu2wkESdhRACR3NKMLB3KEIDfaGtN8gdiciCTaXx9ttvY/v27RBCIDQ0FH5+PN5K1JEKiqugvV6PpARO+knOyabS2L59O7p3747333+fM9kS2cHRHA08PZQYcQ+vA07OyaZzGjU1NUhMTGRhENmBJAlk5pZiaHQY/H3bPFkDkV3ZVBoDBw5sMiU6EXWM3MJKVNXyOuDk3GwqjdTUVBw8eBAfffQRjEajvTIRdUpHz2rg5+OBuOgwuaMQ3ZVNfwN/8cUX6NOnD15//XW8+eabiIyMbPbyrwqFAjt27OiwkETuzmA04fj5Uoy4RwVvXgecnJhNpbFt2zbz5w0NDbh8+XKz4zhXDpFtsi9WQNdg4oy25PRsKo28vDx75SDq1I7mlCDI3wuxvUPkjkLUIl50mEhmugYjfsovx6gBEfBQ8iFJzs2mvzSOHTvW6hilUgk/Pz9ERkaia9eubQ5G1FmcOK+F0SQhnq+aIhdgU2ksXLjQpvMVMTExeOWVVzBq1CibgxF1FkdzNAgP9kW/7kFyRyFqlU2lsXbtWqxfvx5XrlzB2LFjMXz4cERERKCmpgY//fQT9u/fj6CgIMyZMwdarRb79u3DU089hS1btmDAgAH2+jcQuawbtXrkXKrE1PhefAEJuQSbSqOsrAzFxcX45z//iQkTJjTZfuTIETz11FNQq9X44x//iMLCQsyZMwcbNmxAWlpah4UmchdZeaU3rwPOQ1PkGmw667ZlyxZMnTq12cIAgPj4eCQmJuKjjz4CAPTq1QsPP/ywVedCiDqjIzkl6KEKQJSKFzYj12BTaZSVlSE0NLTFMWFhYdBoNOZltVqN6urqu47fs2cPpk2bhsTERHzyySd3Hfftt9/iwQcftCUukVPTXtfh4tUqxA/iXxnkOmwqjZ49e+K7776DTqdrdnt9fT0OHTqEqKhfLh5z+fJlqFTNz9ip0WiQlpaGzZs3Y9euXfjss8+Qn5/fZFxZWRn+8pe/2BKVyOndug746AFqmZMQWc+m0njiiSdw+fJlJCcnIzMz01weJpMJ2dnZePbZZ1FYWIj58+cDAHbt2oUDBw4gPj6+2a+XkZGB+Ph4hISEwN/fH1OmTEF6enqTcS+//DKWLl1q67+NyKkdydEgpkcwwoN5XRpyHTadCF+wYAEKCgrw0UcfYfHixQAAPz8/6PV6mEwmCCGwYMECPPnkk7hx4wZeeuklBAcH4+mnn27265WWllr8FaJWq5GdnW0x5v/+7/8wcOBADB061NZ/GwAgLKztx4pVqsA239YRnD0f4PwZ5cp3qbgKV7W1eGZOXIsZuP/ah/k6ns2T9q9YsQIzZszArl27cObMGVRWVqJLly4YPHgw5syZg2HDhgEADAYDXnzxRUyfPh0REc0fs5UkyeJlhkIIi+Xz589j79692LRpE0pKSmyNCgAoL6+BJAmbb6dSBUKrvfu5GLk5ez7A+TPKme/rHwqgVCgQ2yPorhm4/9qH+dpGqVS0+GS7TVd6GTx4MAYPHtzimPDwcCQnJ7c4JjIyEllZWeZlrVYLtfqX47vp6enQarWYO3cuDAYDSktL8etf/xqbN29uS2wip9B4HXANBvXtiiB/b7njENmkxdLIy8uDSqVCWFiYedlasbGxrY5JSEjAunXrUFFRAT8/P+zduxevvfaaeXtKSgpSUlIAAEVFRVi0aBELg1zexatVKK+qx+zxvAImuZ4WS2PWrFlYunSp+ST0rFmzrH7Xam5ubqtjIiIikJqaikWLFsFgMGDevHmIi4vDkiVLkJKSgiFDhlj1vYhcyZGcEnh5KjG8P68DTq6nxdKYPXu2xfQftpSGtZKSkpCUlGSxbuPGjU3G9ejRAwcOHOjQ703kaEaThGN5pRgWEw4/H14HnFxPi/faN954w2J57dq1zY4zmUwoKipCeHg4unTp0nHpiNxM7uVKVNcZOG0IuSybJ+8/duwYUlNTYTKZADSe55g8eTIeeeQRJCQk4N133+3wkETu4miOBv4+nhjcj9cBJ9dkU2kcPnwYTz75JNLT01FcXAyg8Y13JSUlGDNmDKKiorB+/Xrs3r3bLmGJXJneYMLx81rcd68KXp682BK5Jpvuue+//z66dOmCbdu2oUePHrh48SLOnDmDcePGYdOmTdi1axf69evHVzgRNePUxXI06E08NEUuzabSOHPmDKZNm2Z+j8bBgwehUCgwdepUAIC3tzceeOABXLhwoeOTErm4I2dLEBzgjXt7tTzpJ5Ezs6k0GhoaEBj4y9veDx06BAAYO3aseZ0kSfD05KtCiG5XV2/A6YJyjI6NgFLJiy2R67KpNHr16oVTp04BAEpKSnDixAnExMQgMjISAKDX6/Hdd9+hZ8+eHZ+UyIUdP6eF0SQ4DTq5PJtKIzExEZmZmVi4cCF+85vfwGQyYe7cuQAar3exYMECi1luiajRkRwN1KF+6BPpehPUEd3OpuNIzz77LLRaLbZt2wYhBKZNm4aFCxcCAE6ePIm8vDw8+eSTLA2i21yvaUDe5Uokje3D64CTy7OpNDw8PPDqq69i2bJlEEJYnN947LHHsHDhQoSHh3d4SCJXlplbCgFgDF81RW6gTWesAwKaTpvbo0ePdochckdHczToFRGAbmGcLYFcH99hRGRHmso6/FxchfiBkXJHIeoQLA0iOzqao4ECvA44uQ+WBpGd3LrY0j09Q9A1yFfuOEQdgqVBZCdXSmtQXF7HE+DkVlgaRHZyJEcDD6UCI2N5aIrcB0uDyA6km4emBvftigA/L7njEHUYlgaRHVy4ch2V1Q0Yw2lDyM2wNIjs4GiOBt5eSgyP4XXAyb2wNIg6mNEkIeucFsP7q+Dj7SF3HKIOxdIg6mA/nC5Gjc6AhMF8Qx+5H5YGUQdq0Juw+/ufEdMjGIP7dpU7DlGHY2kQdaC9WVdwo1aP+RNjOKMtuSWWBlEHqarT4+sjlzG8fzhiegTLHYfILlgaRB3kXxmX0GAwYe6EaLmjENkNS4OoA5Re1+Hgiat4IK47uodzCnRyXywNog6w61ABPJQKzBzXV+4oRHbF0iBqp8sl1TiSo8HDo3oiNNBH7jhEdsXSIGqnz7/NR4CfF6aO6S13FCK7Y2kQtcPZnytw9lIlHk3oA3/fNl09mcilyF4ae/bswbRp05CYmIhPPvmkyfZ9+/Zh5syZmDFjBp577jncuHFDhpRETUlCYNu3+QgP9sWk4VFyxyFyCFlLQ6PRIC0tDZs3b8auXbvw2WefIT8/37y9pqYGf/7zn7FhwwZ88cUXuPfee7Fu3ToZExP9IjNHg0JNDWaP7wcvT9mffxE5hKz39IyMDMTHxyMkJAT+/v6YMmUK0tPTzdsNBgNWrVqFiIjG6aXvvfdeFBcXyxWXyMxglLDjUAF6qQN4ZT7qVGQtjdLSUqhUv0wdrVarodFozMuhoaF4+OGHAQD19fXYsGEDHnroIYfnJLrTtyevouxGPeZNioaS04VQJyLrmTtJkizm5xFCNDtfT3V1NZ5//nnExsZi9uzZNn2PsLCANudTqQLbfFtHcPZ8gPNnbEu+Wp0B/zp8GUP7h2PiqN52nWPKHfefIzFfx5O1NCIjI5GVlWVe1mq1UKstr6dcWlqK3/72t4iPj8eKFSts/h7l5TWQJGHz7VSqQGi11TbfzlGcPR/g/Bnbmm/HoYuortNj5tg+KCursUOyRu66/xyF+dpGqVS0+GRb1sNTCQkJOHz4MCoqKqDT6bB3716MHz/evN1kMuGZZ57B1KlTsXLlSs4aSrK7XtOAvZlXMHqAGn0ig+SOQ+Rwsv6lERERgdTUVCxatAgGgwHz5s1DXFwclixZgpSUFJSUlCAnJwcmkwnffPMNAGDw4MFYs2aNnLGpE9v9w88wSQJzOCkhdVKyvxspKSkJSUlJFus2btwIABgyZAjy8vLkiEXURHF5Lb4/VYxJI6KgDvGTOw6RLPjiciIrbf+uAN5eSiSN7SN3FCLZsDSIrJB/9QZOnNdi6pheCPL3ljsOkWxYGkStEEJg28F8BHfxRuKoXnLHIZIVS4OoFafyy3Gh6AZmjusLH28PueMQyYqlQdQCkyTh8+8uIqKrP8bFdZM7DpHsWBpELcg4XYJrZbWYN6EfPD34cCHio4DoLhoMJuz64WdEdw/CiHtUrd+AqBNgaRDdxf7jRaisbsC8idGcjYDoJpYGUTNqdAZ8efgyhkaH4d5eoXLHIXIaLA2iZnx5+BLq9UbMncjpQohux9IgusP5K9ex/3gRxg7uhh6qtk+tT+SOZJ97isiZZOWVYsOeHIQF+2HOhH5yxyFyOiwNopv+nXUFW/ZdQL+oIKTMjUMgpwshaoKlQZ2eJAQ+P3gR6ZmFGN4/HE/PGARvL77zm6g5LA3q1AxGCR98mYPM3FI8OCIKv37oHiiVfHkt0d2wNKjTqtEZkLb1J+QVXse8idGYOqYX349B1AqWBnVKFVX1WLfpGIpKa7AkaWsBHSYAAA6JSURBVCDuHxQpdyQil8DSoE6nqLQGadtOocFgQur8oRjYp6vckYhcBkuDOpXcy5V4d0c2fLw8sPb5cQjw4luViGzBRwx1GkdySvDWZz8hNNAXKxeORN/uwXJHInI5/EuD3J4QAumZhdh28CLu6RmC380dgi6+XnLHInJJLA1ya5Ik8On+C9h/vAijYtV46tEB8PLkezCI2oqlQW5LbzBh454cHD+vReKonpj/YAyUfEktUbuwNMgt1egMeGd7Ni4W3cCCB2OQOLqX3JGI3AJLg9yKrsGIE+e1+Nfhyyi/ocMzswZjVKxa7lhEboOlQS7PJEnIuVSJw2dKcOKCFnqDBHWIH/7wq2G8gBJRB2NpkEsSQqBQU4PDZ0twNEeDG7V6dPH1RMKgSNw/OBIxUcGcEoTIDlga5FIqqupxJEeDw2dKcLWsFh5KBeKiw5AwOBJx0eHw8uRbj4jsiaVBTk/XYMTxc1ocPluCvMuVEACio4KwMPEejBoQgQA/vueCyFFYGuSUTJKEsz9X4vDZEpw8r4Xe2HieImlsH9w/OBIRof5yRyTqlFgaJCtdgxHa6zqUVuqgqaxDaaUO2us6FGlrUaMzNJ6nGNINCYMiER0VxPMURDKTvTT27NmD9957D0ajEYsXL8YTTzxhsT03NxcrV65EbW0tRo4ciVdffRWenrLHJisJIVBbb4Smsg7aylvloLtZFHWoqjNYjA/y94I61B9Do8MwrL8KcdFhPE9B5ERk/e2r0WiQlpaGHTt2wNvbGwsWLMCYMWMQExNjHrNs2TL893//N4YNG4YVK1Zg69at+PWvfy1j6s7JYJRQrzeiXm+CrqHxY+N/Nz9vMEJ323JtvRHXa/W4VlqDugajxdfqGuQDdYgfhvUPhzrUH+oQP6hD/aAK8YOfD58QEDkzWR+hGRkZiI+PR0hICABgypQpSE9Px9KlSwEAV69eRX19PYYNGwYAmDNnDt555x27l4YkBLJyNdBoq+86Rog7ltFkxS9rxC/bb7+duG29xXg0PkMXt8YIASEac4mbK/27+KC6pt5i+53jTJKASZJgMokmnxslAZNJgkkSkKSb228u39p2ezEYTXf8++7C20sJP29P+Pl4orsqAL0GRdwsBf+bxeDLuZ+IXJispVFaWgqVSmVeVqvVyM7Ovut2lUoFjUZj0/cICwuwOVf+let49f0jNt/O2Xh6KODhoYSn8uZH87ISHh4KeNy+XqmEj48XPDwU8Ly5ztfHE/4+jQXg5+sJfx8v+Pl4wt/39nWe8PPxgr+vJ3x9POHhYtfXVqkC5Y7QIuZrH+breLKWhiRJFic2hRAWy61tt0Z5eQ0kybpnybcE+3rg/ZUPo1hTZbH+zu/cWhSFQvHLbRS33f629YrbvrACCouvqVAooFTc/Dq3f4QCKlUAKsprb66/Y8zN2zqM0Yi6GiPqaixXq1SB0Lbw15rcmK99mK99nDWfUqlo8cm2rKURGRmJrKws87JWq4VarbbYrtVqzctlZWUW2+0poqs/lCaTQ75XW/j7eqHWm4d5iMixZH1ZSkJCAg4fPoyKigrodDrs3bsX48ePN2+PioqCj48Pjh8/DgDYvXu3xXYiInIsWUsjIiICqampWLRoEWbNmoVHH30UcXFxWLJkCU6fPg0AePPNN/HGG2/gkUceQV1dHRYtWiRnZCKiTk0hxJ2vA3IvbTmnATjv8cZbnD0f4PwZma99mK99nDVfa+c0+K4pIiKyGkuDiIisxtIgIiKruf2cDcp2vNmsPbd1BGfPBzh/RuZrH+ZrH2fM11omtz8RTkREHYeHp4iIyGosDSIishpLg4iIrMbSICIiq7E0iIjIaiwNIiKyGkuDiIisxtIgIiKrsTSIiMhqbj+NSEvOnTuH3//+9/jyyy/N6z788ENs3boVQgj84Q9/QGJiYpPbZWRk4I033kBDQwOmTp2K1NRUu2V85ZVXcOrUKfPy+fPnkZaWhkceecRi3MKFC1FRUQFPz8Yf6erVqzF06FC75brFYDBgzJgx6Nmzp3ndjh074OFheVVBa/arPdTW1mLFihUoKCgAADzzzDOYPn16k3Fy7b89e/bgvffeg9FoxOLFi/HEE09YbM/NzcXKlStRW1uLkSNH4tVXXzVndIR3330XX3/9NQBgwoQJePHFF5ts3759O4KCggAA8+fPb/JvsLfWfnaOfLzeadu2bfj444/Ny0VFRZg5cyZeeeUV8zpn2Ic2EZ3Uzp07xbhx48SkSZPM606dOiVmzpwp6uvrRVlZmZg8ebKorKy0uJ1OpxMTJkwQhYWFwmAwiOTkZPHtt986JPO2bdtEcnKykCTJYr0kSWLcuHHCYDA4JMftTp8+LZKTk1scY81+tZe33npLrF27VgghRFlZmRg7dqzQarUWY+TafyUlJWLSpEmisrJS1NbWiqSkJHHhwgWLMdOnTxcnT54UQgixfPly8cknnzgs348//ih+9atfiYaGBqHX68WiRYvE3r17LcY8/fTT4sSJEw7LdKfWfnZyPl7vdP78efHwww+L8vJyi/Vy70NbdcrDU9XV1di/fz/eeusti/WHDh3Cww8/DB8fH4SFhWH06NH49ttvLcZkZ2ejd+/e6NmzJzw9PZGUlIT09HS7Z66srMQ777yD1atXQ6GwnFDs1rPo5ORkzJgxw+KZjb2dPn0aFRUVmDNnDubPn4/MzMwmY6zZr/YyevRoLFy4EAAQFhaGkJAQlJWVWYyRa/9lZGQgPj4eISEh8Pf3x5QpUyzuS1evXkV9fT2GDRsGAJgzZ45D7mu3qFQqvPTSS/D29oaXlxeio6Nx7do1izFnzpzBP//5TyQlJWH16tVoaGhwWD6g9Z+dXI/X5vz5z39GamoqunbtarFe7n1oq05ZGoGBgVi3bh26detmsb60tBRqtdq8rFKpUFJS0mSMSqUyL6vVamg0GvsGBrBp0yZMnz4dUVFRTbZVVVXh/vvvx/r167Fp0yZs2bIFP/74o90zAYBCocDkyZPx2WefmR8UFRUVFmOs2a/2MnbsWHTv3h0A8NVXX0Gv1yMmJsZijFz7r7X70p3bVSqVQ+5rt/Tv399cWJcuXcLXX3+NCRMmmLfX1tZiwIABWLZsGXbu3Imqqir84x//cFg+oPWfnVyP1ztlZGSgvr4eU6dOtVjvDPvQVp2yNO5GkqQm65RKZZMxtz/TF0I0eeZvj1zbt2/H4sWLm90+fPhw/PWvf0VgYCC6du2KefPm4bvvvrNrplsWLFiApUuXwsvLCwMHDkRcXBxOnDhhMcaa/WpvX3/9NV5//XW88847Tc4JyLX/WrsvyXFfa86FCxeQnJyMF198EX369DGv79KlCzZu3Ijo6Gh4enoiOTnZYfe7W1r72TnLPtyyZQv+4z/+o8l6Z9iHtuo0pfH2229j5syZmDlzJvbv39/smMjISGi1WvOyVqu1eIZs7ZiOznry5En06dMHkZGRzY7PysrC4cOHzctCCLueLL09386dO1FYWGjxvb28vCzGO2Kf3S3f/v378dFHH+Evf/kLPvjgA8TGxjYZ7+j9d0tr++XO7WVlZXbdb805fvw4nnzySfzhD3/A7NmzLbZdu3YNn3/+uXnZUfvtdq397Bx932uOXq/HsWPH8OCDDzbZ5gz70GZynUxxBleuXGn2RHhdXZ0oLy8XkyZNEiUlJRa3qa+vF+PHjxeXLl0SRqNR/Pa3vxVfffWVXXO+//774q233rrr9gMHDohZs2aJ+vp6UV1dLZKSkhx2Ym3t2rVi1apVQgghLl68KMaPHy+qq6stxlizX+3l3//+t3jwwQfFtWvX7jpGrv1360R4eXm5qKurEzNmzBCnTp2yGDN9+nSRlZUlhBDi5ZdfFhs3brR7rluuXbsmxowZIzIyMprdXl5eLkaPHi0KCwuFJEli+fLl4n/+538clk+I1n92cjxe75SdnS0WLFjQ7DZn2Ie2YmncVhpCCPHBBx+IadOmicTERLFz507z+hkzZph/0WVkZIikpCSRmJgo1qxZ0+TVTB1t1apVYvPmzRbrSkpKxIwZM8zLaWlp4pFHHhGJiYli06ZNds1zu+rqavG73/1OTJ8+XTz66KPi8OHDQojGB8pTTz1lHne3/WpvSUlJYuzYsWLGjBnm/7Kzs51m/33xxRdi+vTpIjExUWzYsEEIIcRTTz0lsrOzhRBC5Obmirlz54opU6aI3//+96KhocFh2V577TUxbNgwi323efNmi3zp6enm/C+99JJD893S3M9Ozsfrnb788kvxwgsvWKxztn1oC165j4iIrNZpzmkQEVH7sTSIiMhqLA0iIrIaS4OIiKzG0iAiIquxNIiIyGosDSIishpLg4iIrMbSICIiq7E0iBwgKysLsbGxmDhxImpra83r9Xo9kpKSMGDAAGRlZcmYkMg6LA0iBxg5ciR+85vfoLi4GOvWrTOvX7duHc6fP4/k5GSMHDlSxoRE1uHcU0QOotPpkJSUhOLiYmzfvh16vR4LFixAdHQ0tm/fDm9vb7kjErWKpUHkQEePHsXixYtx33334fr167h8+TI+//zzZq/zQeSMeHiKyIHGjBmDxx9/HFlZWcjPz8d//ud/sjDIpbA0iBwsMTHR/Pnw4cNlTEJkOx6eInKghoYGzJo1C9euXQMAdOvWDbt374aPj4/MyYisw780iBzo7bffRkFBAZ5//nk8/fTT+Pnnn/H222/LHYvIavxLg8hBsrOzza+W2rFjB4QQmDVrFi5duoQtW7YgLi5O7ohEreJfGkQOoNfrsXz5ckiShFdffRVeXl7w9vbG6tWrIUkSli9fDr1eL3dMolaxNIgcYN26dcjPz8f8+fMxYsQI8/qRI0di7ty5yM/Pt3jTH5Gz4uEpIiKyGv/SICIiq7E0iIjIaiwNIiKyGkuDiIisxtIgIiKrsTSIiMhqLA0iIrIaS4OIiKzG0iAiIquxNIiIyGr/D3l0V3wrj960AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def sigmoid(x):\n",
    "    return 1/(1 + np.exp(-x))\n",
    "\n",
    "plt.plot(range(-10, 10), [sigmoid(i) for i in range(-10, 10)])\n",
    "plt.xlabel('x', fontsize = 20)\n",
    "plt.ylabel('sigmoid', fontsize = 20);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T12:55:36.324672Z",
     "start_time": "2020-05-24T12:55:36.088573Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAESCAYAAAAfXrn0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deVSU9eI/8Pew7yIIorigoLK4a6hpkpq44IChld4Wk0y8x/qV1fXq/XqsvHWvdb8njtkttVxullkmhqSYW65QRpooi4iCqIDssjPAfH5/dOUricIwy2dmeL/O6ZzmeZ6ZefuZGd7zPM/MZxRCCAEiIurULGQHICIi+VgGRETEMiAiIpYBERGBZUBERGAZEBERWAZERATASnYAbZSVVUOt1vxrEu7uTigpqdJDIt1gPu0wn3aYT3vGmtHCQoGuXR1bXWfSZaBWiw6VwZ3rGjPm0w7zaYf5tGcKGe/Gw0RERMQyICIilgEREYFlQEREYBkQERFYBkREJkVfvzrAMiAiMhGZ18vx1w1JKC6v1fltswyIiExAVW0DNsWnwkKhgLODjc5vn2VARGTkhBDYlpCB21UqREcEwdbGUuf3wTIgIjJyx87dxNnMIswJ8UW/Hi56uQ+WARGREbtRWIWvjmRhcH83hAb31tv9sAyIiIxUfUMTPom7CAc7K7wQFggLhUJv98UyICIyUjuPXEZ+SQ1enBWILo66P2l8N5YBEZER+iWjEMd/y8OMsX0Q1M9N7/fHMiAiMjLF5bXYlpCBfj1c8Pgj/Q1ynywDIiIj0qRWY2N8KgCB6IggWFka5s80y4CIyIjEncrGlZsVeG6aPzxd7Q12vywDIiIjkZ5Tin2J1zBhaA+MCexu0PtmGRARGYGKGhU2fZ+G7m4OePqxgQa/f5YBEZFkQghs2ZeO6toGLNHTdBNtYRkQEUl2OPkGUq6U4MlJfujT3VlKBpYBEZFE1woqsetYFob7dcOUUb2k5WAZEBFJUqdqxIa9qXB2sEFUWAAUepxuoi0sAyIiSb48lInC0t+nm3Cyt5aahWVARCTBT6kFOH2hALMe9oF/366y48gvg7i4OISFhSEsLAzvvfee7DhERHpXWFaDz3+4hAG9uiB8go/sOAAkl0FtbS3effddbN++HXFxcUhOTkZiYqLMSEREetXYpMbGvb//fOViZRAsLaS/JwcguQyampqgVqtRW1uLxsZGNDY2wtbWVmYkIiK9ij1xFdn5lVg40x/uXexkx2lmJfPOnZyc8Morr2DGjBmwt7fHQw89hJEjR8qMRESkNxezS3Dg51w8OsIbowZ5yo7TgkIIIWTdeUZGBlasWIHNmzfD2dkZb7zxBoYOHYpFixbJikREpBdllXX4f/97DC5ONvjg1RDYWhv+W8YPInXP4NSpUxg3bhzc3d0BAJGRkdixY0e7y6CkpApqteZd5uHhjKKiSo2vZyjMpx3m0w7zae+PGdVCIOab86iua8DrTw1DRXmNlFwWFgq4uzu1vs7AWVrw9/dHYmIiampqIITA0aNHMWTIEJmRiIh07oczuUjNLsX8KQPg7dH6H2PZpO4ZTJgwAWlpaYiMjIS1tTWGDBmCxYsXy4xERKRT2fkViD1+FaMGeSBkeE/Zce5LahkAwOLFi1kARGSWausbsSHuIlydbPD8DH+p0020xTg+4EpEZGaEENj+wyWU3K7H4vAgONrJnW6iLSwDIiI9OH2hAD+l3ULEBB8M6OUqO06bWAZERDp2o7ASXx7KhH8fV4SN85Edp12knzMgIjInDY1q/Ourc7C2ssCLyiBYWBjveYK7cc+AiEiHdh3LwtWbtxE1MwBdnU1neh2WARGRjvyWVYzDyTegfKQ/hg/oJjuORlgGREQ6UFZZjy370tHH0wkLZwXKjqMxlgERkZbUaoFP41OhamxCdEQQrK2Ma96h9mAZEBFpad9P15CRW46npw5ED3dH2XE6hGVARKSFrBu3EXcyG2MCu2PCkB6y43QYy4CIqINq6hqwcW8q3Fxs8WzoIKOebqItLAMiog4QQmBbQgbKq+qxJGIwHOxM+2tbLAMiog44cT4PyZeKEDmxP/r3dJEdR2ssAyIiDd0srsZXhy8jyKcrpo3pIzuOTrAMiIg0oGpowoa4i7CzscSiWYGwMOHzBHdjGRARaeDrH7Nws6gai2YFoouT6Uw30RaWARFRO53NLMKPZ29ienAfDO7vLjuOTrEMiIjaoeR2HbbuT4ePlzMiQ/rLjqNzLAMiojY0qdXYFJ+KRrVAdEQQrCzN70+n+f2LiIh0LP50Di7fuI3npg1C964OsuPoBcuAiOgBLuWWIT4xBw8P9sK4IC/ZcfSGZUBEdB9VtQ3YFJ8GT1d7PBM6UHYcvWIZEBG1QgiBLfvSUVGtwpKIwbCzMe3pJtrCMiAiasXRszfxW1YxnnjUF329nGXH0TuWARHRH+TeqsTXR7Mw1NcdUx/qLTuOQbAMiIjuUq9qwsa9qXC0s0JUWIBJT0utCZYBEdFddhzOREFJDV5UBsLFwUZ2HINhGRAR/deZ9Fs4mZKPmeP6ItDHTXYcg2IZEBEBKCqvxX8OZMC3pwsiJvSTHcfgWAZE1Ok1NqmxcW8qAGBxuHlON9GWzvcvJiL6g+9OZuNqXgUWTPeHh6u97DhSsAyIqFNLzSlFwk/XMHFYDwQHdJcdRxqWARF1WhXVKnwWnwYvdwfMn2Le0020hWVARJ2SWghs3peO6rpGLIkYDFsbS9mRpGIZEFGndOiX67hwtQRPTfZDb08n2XGkk14GR48eRWRkJGbMmIF33nlHdhwi6gRyCirw7bErGDGgGyaP9JYdxyhILYPr16/jzTffxMcff4y9e/ciLS0Nx48flxmJiMxcbX0jNsSlwsXRBgtndp7pJtoidU7WQ4cOYebMmfDy+v0HI2JiYmBrayszEhGZuS8OZqKovBbL54+Ak7217DhGQ+qewbVr19DU1IQlS5YgIiICO3bsQJcuXWRGIiIzlnSxAEmpBVA+7INBfbrKjmNUFEIIIevOV61ahXPnzmH79u1wcHDAn//8ZyiVSkRGRsqKRERmKq+oCq/GHEO/nl3wjz+Ph2Un/Jbxg0g9TNStWzeMGzcObm6/Twj12GOPISUlpd1lUFJSBbVa8y7z8HBGUVGlxtczFObTDvNpxxzzNTap8Y/tv8JCocDC6f4oLa3WU7rfGesYWlgo4O7e+ienpFbjpEmTcOrUKVRUVKCpqQknT55EUFCQzEhEZIa+PXYF1woq8fyMALh3sZMdxyhJ3TMYNmwYFi1ahD/96U9oaGjA+PHjMWfOHJmRiMjMpFwpwcFfrmPSCG+MGuQhO47Rkv4Lz3PnzsXcuXNlxyAiM1ReVY/N+9LQy8MRT032kx3HqPEMChGZJbUQ+Oz7NNSrmhAdMRg21p17uom2sAyIyCwd+DkXaTllmPfYAHh3c5Qdx+ixDIjI7FzJu409J65itL8nQob1lB3HJLAMiMis1NQ1YmNcKlydbPH89EGcbqKdWAZEZDaEEPj8hwyUVtQjOjwIDnacbqK9WAZEZDZOpeTjTHohIh7pB79enNpGEywDIjILecXV+PJwJvz7uCJsbF/ZcUwOy4CITF5DYxM2xKXCxsoSLyqDYGHB8wSaYhkQkcn75scruFFUhRfCAtDVmdPgdwTLgIhM2rnLRTjy6w08NroXhvl1kx3HZLEMiMhklVbUYcu+dPTp7oQnHuV0E9pgGRCRSVKrBT6NT0Njk8CSiMGwtuKfM21w9IjIJH2flINL18vxTOhAeLk5yI5j8lgGRGRyMq+XI+5UNsYGdsfDg71kxzELLAMiMimVNSpsik+FRxd7PDuN003oCsuAiEyGEALrv/kNt6tUiI4Igr2t9J9kMRsajWRwcHC7tlMoFPj55587FIiI6H6O/ZaHpAv5eGKSL/r1cJEdx6xoVAZOTq3/kHJdXR3Ky8uhVqsxcOBA9O7dWyfhiIjuuFFUhZ1HLmPkIE9MC+4jO47Z0agMjh49et91lZWV+OSTT7B7927ExMRoHYyI6I76hiZsjEuFva0VXp0/Ao11DbIjmR2dnTNwdnbG8uXL4efnh3/961+6ulkiInx95DJuFldj0awAdHW2kx3HLOn8BPKIESOQnJys65slok4qOaMQx37Lw/QxfTC4n7vsOGZL52WQnp7Oj3oRkU4U367FtoQM9OvhjMiJ/WXHMWsanTM4cuRIq8uFEKipqcGxY8eQmJiIqVOn6iQcEXVeTWo1NsWnQS0EosODYGXJT8Lrk0ZlsHTp0ge+6xdCwNPTE2+88YbWwYioc4s7lYOsG7exWBkIz66cbkLfdFYGNjY26N+/P0JCQmBtzd8dJaKOy7hWhn2JORg/xAtjgzjdhCFoVAYvv/yyvnIQEQH4v+kmurs54OmpA2XH6TR4EI6IjIYQAlv3Z6CqtgHR4UGws+F0E4bywJF+/PHHO3SjCoUCsbGxHbouEXVeR369gd+yijF/ygD09XKWHadTeWAZpKend+hG+dFSItJU7q1KfPNjFob5uuOx0b1kx+l0HlgGGRkZhspBRJ1YvaoJG+JS4WhvjaiwAL6hlIDnDIhIui8PZ+JWaQ0WK4Pg7GAjO06n1KGzMydOnEBsbCzS09NRUVGBpKQk7N27F7m5uXjhhRdgb2+v65xEZKZ+TruFUyn5mPVwXwT07So7TqelcRmsXr0au3btghAClpaWUKvVAICLFy/i888/x8mTJ7FlyxY4OjrqPCwRmZfC8lr850AG/Ly7IGJCP9lxOjWNDhPt3LkT33zzDUJDQ3Hw4EEsWbKked3SpUsxZ84cnD9/Hlu3btV5UCIyL41NamyMS4VCocDi8EBYWvCotUwal8GgQYOwbt069OnTp8VJni5duuDdd9/FkCFDkJCQoPOgRGRe9py8iuz8Ciyc4Y9uXXhoWTaNyiA7OxuPPPLIA7cJDg7GzZs3NQrx3nvvYcWKFRpdh4hMV2p2KRJ+ykXI8J4Y7e8pOw5BwzKws7NDSUnJA7cpLCyEnV37f3wiKSkJe/bs0SQGEZmwimoVPv0+DT27OWLelAGy49B/aVQGo0aNwqFDh5Cfn9/q+pycHBw+fBgjR45s1+2Vl5cjJiamxbkHIjJfaiHw2b401NY3Ykl4EGytLWVHov/SqAyWLl0KlUqFJ554Alu3bkV2djYA4MyZM9i8eTPmzZuHhoYGREdHt+v2Vq9ejWXLlsHFxUXz5ERkcg6euY6LV0sxb7Ifenk6yY5Dd1EIIYQmVzhx4gT++te/oqys7P9uRKGAEAJOTk545513MH369DZvZ9euXcjKysLKlSsRGxuLM2fOYO3atZr/C4jIJFy+Xobl60/ioUAvrFzwEL9lbGQ0KoNly5Zh9OjRmDNnDo4cOYLU1FRUVlbCwcEBgwYNwtSpU+Hs3L7JpRYuXIiioiJYWlri9u3bqKmpwezZs/G3v/2t3eFLSqqgVmvUZQAADw9nFBVVanw9Q2E+7TCfdvSRr7a+EW9v+wWNTWq8tTAYTvYd/80TYx8/wHgzWlgo4O7e+h6ZRl86+/HHH9G1a1fY2dkhLCwMYWFhHQ5193cR7uwZaFIERGQ6vjh4CUXltfjrn0ZqVQSkPxqdM3Bzc0NVVZW+shCRGTp9IR9JqbcQMb4fBvZ2lR2H7kOjPYO33noLy5Ytw/vvv49p06bB29v7vh8jdXJq/8mhyMhIREZGahKFiExAQWkNvjiYiYG9XTHrYR/ZcegBNC4DIQS2bt36wCknFAoF0tLStA5HRKarofH36SasLBVYrAyEhQVPGBszjcrA29sb3t7e+spCRGZk9/EruHarEi9HDoGbS/u/iEpyaFQG27dv11cOIjIjKVeKcfCX65gyshdGDPSQHYfagdMEEpFOlVfV47Pv09HLwwlPTvaVHYfaiWVARDqjFgKfxqdB1diEJRFBsLbidBOmgmVARDqT8NM1pF8rw58eG4ie3fgDV6aEZUBEOpF18zb2nMhGcIAnHhnaQ3Yc0hDLgIi0VlPXgI1xqXBzscVz0/w575AJYhkQkVaEEPjPgUsoq6xHdHgQHOw0/ml1MgIsAyLSysmUfPySUYjHJ/aDr3cX2XGog1gGRNRhecXV2HEoEwF9u2LG2L6y45AWWAZE1CENjU3YEJcKWxtLvKgMhAXPE5g0lgERdcjXR7Nwo6gKL4QFwtXJVnYc0hLLgIg0djazCEfP3kToQ70x1NdddhzSAZYBEWmktKIOW/eno293Z8wJ4XQT5oJlQETtplYLbIpPQ6Na/He6Cf4JMRd8JImo3eITc5B5vRzPTB2I7m4OsuOQDrEMiKhdLuWWYe/pbIwL6o7xQzjdhLlhGRBRm6pqG7ApPg0ervZ4JnSQ7DikBywDInogIQS27k9HRbUKSyKCYG/L6SbMEcuAiB7ox3M3ce5yMeY+6gsfLxfZcUhPWAZEdF83Cquw80gWBvd3w9SHesuOQ3rEMiCiVtU3NOGTuItwtLPCojBON2HuWAZE1KqvDl9GQUkNFikD4eJoIzsO6RnLgIju8UtGIU6cz8OMsX0R5OMmOw4ZAMuAiFq4VVqDbQkZ6N/TBbMf6Sc7DhkIPyNGRM0am9T44KtzAASiw4NgZcn3i50FH2kiahZ3KhsZ18qwYLo/PFztZcchA2IZEBEAID2nFPuTrmFqcB8EB3SXHYcMjGVARKioUWHT92nwcnfA4tlDZMchCVgGRJ2cEAJb9qWjurYR0eFBsON0E50Sy4CokzuUfAMpV0rw1GQ/9OnuLDsOScIyIOrErhVUYtePWRju1w2TR3rLjkMSsQyIOqk6VSM2xF2Ei6MNosICoOB0E50ay4Cok/ryYCYKy2uxWBkIJ3tr2XFIMulnij766CMkJCQAAEJCQrB8+XLJiYjMX1JqAU5fLED4eB8M6tNVdhwyAlL3DBITE3Hq1Cns2bMH3333HVJTU3Ho0CGZkYjMXmFZDbb/cAkDenWBcryP7DhkJKTuGXh4eGDFihWwsfl9RkRfX1/k5eXJjERk1hqb1NgQlwpLCwUWK4NgacEjxfQ7hRBCyA4BADk5OZg/fz6++uor+Pj4yI5DZJa2xqci9lgWVi54CA8P7Sk7DhkR6ecMAODy5cuIjo7G8uXLNSqCkpIqqNWad5mHhzOKiio1vp6hMJ92mK91F6+WIPZYFiaN8MaAHvfPwPHTnrFmtLBQwN3dqfV1Bs5yj19//RXPP/88Xn/9dTz++OOy4xCZpdtV9fjs+zR4ezjiqcl+suOQEZK6Z5Cfn4+lS5ciJiYG48aNkxmFyGyphcBn36ehTtWEv8wPgo21pexIZISklsHmzZtRX1+PtWvXNi+bN28e5s+fLzEVkXn54UwuUnPK8Nz0QfD2aP0QAZHUMli1ahVWrVolMwKRWbuaV4HY41cxapAHQobxhDHdn/RzBkSkH7X1jdi49yJcnWzw/Ax/TjdBD8QyIDJDQgh8/sMllNyuR3T4YDjacboJejCWAZEZOn2hAD+n3ULEI/3g16uL7DhkAlgGRGYmv6QaXxy6BP8+rggb21d2HDIRLAMiM9LQqMbGuFTYWFniRWUQLCx4noDah2VAZEZ2HctCbmEVosIC0NXZVnYcMiEsAyIz8VtWMQ4n38Bjo3phuF832XHIxLAMiMxAWWU9tuxLRx9PJzwxidNNkOZYBkQmTq0W+DQ+FQ2NakRHBMHaii9r0hyfNUQmbt9P15CRW46npw5ED3dH2XHIRLEMiExY1o3biDuZjTGB3TF+iJfsOGTCWAZEJqqmrgEb916EexdbPDdtEKebIK2wDIhMkBAC2xIyUF6lQnT4YNjbGsXvVJEJYxkQmaDj5/OQfKkIkRP7o39PF9lxyAywDIhMzM2iKnx1+DKC+rlh2pg+suOQmWAZEJkQVUMTNuxNhb2NJRaFBcCC5wlIR1gGRCbk66NZuFlUjUWzAtHFidNNkO6wDIhMxK+XCvHjuZuYHtwHg/u7y45DZoZlQGQCSm7XYev+DPTr4YzIkP6y45AZYhkQGbkmtRqb4lOhFgLR4UGwsuTLlnSPzyoiIxd/OgeXb9zGs9MGwbOrg+w4ZKZYBkRG7FJuGeITczB+sBfGBXG6CdIflgGRkaqqbcCm+DR4utrj6dCBsuOQmWMZEBkhIQS27EtHRbUKSyIGw86G002QfrEMiIzQ0bM38VtWMZ6Y5Ie+Xs6y41AnwDIgMjK5tyrx9dEsDPV1x9TRvWTHoU6CZUBkROpVTdi4NxWO9laICgvgtNRkMCwDIiOy43AmCkpqsHhWIFwcbGTHoU6EZUBkJM6k38LJlHzMHNcXAT5usuNQJ8MyIDICReW1+M+BDPh6uyBiQj/ZcagTYhkQSdbYpMbGvakAFIhWcroJkoPPOiLJvjuZjat5FXh+hj+6udrLjkOdFMuASKLUnFIk/HQNE4f1xEP+nrLjUCfGMiCSpKJahc/i0+Dl7oD5jw2QHYc6OZYBkQRqIfDZvjRU1zXizxGDYWttKTsSdXLSyyA+Ph4zZ85EaGgovvzyS9lxiAzi0C/XcfFqKeZN8UMvTyfZcYggdfarW7duISYmBrGxsbCxscG8efMwZswY+Pn5yYxFpFdZ18vx7bErGDGgGyaN8JYdhwiA5DJITEzE2LFj4erqCgCYNm0aDhw4gJdeeklv91lb34jTKXm4fbtWb/ehLZe8ClRU1MmOcV/M13FCCMSdzoGLow0WzuR0E2Q8pJZBYWEhPDw8mi97enoiJSWl3dd3d9d89/q741nYvDdV4+sR6YqVpQJ/j34Y/foY77eMPTyMe6ZUY88HmEbGu0ktA7Va3eKdkRBCo3dKJSVVUKuFRvc5LsATo/y7o6SkSqPrGVJXN0eUlVbLjnFfzKedvr27orGuAUVFlbKjtMrDw9loswHGnw8w3owWFor7vomWWgZeXl5ITk5uvlxUVARPT/1+1tpCoUBvT2fYST91fn8eHs5wsDTewwfMp52uznYoqmuQHYOoBal/Eh9++GEkJSWhtLQUtbW1OHjwICZOnCgzEhFRpyR1z6B79+5YtmwZnnvuOTQ0NGDu3LkYOnSozEhERJ2S9B9WVSqVUCqVsmMQEXVqRnzknIiIDIVlQERELAMiIjKCcwbasLDo+McHtbmuITCfdphPO8ynPWPM+KBMCiGEZt/aIiIis8PDRERExDIgIiKWARERgWVARERgGRAREVgGREQElgEREYFlQEREYBkQERFMfDqKB7l06RJee+017Nu3r3nZli1b8M0330AIgddffx2hoaH3XC8xMRH//Oc/UV9fjxkzZmDZsmV6y7h69WqcP3+++XJmZiZiYmIwffr0Fts9++yzKC0thZXV7w/XmjVrMGzYML3luqOhoQFjxoxB7969m5fFxsbC0tKyxXbtGVd9qK6uxt/+9jdcvXoVALBkyRKEhYXds52s8YuPj8cnn3yCxsZGLFiwAE8//XSL9enp6fif//kfVFdXY/To0Xj77bebMxrCRx99hISEBABASEgIli9ffs/63bt3w8XFBQDw5JNP3vNv0Ke2HjdDvlb/aNeuXfjiiy+aL9+4cQMRERFYvXp18zLZ46cxYYb27NkjJkyYICZNmtS87Pz58yIiIkLU1dWJ4uJiMWXKFFFWVtbierW1tSIkJETk5uaKhoYGERUVJY4dO2aQzLt27RJRUVFCrVa3WK5Wq8WECRNEQ0ODQXLc7cKFCyIqKuqB27RnXPXlgw8+EGvXrhVCCFFcXCzGjx8vioqKWmwja/wKCgrEpEmTRFlZmaiurhZKpVJcvny5xTZhYWHi3LlzQgghVq5cKb788kuD5Tt9+rR46qmnRH19vVCpVOK5554TBw8ebLFNdHS0OHv2rMEy3a2tx03ma/WPMjMzxdSpU0VJSUmL5TLHryPM7jBRZWUljhw5gg8++KDF8hMnTmDq1KmwtbWFu7s7goODcezYsRbbpKSkoG/fvujduzesrKygVCpx4MABvWcuKyvDhx9+iDVr1kChaDmR1J13vVFRUQgPD2/xbkTfLly4gNLSUkRGRuLJJ5/EmTNn7tmmPeOqL8HBwXj22WcBAO7u7nB1dUVxcXGLbWSNX2JiIsaOHQtXV1c4ODhg2rRpLZ5LN2/eRF1dHYYPHw4AiIyMNMhz7Q4PDw+sWLECNjY2sLa2hq+vL/Ly8lpsc/HiRWzcuBFKpRJr1qxBfX29wfK19bjJeq225q233sKyZcvg5ubWYrnM8esIsysDZ2dnrF+/Hj169GixvLCwEJ6ens2XPTw8UFBQcM82Hh4ezZc9PT1x69Yt/QYGsG3bNoSFhcHb2/uedRUVFRg3bhz+/e9/Y9u2bdi5cydOnz6t90wAoFAoMGXKFHz99dfNT/jS0tIW27RnXPVl/Pjx6NmzJwBg//79UKlU8PPza7GNrPFr67n0x/UeHh4Gea7dMWDAgOYiysnJQUJCAkJCQprXV1dXIyAgAH/5y1+wZ88eVFRU4OOPPzZYvrYeN1mv1T9KTExEXV0dZsyY0WK57PHrCLMrg/tRq9X3LLOwsLhnm7vfmQsh7nmnro9cu3fvxoIFC1pdP2LECLz//vtwdnaGm5sb5s6di+PHj+s10x3z5s3DSy+9BGtrawQGBmLo0KE4e/Zsi23aM676lpCQgH/84x/48MMP7znmLmv82nouyXiuteby5cuIiorC8uXL4ePj07zc0dERn376KXx9fWFlZYWoqCiDPe+Ath83Yxm/nTt3YuHChfcslz1+HWEWZbBu3TpEREQgIiICR44caXUbLy8vFBUVNV8uKipq8Y62vdvoOuu5c+fg4+MDLy+vVrdPTk5GUlJS82UhhF5PMt6db8+ePcjNzW1x39bW1i22N8SY3S/fkSNHsH37drz33nvYvHkz/P3979ne0ON3R1vj8sf1xcXFeh231vz66694/vnn8frrr+Pxxx9vsS4vLw/ffvtt82VDjdsdbT1uhn7etUalUuGXX37B5MmT71kne/w6RNbJCn27fv16qyeQa2pqRElJiZg0aZIoKChocZ26ujoxceJEkZOTIxobG8ULL7wg9u/fr9ecn332mfjggw/uu/7o0aNi9uzZoq6uTlRWVgqlUmmwk1Jr1zNa17MAAANgSURBVK4Vb775phBCiCtXroiJEyeKysrKFtu0Z1z15dChQ2Ly5MkiLy/vvtvIGr87J5BLSkpETU2NCA8PF+fPn2+xTVhYmEhOThZCCLFq1Srx6aef6j3XHXl5eWLMmDEiMTGx1fUlJSUiODhY5ObmCrVaLVauXCk2bNhgsHxtPW4yXqt/lJKSIubNm9fqOtnj1xGdpgyEEGLz5s1i5syZIjQ0VOzZs6d5eXh4ePMfsMTERKFUKkVoaKh499137/l0j669+eabYseOHS2WFRQUiPDw8ObLMTExYvr06SI0NFRs27ZNr3nuVllZKV5++WURFhYmZs2aJZKSkoQQv78IFi1a1Lzd/cZV35RKpRg/frwIDw9v/i8lJcVoxm/v3r0iLCxMhIaGik2bNgkhhFi0aJFISUkRQgiRnp4u5syZI6ZNmyZee+01UV9fb7Bsf//738Xw4cNbjN2OHTta5Dtw4EBz/hUrVhg0nxCtP24yX6t/tG/fPvHqq6+2WGZM46cp/tIZERGZxzkDIiLSDsuAiIhYBkRExDIgIiKwDIiICCwDIiICy4CIiMAyICIisAyIiAgsAyKtJCcnw9/fH48++iiqq6ubl6tUKiiVSgQEBCA5OVliQqL2YRkQaWH06NF45plnkJ+fj/Xr1zcvX79+PTIzMxEVFYXRo0dLTEjUPpybiEhLtbW1UCqVyM/Px+7du6FSqTBv3jz4+vpi9+7dsLGxkR2RqE0sAyId+Pnnn7FgwQKMGjUK5eXluHbtGr799ttWf2OByBjxMBGRDowZMwbz589HcnIysrKy8Morr7AIyKSwDIh0JDQ0tPn/R4wYITEJkeZ4mIhIB+rr6zF79mzk5eUBAHr06IG4uDjY2tpKTkbUPtwzINKBdevW4erVq1i6dCmio6ORnZ2NdevWyY5F1G7cMyDSUkpKSvOnh2JjYyGEwOzZs5GTk4OdO3di6NChsiMStYl7BkRaUKlUWLlyJdRqNd5++21YW1vDxsYGa9asgVqtxsqVK6FSqWTHJGoTy4BIC+vXr0dWVhaefPJJjBw5snn56NGjMWfOHGRlZbX4MhqRseJhIiIi4p4BERGxDIiICCwDIiICy4CIiMAyICIisAyIiAgsAyIiAsuAiIjAMiAiIrAMiIgIwP8H4WQIwC1OAqoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Naive scalar relu implementation. \n",
    "# In the real world, most calculations are done on vectors\n",
    "def relu(x):\n",
    "    if x < 0:\n",
    "        return 0\n",
    "    else:\n",
    "        return x\n",
    "\n",
    "\n",
    "plt.plot(range(-10, 10), [relu(i) for i in range(-10, 10)])\n",
    "plt.xlabel('x', fontsize = 20)\n",
    "plt.ylabel('relu', fontsize = 20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Softmax\n",
    "\n",
    "The softmax function, also known as softargmax or normalized exponential function, is a function that takes as input a vector of K real numbers, and normalizes it into a probability distribution consisting of K probabilities. \n",
    "\n",
    "$$softmax = \\frac{e^x}{\\sum e^x}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T07:32:40.058795Z",
     "start_time": "2020-05-24T07:32:40.052704Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.02364054, 0.06426166, 0.1746813 , 0.474833  , 0.02364054,\n",
       "       0.06426166, 0.1746813 ])"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def softmax(s):\n",
    "    return np.exp(s) / np.sum(np.exp(s), axis=0)\n",
    "\n",
    "softmax([1.0, 2.0, 3.0, 4.0, 1.0, 2.0, 3.0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T07:32:41.063124Z",
     "start_time": "2020-05-24T07:32:40.843218Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAESCAYAAAABl4lHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deZhcdZ3v8XdVd6c7vWRrOntC2PLNggJhmVFAFAIKjDIILoOOouMEZpy7zOY4ogLewesz15nxceQqwuByNaIyI4rGBUxwRESWYZHu5MtiN2TpJJ2uTtLdSa917h+nOl1VdNJVne46VdWf1/Ng19m//B6sT53zO+d3YkEQICIikot41AWIiEjpUGiIiEjOFBoiIpIzhYaIiORMoSEiIjmrjLqAKVQNnAu0A8MR1yIiUioqgEXA40B/9sJyDo1zgV9GXYSISIm6EHg4e2Y5h0Y7QFdXL8lk/s+iNDbW09nZM+lFlSq1Rya1xyi1RaZSb494PMbcuXWQ+g7NVs6hMQyQTAYTCo2RbWWU2iOT2mOU2iJTmbTHmJf11REuIiI5U2iIiEjOFBoiIpIzhYaIiORMoSEiIjlTaIiISM4UGiIiZSSZDPiHrz/BY1v3TMn+FRoiImXk5T3d/G7XwSl7VkShISJSRlraEgCsWTFvSvav0BARKSPNrQmWza9nVt2MKdm/QkNEpEz0Dwzzwo4DrD1pas4yQKEhIlI2fPt+hpMBa6fo0hQoNEREykZLW4LKijinLZ09ZceIfJRbM7sO+DhQBXzO3W/PWm7AHcBcYDfwbnfvKnihIiJFrrk1wcpls5lRVTFlx4j0TMPMlgC3ARcAZwIbzGxN2vIY8APgM+5+BvAU8NEoahURKWZd3f3s3Nc7pf0ZEP3lqfXAZndPuHsvcC9wbdrydUCvu/8kNf1p4HZERCTDyK22U9mfAdFfnlpM5tuh2oHz0qZPBXab2b8BZwFbgf9WuPJEREpDS1uChtoqls6vn9LjRB0acSD9scUYkEybrgTeCLzB3Z8ws/8F/DNwfa4HaGyceAM2NTVMeNtypPbIpPYYpbbIVOj2CIKAra/sZ50tYMH8WVN6rKhDYwfhy8tHLAR2pU3vBl5w9ydS098ivISVs87Ongk9Tt/U1EBHR3fe25UrtUcmtccotUWmKNpj+94e9nf3c8qi4z92PB475o/tqPs0HgQuMbMmM6sFrgF+krb8EaDJzM5ITb8VeLLANYqIFLXm1lR/xhR3gkPEoeHuO4GbgC3A08BGd3/MzDaZ2Tnufhi4GrjTzJqBi4G/jq5iEZHi09KWYFFjLXMbqqf8WFFfnsLdNwIbs+Zdkfb5N2R2jouISMrg0DC+fT8Xnbm4IMeL+vKUiIgchxd2HGBwKDnlt9qOUGiIiJSw5rYEFfEYtnxOQY6n0BARKWHNrQlOXTKbmhmF6W1QaIiIlKiDhwZ4ZU8Pawpw19QIhYaISIna2haO3Vqo/gxQaIiIlKzmtgR1NZWsWFi4J9AVGiIiJSgIAppbE6w+cS7xeKxgx1VoiIiUoN2JQ3R19xe0PwMUGiIiJenI0CEF7M8AhYaISElqbk0wf85MmubMLOhxFRoiIiVmaDjJtu37CzJAYTaFhohIifndroP0DwyzpsCXpkChISJScppbE8RisPrEwgwdkk6hISJSYprbEpy8eBa1NVUFP7ZCQ0SkhPT2DdLafrDgd02NUGiIiJSQbS93EQRE0p8BCg0RkZLS3NZFzYwKTl48K5LjKzREREpIc2snq5bPpbIimq9vhYaISInYu/8wHfv7Ink+Y4RCQ0SkRLSkhg5Zs2JuZDUoNERESkRza4J5s6pZOK82shoUGiIiJSCZDNj6chdrV8wjFivcUOjZCvNS2WMws+uAjwNVwOfc/fas5TcDHwS6UrPuzF5HRKTcte4+yKH+oUj7MyDi0DCzJcBtwNlAP/CImW1x95a01c4B3u3uv46iRhGRYtDSmiAGrD4xuv4MiP7y1Hpgs7sn3L0XuBe4Nmudc4CPmdmzZvYFM6speJUiIhFrbk2wfGEDDbUzIq0j6tBYDLSnTbcDS0cmzKweeAr4W2AdMAf4RCELFBGJ2uH+IV7aFd3QIemi7tOIA0HadAxIjky4ew9wxci0mf0TcDdwU64HaGysn3BxTU2Fe1l7KVB7ZFJ7jFJbZJrs9nisZTfDyYDXn7kk8raOOjR2ABemTS8Edo1MmNlyYL27352aFQMG8zlAZ2cPyWQw/opZmpoa6Ojoznu7cqX2yKT2GKW2yDQV7fHI0zuZURmnqX7GlLd1PB475o/tqEPjQeAWM2sCeoFrgA1pyw8D/2hmW4A24MPA9wpdpIhIlFraEqxcPoeqyqh7FCLu03D3nYSXmrYATwMb3f0xM9tkZue4ewdwA3A/4IRnGv8UWcEiIgWWONhHe+ehoujPgOjPNHD3jcDGrHlXpH3+d+DfC12XiEgxaG4Lhw4pltCI/lxHRESOqrk1wey6GSxpqou6FEChISJStJJBQEtbF2siHjoknUJDRKRIbd/TQ8/hQdaeFO1T4OkUGiIiRaqlbWQo9OLozwCFhohI0XquNcHSpjrm1FdHXcoRCg0RkSI0MDjMCzsOFNVZBig0RESK0vM79jM0nIx8KPRsCg0RkSLU3JqgsiLGymVzoi4lg0JDRKQINbd2cdrSOVRXVURdSgaFhohIkTnQO8COjh7WrCieW21HKDRERIrMyK22xdafAQoNEZGi09yaoH5mFcsXFN97ShQaIiJFJAgCmtsSrFkxl3iRDB2STqEhIlJEdu3r5UDPQNE9nzFCoSEiUkSaW4trKPRsCg0RkSLS3NbFwnm1NM6uibqUMSk0RESKxOBQEt/eVbRnGaDQEBEpGi/tPMDAYJI1RTQUejaFhohIkWhuSxCPxVi1XKEhIiLjaG5NcMqSWcysroy6lKPKKzTM7N05rDPXzL458ZJERKafnsODvLy7u6j7MyD/M42NZvYtMxtz2EUzuxpoAcYNFxERGbX15S4CYE0RDh2SLt/QeAB4F/BbM7t0ZKaZNZrZPcC9wGzgk5NXoohI+Wtu7WRmdSUnLSq+oUPS5RUa7v5m4M+AOcBPzOxfzew6wrOLdwKbgde6+2257tPMrjOzFjN7wcw+fIz1rjSz1nzqFREpBUEQ0NzaxeoT51IRL+6u5ryrc/c7gNOBLcCfA/8PqATe6+6XuvuLue7LzJYAtwEXAGcCG8xszRjrLQA+CxTfQCwiIsdpb9dhOg/2sbYIh0LPNtFIWw0sY/RLfDZwvpnNynM/64HN7p5w917Cy1vXjrHeXcCtE6xVRKSoPZcaOqTY+zMgPEPImZktAj4PvB3oBz4CfA+4k/Cy1R+a2f909+/muMvFQHvadDtwXtYx/zvwX8Cj+dQ6orGxfiKbAdDUVNzXFgtN7ZFJ7TFKbZEp3/Z4cddBFsyrZe1p84kV4ci26fK9GXgrMAv4NfABd38+Nf9iM/tz4DPAPWZ2vbtfmcP+4kCQNh0DkiMTZnY6cA1wCbA0z1oB6OzsIZkMxl8xS1NTAx0d3RM5ZFlSe2RSe4xSW2TKtz2Gk0mefbGD81YvYN++nimsLDfxeOyYP7bzvTxVBfwVcEFaYADg7v+X0b6Ot+S4vx3AorTphcCutOl3pJY/AWwCFpvZL/OsWUSkaLXu6uZw/3DRP58xIt8zjTOO1dHt7q8A681sQ477exC4xcyagF7Cs4oj27r7zcDNAGa2AnjI3S/Ms2YRkaL1XGsnMWDVicXfCQ7533Kb651RD+S4v53ATYRnJ08DG939MTPbZGbn5FObiEgpamnrYsWiWdTPrIq6lJzkPcCJmV0BXAfMByoYvYMqRnj5qhE4Ldd9u/tGYGPWvCvGWK8NWJFvvSIixepQ3xC/23WQK163POpScpbv3VNvB77LsZ+X6AV+cDxFiYhMB9te6SIZBCXTnwH5d4T/FTBE+PT3QuApwtttFwIXA08S3g31d5NYo4hIWWpuS1BdVcEpS2ZHXUrO8g2N1wD3ufu97r4XeJjwTqq97v4Q8GbC5zdumtwyRUTKT0trAls+h8qK4h46JF2+ldYA6Z3h24CVZlYN4O4J4D7g9yenPBGR8rRv/2H2dB0uqUtTkH9o7AGa0qZfSu1jbdq8fUzwQTwRkemiua10hg5Jl29o/AK4xsxWpqafSf29Km2d84HE8RYmIlLOmtu6mNtQzeLG2qhLyUu+ofEZYCbh+zSudfc9wP3Ax8zs22b2EGFo5PSchojIdJRMBmxtS7BmxdyiH2sqW74P9zUDbyR8b8aB1Oy/IOzbeAfwBuBx4O8nr0QRkfLy8p5uevuGSq4/AybwcJ+7PwZcnja9A3iNmb0W6ANecPf8RwgUEZkmmkeGQp8OoXE07v7sZO1LRKSctbQlWD6/nll1M6IuJW8TGUbkDODdhEN6VB9ltcDdrzmOukREylL/wDAv7DjApecui7qUCcl3GJFrgHsIx5w6Fl2eEhEZg2/vYjhZWkOHpMv3TONmwn6LDwO/SX0WEZEcNbd2UVUZ57SlpTN0SLp8Q+MU4G53//pUFCMiUu5a2hKsXDqbGVXjXbApTvk+p/EyR+/HEBGRY+jq7mfnvt6Sewo8Xb6h8Vng3al3d4uISB5aUkOHlGp/BuR5ecrd7049j/FfZvYA0EY4qm22wN3/ehLqExEpG81tCWbVVrF0fn3UpUxYvndPXQbcmNru8mOsGgAKDRGRlCAIaGnrYs2KecRLbOiQdPl2hH+G8Hbb/w38ivAtfSIiMo4dHb0c7B0oyafA0+UbGquAr7u7XrIkIpKHkaFD1pZwJzjk3xG+G+iZikJERMpZc1uCxSfUMbehtG9AzTc0vkx499SSqShGRKQcDQ4N8/z2/axZMTfqUo5bvpenHgHeCTxnZv9B+OrXMfs13P3zuezQzK4DPg5UAZ9z99uzll8N3ErYl/I4sMHdB/KsW0QkMs/vOMDgULKkb7UdkW9oPJT2+QPHWC8Axg2N1BnLbcDZhLfuPmJmW9y9JbW8DvgCsM7d95jZPcD1hGc8IiIloaU1QUU8hi2fE3Upxy3f0PggkzsY4Xpgs7snAMzsXuBa4FMA7t5rZivcfdDMaoH5QNckHl9EZMo1tyU4dclsamZM2tsoIpPvw31fzWW91Bd8LhYD7WnT7cB5WcccNLPLgW8AO4Gf5bhvEZHIHTw0wCt7erj6DSdHXcqkyPfhvt8B/+Lu/3qMdT5J+ArY+TnsMk7mmUsMSGav5O4/BhrN7NPAF4Hrcq25sXHiT142NTVMeNtypPbIpPYYpbbIlN4eLf+1A4ALzlpaFu10zNAwsxXArLRZK4DVqaFExjKD8JJTXY7H3wFcmDa9ENiVdvx5wDnuPnJ28U3g2znuG4DOzh6SyfyvqDU1NdDR0Z33duVK7ZFJ7TFKbZEpuz0efXYXdTWVzK6uKIl2isdjx/yxPd6Zxu8DGxk9GwiAG1L/HE2M3C8hPQjcYmZNhHdhXQNsyNrXN8zsHHd/BXgH8HCO+xYRiVQQBDS3JVh94lzi8dIdOiTdMUPD3e8xs7MILzXFgPcBzwBPj7F6AAwS9jvcPsbysfa/08xuArYQnqXc5e6Pmdkm4JPu/oSZbQB+aGYB0EI49pWISNFr7zxEV3d/SQ+Fnm28y1MXAf/k7nvTpr+S6zMYuXD3jYRnM+nzrkj7fB9w32QdT0SkUJrLYCj0bOM9Ef5d4C/Tph8iHA5dRETG0dKaYP7cmTTNmRl1KZNmvNBoILMj/P3AGVNXjohIeRgaTrJt+/6yOsuA8TvCXwA+YGb1QGdq3uVmNt4AKnoJk4hMay/tPED/wHDJD4WebbzQ+CjhJao/Tk0HhHdU/f442+klTCIyrTW3dRGPxVh9YukPHZJuvLunNqXGh1oF1ACbga8CX5v60kRESldLW4KTFjdQW1MVdSmTatwnwt19P/AogJn9AnjI3X8x1YWJiJSq3r5BWtsP8tbXr4i6lEmX79hTbxr5bGbLCTvFawn7O1rcfdfRthURmS62tnURBKX/lr6x5D3kYmpokTuBi7MWBWa2GbjB3VsnoTYRkZLU0pagZkYFJy2aNf7KJSbfAQsXAr8CFhG+EOlXhGNFzQUuIhx36hdmts7d901yrSIiJaG5LcGq5XOprMj35ajFL98zjZsJA+PP3P2O7IVm9iHCFyR9DPir4y9PRKS0tO/rpWN/H5eduzzqUqZEvjF4JfDAWIEB4O53AQ8AVx1vYSIipejp5/cC5dmfAfmHxgLg2XHW+S3hy5VERKadp57voHFWNQvmls/QIenyDY09wNHepTHitYD6M0Rk2hlOJnn2xX2sWTGPWKw8hkLPlm9obALWm9kHxlpoZjcAlwA/Ot7CRERKTVt7N72HB8v20hTk3xF+C/CHwF1m9j7gl8ABwstRFwJnE56NfGoSaxQRKQnNbQliMVh94njD85WufB/u221m5xPeIfUmwtts020hfE5DD/mJyLQyNJzk8W17OWXJbBpqZ0RdzpTJ++E+d38JuCQ1JtVZhEOn9wAJ4Gl375ncEkVEit+9D73Ezo5ePnb9eVGXMqXyfvLEzN5gZvcAu939h0Az8AXgF8BeM7t5kmsUESlqz7y4j589vp2L1y3hda9ZFHU5Uyqv0DCzi4GfA+8AlqVm3wksJbw01QZ80szeO4k1iogUra7ufv7tR1tZNr+ed118atTlTLl8zzQ+AnQD57l7m5mtBs4Bfuru64EzgW3Ahye3TBGR4pNMBtx5fzODQ0luvGotVZUVUZc05fINjXOBe9z9ydT0HxC+cOk7AO4+APwEWDtpFYqIFKkfPtLGtlf2897LVrKosS7qcgoi39CoIbzFdsTlqb8PZO1z6HiKEhEpdv5KF9//VSuvW7uQ88u8HyNdvndPvQj8HoCZLQXOB5rdfUdq3gzC8aleynWHZnYd8HGgCvicu9+etfwq4FYgBrQCH3D3rjzrFhGZNN2HBvjy/S3MnzOT9162MupyCirfM43/AN5oZlsI75aqBL4CYGZXAr8GTiHsHB9X6rbd24ALCPtDNpjZmrTls4AvAle6+xmE417dkmfNIiKTJggC7v7RVroPDXDjVaczszrvJxdKWr6h8Q/AHYRPf68Avg18PrXs9YRv8vtncgwNwvdvbHb3hLv3AvcC16YtrwI+7O47U9PPAuU53rCIlIQHn9jBMy918s43ncqJCxuiLqfg8n0ifBj4MzP7CBB39/T+jTuBz7v7njx2uRhoT5tuB448GePuncD3AMxsJvBR4F/zqVlEZLK07T7Id7a8yJmnnsAlZy+NupxITOi8yt27x5jXNoFdxQnvvhoRA5LZK5nZbMLweMbdv5bPARob6ydQVqipafr9ijgWtUcmtceo6dAWh/oGufPO3zC3oZq/fd+5zKo7+lAh5dweUV+M20F4qWvEQsLXxx5hZouAnwKbgb/M9wCdnT0kk8H4K2Zpamqgo+NV2ThtqT0yqT1GTYe2CIKAO+9vYXeil7+7bh39h/rpONQ/5rql3h7xeOyYP7ajDo0HgVvMrAnoBa4BNowsNLMK4H7gO+7+D9GUKCLT3cO/befRlj1cfeFJrFw2J+pyIhVpaLj7TjO7iXAIkhnAXe7+mJltAj5JOFTJOqDSzEY6yJ9w9w9FU7GITDe79vXyzQeeZ9XyOVz5uhVRlxO5qM80cPeNwMaseVekPj7BBAZVFBGZDAODw3zp+81UV1Xwp29dSzxenm/jy4e+kEVEjuLbm19kR0cPf3LlGuY2VEddTlFQaIiIjOGJbXvZ8tRO3nLecl57SmPU5RQNhYaISJZ9+w/zlR9v46RFs3j7RSdHXU5RUWiIiKQZGk5yxw+agYAbrlpLZYW+JtOpNURE0tz3y1Ze2nWQ979lFfPnzIy6nKKj0BARSXmutZNNj77MG85YzHmrF0RdTlFSaIiIAAd6+rnr/haWnFDHH60/LepyipZCQ0SmvWQQcNcPW+gbGObGq9ZSXVX+r22dKIWGiEx7P370ZZrbuvij9aexpGnig5xOBwoNEZnWXtx5gO/9ZyvnrprPG85YHHU5RU+hISLTVm/fIHd8/znmzarm/W9ZRSymYULGo9AQkWkpCAK+umkb+3vC17bW1kQ+FF9JUGiIyLT00FM7efL5Dq656BROXjwr6nJKhkJDRKadV/Z0862fv8hrTm7ksvOWRV1OSVFoiMi00j8QDndeN7OSP7lyNXH1Y+RFoSEi08o3H3iePYlDbPiDNcd8z7eMTaEhItPGr5t38/Bv2/mD169g9Yp5UZdTkhQaIjIt7Ekc4us/dU5bOpu3XbAi6nJKlkJDRMre4FCSL32/mcp4jBvetpaKuL76JkotJyJl77sPvcjLe7r54JWrmTerJupySppCQ0TK2lMvdPDgEztYf/ZSzjqtKepySp5CQ0TKVuJgH3f/aCvLF9TzjjedGnU5ZUGhISJlaTiZ5Ms/aGZoOODGq06nqlJfd5Mh8lY0s+vMrMXMXjCzDx9jva+b2fUFLE1EStj9v2rj+R0HeN+bjYXzaqMup2xEGhpmtgS4DbgAOBPYYGZrstZZbGb3A9dGUKKIlKCtL3dx/6/aOP/0hbzu9IVRl1NWoj7TWA9sdveEu/cC9/LqcHgP8H3gO4UuTkRKz8FDA3z5/mYWzKvlPZetjLqcshP1WMCLgfa06XbgvPQV3P3/AJjZBRM5QGPjxN/C1dTUMOFty5HaI5PaY1SxtEUyGXD7fY9yqG+IT214PcuWzI6kjmJpj6kQdWjEgSBtOgYkJ/MAnZ09JJPB+CtmaWpqoKOjezJLKWlqj0xqj1HF1BY/fewVnty2l/dcupKGGfFI6iqm9piIeDx2zB/bUV+e2gEsSpteCOyKqBYRKWGt7Qe596GXWLeyiYvXLYm6nLIV9ZnGg8AtZtYE9ALXABuiLUlESs2hviG+9P3nmFM/gw9code2TqVIzzTcfSdwE7AFeBrY6O6PmdkmMzsnytpEpPglg4DHt+3l0994ks4D/Wx421rqaqqiLqusRX2mgbtvBDZmzbtijPWuL1RNIlLckkHAk97BDx5uZee+XhY11vIXb38Npy2dE3VpZS/y0BARydVYYXHD29Zy7qr5xOO6JFUICg0RKXoKi+Kh0BCRoqWwKD4KDREpOgqL4qXQEJGiobAofgoNEYncWGGx4W1rOG/VAoVFkVFoiEhkFBalR6EhIgWnsChdCg0RKRiFRelTaIjIlFNYlA+FhohMGYVF+VFoiMikU1iUL4WGiEwahUX5U2iIyHFTWEwfCg0RmbDBoWEefmYn39i0VWExTSg0RGRcQRBwoHeA7Xt7Mv7Z3XmIZBAoLKYRhYaIZBgaTrK78xCv7O3OCIjuQ4NH1pk3q5rl8xtYt7KJs1Yt4MQTahUW04RCQ2Qa6zk8yPY9meGwq7OXoeEAgMqKOEtOqOOMU05g2fx6ls2vZ+n8eupnjr5StampgY6O7qj+FaTAFBoi00AyGbCn69CrLi91dfcfWWdW3QyWza/n0pPmHQmIhY21VMTjEVYuxUahIVJmDvcPvSocdu7rYWAwCUA8FmPRCbXY8jlHwmHZ/AZm182IuHIpBQoNkRIVBAH7DvRlhMMre7rZd6DvyDp1NZUsm1/PRWcsORIQi0+oo6pSZw8yMQoNkYgFQUDfwDC9fYP0Hh7iUN8gvX1D9PYNcqhv6Mjn3r7UssPhdPfhQfoHhgGIAfPn1bJi0SwuPGMxy+bXs3x+PXMbqonF1EEtkyfy0DCz64CPA1XA59z99qzlZwJ3AbOA/wRudPehghcqcgxBEDAwlAy/5A8PHvnC78n64j80EgBp4XCob4hkEBx13xXxGHU1ldTNrKK2ppLZ9TNYfEItdTOrWNxYF3ZON9VTPaOigP/GMl1FGhpmtgS4DTgb6AceMbMt7t6Stto3gA+5+6Nm9m/AnwJfLHy1UuyCIGA4GTA8HDCUTDI0lGQo+/NwMvwnGRyZN5xMMjiUZDiZWj6UtjyZ2m5kXmr74QC6DvRlBMHIHUdjicWgtjr84q+rqaS2poqmOTXU1YRBUFczOr9+Zvi3LjV/RlVcZwtSNKI+01gPbHb3BICZ3QtcC3wqNX0iMNPdH02t/1XgVqY4NJJBwBNb97BnjNsI038QBmRMjPUxa37ml8rRfly+6hjB6G6C1MIg9T9BaoMge7vUvPR1RrYbWe+o+87aL0DtzBn09PaTDML1kkFAkAzbKghG/mZ+TiZH1k3bJhjdJlwnPEYyGWStm7nNyP6Gk1lf/kc+BwwPJ7Na+PjFgMrKOJUVcSorYkf+NtRVU10ZZ/EJdWlf+COhEAZB/ZFAqKSmupK4vvilDEQdGouB9rTpduC8cZYvzecAjY31eRf14vb93HrXo+OvOA3FYxCLxYjHY8RiMSri4XQsFiMeixFPTcdjsXDd+Mjn0WXhdjFiafs6st9YjIqKOFWxcHn8yLHC5ZUVMaoqK458gVdVxqmsjFNVkfY3bf6Rz+l/R9bNmp+5XXicCj2wlpOmpoaoSygq5dweUYdGnMwf5jEgmcfycXV29pBM5vf7c3ZNBXfddCntew5mHPjI56N8j6RfQshYJTbmx1ftLOMYWdvHiB1ZNeM4sdS6sVjq7+j2I+sdWYfR6dH5sdT+08uJvWqdpqYGOjt7xvz3LgtBEoaSDA7B4Phr64G2NGqLTKXeHvF47Jg/tqMOjR3AhWnTC4FdWcsXHWP5lFkwr5b48HAhDlUSNESEiED4Sz5KDwKXmFmTmdUC1wA/GVno7i8DfWZ2fmrWHwM/LnyZIiICEYeGu+8EbgK2AE8DG939MTPbZGbnpFZ7D/AvZrYNqAc+H021IiIS9eUp3H0jsDFr3hVpn58hs3NcREQiEvXlKRERKSEKDRERyZlCQ0REchZ5n8YUqoDju1VUt5lmUntkUnuMUltkKuX2SKt9zMHMYkEw2QMvFI0LgF9GXYSISIm6EHg4e2Y5h0Y1cC7h0CN6Sk9EJDcVhA9VP044kGyGcg4NERGZZOoIFxGRnCk0REQkZwoNERHJmUJDRERyptAQEZGcKTRERCRnCg0REclZOQ8jMmFmdh3wcaAK+Jy73x5xSZExs5uBd6Ymf8t4D10AAAPYSURBVOTuH4mynmJhZp8FTnD366OuJUpm9lbgZqAO+Jm7/4+IS4qMmb0X+PvU5I/d/W+irGeq6Ewji5ktAW4jHIbkTGCDma2JtqpomNl64DLgLMK2ONvMro62quiZ2SXA+6OuI2pmdjLwJeAPgdcC68zs8mirikbqzaOfBy4CzgAuTP3/p+woNF5tPbDZ3RPu3gvcC1wbcU1RaQf+2t0H3H0Q2Aosj7imSJnZPMIfFZ+OupYicDXwbXffkfrv413AbyKuKSoVhN+ndYRXKKqAw5FWNEV0eerVFhN+WY5oZ5q+OdDdm0c+m9lphJepzj/6FtPCHYSvKF4WdSFF4FRgwMx+QPhj4ofAJ6ItKRru3m1mnwC2AYeAXwCPRFvV1NCZxqvFgfQBuWJAMqJaioKZrQUeAP7W3V+Iup6omNmHgO3u/vOoaykSlYRn5n8CvA74PabpZTszey3wQeBEwh+ew4D6NKaJHYQjPI5YCOyKqJbImdn5wM+Bj7r716KuJ2LvAi4zs6eBTwFvM7N/ibimKO0GHnT3Dnc/DHyPaXpWDrwZ+Lm773X3fuCrwBsjrWiK6PLUqz0I3GJmTUAvcA2wIdqSomFmy4D7gHe5++ao64mau1868tnMrgfe6O5/GV1Fkfsh8DUzmwN0A5cT/vcyHT0D/KOZ1RFennor4dDiZUdnGlncfSfhNestwNPARnd/LNqqIvM3QA3wz2b2dOqfG6MuSoqDu/8G+EfCF/W0AC8DX4m0qIi4+8+AbwFPAs8SdoR/JtKipojepyEiIjnTmYaIiORMoSEiIjlTaIiISM4UGiIikjOFhoiI5EyhISIiOVNoiIhIzhQaIiKSM4WGiIjkTGNPiRSAmV1IOFz2DmCNu/ek5s8gHHpiDeFYVr+MrkqR8elMQ6QAUmHwBcL3cNyatuhW4HTgswoMKQUae0qkQFKvBP0t4QuLziYcDPIRwsH+znH3gQjLE8mJQkOkgMzsjcBmwpFhGwnffneuuz8bZV0iudLlKZECcveHgC8CFxL2Y3xCgSGlRKEhUnj/nva5LN8jLeVLl6dECsjMaoCnCN8lDfAKcKa790VXlUjudKYhUlifAlal/n4asNRnkZKgMw2RAjGz8wgvR20F1gExwlcKrwReP41fKywlRGcaIgVgZtWE78+OAze4+2DqFtsbUvO+klpHpKgpNEQK4xbCu6W+7O5HOr9TD/TdnVp2SySVieRBl6dERCRnOtMQEZGcKTRERCRnCg0REcmZQkNERHKm0BARkZwpNEREJGcKDRERyZlCQ0REcqbQEBGRnCk0REQkZ/8fyxYXTageFVgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(10), softmax(range(10)))\n",
    "plt.xlabel('x', fontsize = 20)\n",
    "plt.ylabel('softmax', fontsize = 20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Softmax is often used in neural networks, to map the non-normalized output of a network to a probability distribution over predicted output classes.\n",
    "\n",
    "- Prior to applying softmax, some vector components could be negative, or greater than one; and might not sum to 1;\n",
    "- After applying softmax, each component will be in the interval (0,1), and the components will add up to 1, so that they can be interpreted as probabilities. Furthermore, the larger input components will correspond to larger probabilities. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Logistic Regression \n",
    "\n",
    "![image.png](./images/nn15.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](./images/nn16.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](./images/nn17.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T12:56:55.850226Z",
     "start_time": "2020-05-24T12:56:55.846210Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from torch import tensor\n",
    "from torch import nn\n",
    "from torch import sigmoid\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "\n",
    "# Training data and ground truth\n",
    "x_data = tensor([[1.0], [2.0], [3.0], [4.0]])\n",
    "y_data = tensor([[0.], [0.], [1.], [1.]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T12:57:02.236442Z",
     "start_time": "2020-05-24T12:57:02.232697Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class Model(nn.Module):\n",
    "    def __init__(self):\n",
    "        \"\"\"\n",
    "        In the constructor we instantiate nn.Linear module\n",
    "        \"\"\"\n",
    "        super(Model, self).__init__()\n",
    "        self.linear = nn.Linear(1, 1)  # One in and one out\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        In the forward function we accept a Variable of input data and we must return\n",
    "        a Variable of output data.\n",
    "        \"\"\"\n",
    "        y_pred = sigmoid(self.linear(x))\n",
    "        return y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T12:57:13.366861Z",
     "start_time": "2020-05-24T12:57:13.363187Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# our model\n",
    "model = Model()\n",
    "\n",
    "# Construct our loss function and an Optimizer. The call to model.parameters()\n",
    "# in the SGD constructor will contain the learnable parameters of the two\n",
    "# nn.Linear modules which are members of the model.\n",
    "criterion = nn.BCELoss(reduction='mean')\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T12:57:54.191754Z",
     "start_time": "2020-05-24T12:57:53.997444Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1000 | Loss: 0.3840\n",
      "Epoch 101/1000 | Loss: 0.3748\n",
      "Epoch 201/1000 | Loss: 0.3661\n",
      "Epoch 301/1000 | Loss: 0.3579\n",
      "Epoch 401/1000 | Loss: 0.3501\n",
      "Epoch 501/1000 | Loss: 0.3427\n",
      "Epoch 601/1000 | Loss: 0.3357\n",
      "Epoch 701/1000 | Loss: 0.3290\n",
      "Epoch 801/1000 | Loss: 0.3226\n",
      "Epoch 901/1000 | Loss: 0.3166\n"
     ]
    }
   ],
   "source": [
    "# Training loop\n",
    "for k, epoch in enumerate(range(1000)):\n",
    "    # Forward pass: Compute predicted y by passing x to the model\n",
    "    y_pred = model(x_data)\n",
    "\n",
    "    # Compute and print loss\n",
    "    loss = criterion(y_pred, y_data)\n",
    "    if k%100==0:\n",
    "        print(f'Epoch {epoch + 1}/1000 | Loss: {loss.item():.4f}')\n",
    "\n",
    "    # Zero gradients, perform a backward pass, and update the weights.\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T13:01:50.071575Z",
     "start_time": "2020-05-24T13:01:50.066628Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Let's predict the hours need to score above 50%\n",
      "==================================================\n",
      "Prediction for x = 1.0, y_pred = 0.1998 | Above 50%: False\n",
      "Prediction for x = 7.0, y_pred = 0.9969 | Above 50%: True\n"
     ]
    }
   ],
   "source": [
    "# After training\n",
    "print(f'Let\\'s predict the hours need to score above 50%\\n{\"=\" * 50}')\n",
    "y_pred = model(tensor([[1.0]]))\n",
    "print(f'Prediction for x = 1.0, y_pred = {y_pred.item():.4f} | Above 50%: {y_pred.item() > 0.5}')\n",
    "y_pred = model(tensor([[7.0]]))\n",
    "print(f'Prediction for x = 7.0, y_pred = {y_pred.item():.4f} | Above 50%: { y_pred.item() > 0.5}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "####  Diabetes Classification\n",
    "\n",
    "![image.png](./images/nn18.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T13:18:56.152030Z",
     "start_time": "2020-05-24T13:18:56.137068Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X's shape: torch.Size([759, 8]) | Y's shape: torch.Size([759, 1])\n"
     ]
    }
   ],
   "source": [
    "from torch import nn, optim, from_numpy\n",
    "import numpy as np\n",
    "\n",
    "xy = np.loadtxt('../data/diabetes.csv.gz', delimiter=',', dtype=np.float32)\n",
    "x_data = from_numpy(xy[:, 0:-1])\n",
    "y_data = from_numpy(xy[:, [-1]])\n",
    "print(f'X\\'s shape: {x_data.shape} | Y\\'s shape: {y_data.shape}')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T13:19:17.672598Z",
     "start_time": "2020-05-24T13:19:17.667176Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class Model(nn.Module):\n",
    "    def __init__(self):\n",
    "        \"\"\"\n",
    "        In the constructor we instantiate two nn.Linear module\n",
    "        \"\"\"\n",
    "        super(Model, self).__init__()\n",
    "        self.l1 = nn.Linear(8, 6)\n",
    "        self.l2 = nn.Linear(6, 4)\n",
    "        self.l3 = nn.Linear(4, 1)\n",
    "        self.sigmoid = nn.Sigmoid()\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        In the forward function we accept a Variable of input data and we must return\n",
    "        a Variable of output data. We can use Modules defined in the constructor as\n",
    "        well as arbitrary operators on Variables.\n",
    "        \"\"\"\n",
    "        x = self.sigmoid(self.l1(x))\n",
    "        x = self.sigmoid(self.l2(x))\n",
    "        y_pred = self.sigmoid(self.l3(x))\n",
    "        return y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T13:20:09.530806Z",
     "start_time": "2020-05-24T13:20:09.526832Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# our model\n",
    "model = Model()\n",
    "\n",
    "\n",
    "# Construct our loss function and an Optimizer. The call to model.parameters()\n",
    "# in the SGD constructor will contain the learnable parameters of the two\n",
    "# nn.Linear modules which are members of the model.\n",
    "criterion = nn.BCELoss(reduction='mean')\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T13:22:13.307900Z",
     "start_time": "2020-05-24T13:22:12.669459Z"
    },
    "code_folding": [],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/1000 | Loss: 0.6444\n",
      "Epoch: 201/1000 | Loss: 0.6441\n",
      "Epoch: 401/1000 | Loss: 0.6437\n",
      "Epoch: 601/1000 | Loss: 0.6431\n",
      "Epoch: 801/1000 | Loss: 0.6424\n",
      "Epoch: 1000/1000 | Loss: 0.6413\n"
     ]
    }
   ],
   "source": [
    "# Training loop\n",
    "for k, epoch in enumerate(range(1000)):\n",
    "    # Forward pass: Compute predicted y by passing x to the model\n",
    "    y_pred = model(x_data)\n",
    "\n",
    "    # Compute and print loss\n",
    "    loss = criterion(y_pred, y_data)\n",
    "    if k % 200 ==0:\n",
    "        print(f'Epoch: {epoch + 1}/1000 | Loss: {loss.item():.4f}')\n",
    "\n",
    "    # Zero gradients, perform a backward pass, and update the weights.\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "print(f'Epoch: {epoch + 1}/1000 | Loss: {loss.item():.4f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![image.png](./images/nn19.png)\n",
    "\n",
    "The images in CIFAR-10 are of size 3x32x32, i.e. 3-channel color images of 32x32 pixels in size. http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T14:21:11.794120Z",
     "start_time": "2020-05-24T14:21:11.787466Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 6, 5) # in_channels = 3, out_channels = 6, kernel_size= 5\n",
    "        self.pool = nn.MaxPool2d(2, 2) # pool of square window of size = 2, stride = 2\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5) # in_channels = 6, out_channels = 16, kernel_size= 5\n",
    "        self.fc1 = nn.Linear(16 * 5 * 5, 120)  # in_features = 16*5*5, out_features = 120\n",
    "        self.fc2 = nn.Linear(120, 84) # in_features = 120, out_features = 84\n",
    "        self.fc3 = nn.Linear(84, 10)  # in_features = 84, out_features = 10\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.pool(F.relu(self.conv1(x)))\n",
    "        x = self.pool(F.relu(self.conv2(x)))\n",
    "        x = x.view(-1, 16 * 5 * 5) # Flatten the data (n, 16, 5, 5)-> (n, 400)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Run in Google Colab\n",
    "\n",
    "![image.png](./images/nn20.png)\n",
    "\n",
    "https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/cifar10_tutorial.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](./images/nn21.png)\n",
    "\n",
    "深度学习 Deep Learning 视频系列 https://space.bilibili.com/88461692/channel/detail?cid=26587"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "\n",
    "![image.png](./images/end.png)\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "170px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}