Artificial Neural Network_downloaded.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Artificial Neural Networks Explained"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Definition:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[**Artificial neural networks**](https://en.wikipedia.org/wiki/Artificial_neural_network) (ANNs) or connectionist systems are computing systems vaguely inspired by the biological neural networks that constitute animal brains. Such systems \"learn\" to perform tasks by considering examples, generally without being programmed with any task-specific rules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Architecture of an Artificial Neural Network:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ANN is a set of connected neurons organized in layers:\n",
    "- **input layer:** brings the initial data into the system for further processing by subsequent layers of artificial neurons.\n",
    "    <br>\n",
    "- **hidden layer:** a layer in between input layers and output layers, where artificial neurons take in a set of weighted inputs and produce an output through an activation function.\n",
    "    <br>\n",
    "- **output layer:** the last layer of neurons that produces given outputs for the program."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Types of ANNs:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Perceptron:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The simplest and oldest model of an ANN, the Perceptron is a linear classifier used for binary predictions. This means that in order for it to work, the data must be linearly separable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://jtsulliv.github.io/images/perceptron/linsep_new.png?raw=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Its Architecture:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](http://ataspinar.com/wp-content/uploads/2016/11/perceptron_schematic_overview.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-layer ANN:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More sophisticated than the perceptron, a Multi-layer ANN (e.g.: Convolutional Neural Network, Recurrent Neural Network etc ...) is capable of solving more complex classification and regression tasks thanks to its hidden layer(s)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Its Architecture:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://www.pyimagesearch.com/wp-content/uploads/2016/08/simple_neural_network_header.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Activation Functions:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Definition:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In artificial neural networks, the [**activation function**](https://en.wikipedia.org/wiki/Activation_function) of a node defines the output of that node given an input or set of inputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sigmoid:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "A [**sigmoid function**](https://en.wikipedia.org/wiki/Sigmoid_function) is a mathematical function having a characteristic \"S\"-shaped curve or sigmoid curve. Often, sigmoid function refers to the special case of the logistic function which generate a set of probability outputs between 0 and 1 when fed with a set of inputs.\n",
    "The sigmoid activation function is widely used in binary classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Equation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Sigmoid(x) = \\frac{1}{1 + e^{(-x)}} = \\frac{e^x}{1 + e^x}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Graphical Representation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](http://ronny.rest/media/blog/2017/2017_08_10_sigmoid/sigmoid_plot.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Programmatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# import our dependencies\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn; seaborn.set()\n",
    "from numpy import exp, array, append"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Sigmoid activation function\n",
    "def Sigmoid(x):\n",
    "    return 1 / (1 + exp(-x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAETCAYAAAA/NdFSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xlc1HX+B/DXHBwDw6UcmggJQl6hoKuSB4mRd2moYIW7\nZdm2drlmar9y1UzpWjfNsmOP2ketB7ltumuHR2qoKCoqIB6ooBYwKiDDwAwz38/vD2QUQVRk7tfz\n8Ujme8yX13wbvu/5fr7f+XxkQggBIiJyOXJbByAiIttgASAiclEsAERELooFgIjIRbEAEBG5KBYA\nIiIXxQJAtywnJwdpaWkYN24cxo4di6eeegonTpwAABw5cgQvvPCCxTNs2bIFixcvbnbZ2LFjkZWV\n1Sa/57XXXkNubi4A4P/+7/+wa9euVm3n7NmzeP755wEApaWlSE1NbZN8APDzzz9j2LBhSE5ORm1t\nbZttF2i71092ThDdAr1eL/r37y9yc3PN87755huRkJAgjEajDZNdNWbMGLFnz5422dawYcPE4cOH\n73g7e/bsEWPGjGmDRE3NnTtXrFy50iLbbqvXT/ZNaesCRI6hpqYGVVVV0Ol05nkPPfQQ1Go1TCYT\nsrOz8cYbb2Djxo24dOkS5s2bh+LiYvj7+yMoKAhRUVF4/vnnce+99+J3v/sdfvrpJ2i1WsyePRvf\nffcdjh8/juDgYKxatQpeXl7Izs7G22+/jZqaGri5ueGll17C0KFDsX79enz//ff4+OOPcfLkSbz6\n6quoqalBREREo2zXysnJwTvvvAODwQCNRoP77rsPS5YsAQBs27YNf/nLXyBJEry8vLBw4UJs2rQJ\nZWVlePnll/H222/j3XffxWOPPYb8/HxotVrMnz8fALBjxw6sWLEC69atw6pVq7B582bo9XrU1NRg\nzpw5SExMxGuvvYbS0lJMmzYNCxcuxLhx43Dw4EHU1dUhPT0du3fvhkKhQExMDObNmwe1Wo3ExERM\nmDABu3fvxq+//opRo0bhlVdeafSaPvvsM2zZsgUeHh6oqqqCl5cXysvLzdlWrFhhnk5LS0OfPn1w\n4MAB/Prrr+jbty/eeustyOXy23r9I0eOxObNm/HBBx/AZDJBrVZj3rx5iImJwYoVK3D+/HloNBqc\nP38e7dq1w7JlyxASEmKJtyO1FVtXIHIcf/vb30RMTIxITEwUL7/8sli3bp3Q6XRCiMafdGfOnCne\nfvttIYQQpaWlYtCgQWL58uVCCCGio6PF559/LoQQ4uOPPxaxsbGipKREmEwmMWHCBPHtt9+KS5cu\nifj4eJGTkyOEEOL48eOif//+ori4WHz99ddi+vTpQgghHn74YbF27VohhBDZ2dninnvuafYMYObM\nmeb5Wq1WDBgwQBw5ckRoNBrRt29fkZ+fL4QQ4vvvvxfTpk0TQjT+BPz444+LTZs2ieLiYjFgwACh\n1+uFEEK8+OKLYu3ateLcuXMiLS1N1NTUCCGE2Lhxoxg7dmyT/XL27FnRp08fIYQQ77//vnjuueeE\nwWAQJpNJzJ07V7z++uvm352eni6EEKKkpETce++9ori4uMnrmjNnjvjss8+EEEIsX75cLFy40Lzs\n2unHH39cvPDCC8JkMomqqioxePBgsXv37tt+/SdPnhT33XefOcuuXbvEoEGDRFVVlVi+fLkYPny4\nqKqqEkII8cwzz4j333+/SWayL7wGQLfsiSeeQGZmJl577TUEBQXh008/xfjx41FVVdVove3btyMl\nJQUAEBwcjJEjRzZaPmLECABAWFgYoqOjERISArlcjtDQUFRWVuLw4cMICwtD7969AQBRUVGIi4vD\n3r17zdsoLy/HsWPHMH78eABA3759ERUV1Wzu9PR0VFVVYdWqVVi4cCFqa2uh0+lw4MABREVFoXv3\n7gCABx98EJ999tkNX3/nzp3RrVs3bN26FZWVldi9ezdGjx6NTp064a233sKGDRvw7rvvYvXq1aiu\nrm5xX+7YsQOpqalwc3ODXC5HWloadu7caV4+fPhwAEBISAjat2+PysrKFrd3M8OGDYNcLodarUZ4\neDgqKytv+/Xv2bMHAwcOROfOnQEA8fHxaNeunflaQf/+/aFWqwEAPXr0uOPMZHksAHRL9u/fj88+\n+wxqtRrDhg3DK6+8gv/+97+Qy+XIzMxstK5SqYS4pospubzx28zNza3Zxw0kSWoyTwgBo9FonpbJ\nZOb51/7e5jz22GPYvn07IiIiMGPGDISEhEAIAYVCYd5Ow7YKCgqa3UaDSZMm4ZtvvsHGjRuRlJQE\nb29v5OXlITU1FVqtFoMGDcJTTz3V4jaae42SJKGurs487eHh0ei1ipt02XX9OtduCwA8PT2brHu7\nr7+5DNf+f2nud5B9YwGgW9KuXTt89NFHyM7ONs/TaDSoqalBdHR0o3UTEhKQkZEBoP6T+ubNmxsd\naG6md+/eOH36NA4fPgwAOHHiBPbt24f+/fub1/H390fPnj2xbt06AEBeXh6OHz/eZFuVlZXIzc3F\nyy+/jAcffBClpaUoLi6GJEno3bs3CgsLzXcybdmyBbNnzwYAKBSKRgWnQVJSEvLy8rB27VpMnjwZ\nALBv3z706tULTzzxBPr3748tW7bAZDKZt3P9wRgAhgwZgtWrV6Ourg6SJOHLL7/EoEGDbnkfXS8g\nIAB5eXkQQkCn0+Hnn3++6XNu9/UPHDgQmZmZOHv2LACYr1E0nKmR4+FFYLolXbp0wcqVK7Fs2TKU\nlJTAw8MDPj4+WLRoESIiIqDRaMzrzps3D6+99hrGjRsHf39/3HXXXY0+Hd5Mu3bt8P777+ONN95A\nbW0tZDIZli5dii5duuDgwYPm9f785z9j3rx5WL16NcLCwhAREdFkW35+fpg+fTomTJgAf39/BAQE\nIC4uDkVFRYiPj8e7776LOXPmmC9qLlu2DADwwAMPYObMmU1uOXV3d8fo0aOxa9cuxMTEAKi//fSH\nH37A6NGj4ebmhvj4eFRWVkKr1SIqKgoKhQITJ040bxsAnn32Wbz11lsYP348jEYjYmJi8Prrr9/y\nPrreQw89hJ07d+LBBx9ESEgIYmNjb/oJPDAw8LZef9euXfGnP/0Jzz33HEwmEzw9PbFq1Sr4+Pi0\nOjfZlkzwPI3a2JdffokePXogNjYWBoMBjz76KJ5//nkkJCTYOhoRXYNnANTmunbtijfeeMPcrj1y\n5Ege/InsEM8AiIhcFC8CExG5KBYAIiIX5TDXADSaqpuvdAMBAV4oL2++mwB7YO/5APvPyHx3hvnu\njD3nCwq68V1aLnEGoFQqbB2hRfaeD7D/jMx3Z5jvzth7vhtxiQJARERNsQAQEbkoFgAiIhfFAkBE\n5KJYAIiIXBQLABGRi7JoATh06BDS0tKazN+6dSuSk5ORkpKCtWvXWjICERHdgMW+CPbpp5/i22+/\nhUqlajS/rq4OS5cuRUZGBlQqFaZMmYLExEQEBgZaKgoRuRhJEqg1mFBrMKLWYILRJMEkCZhMAiZJ\ngvHKY0mqn25YZpSuXe/KMpOAEAJCANKVrtMkAfM8IQRUKndU6/TmdRrmCwDiyj/mxwAAYX4srvxT\nv0bD4ysPBSCXAffHdULkXX5tvp8sVgDCwsKwYsWKJoNZFxYWIiwsDH5+9S+mb9++2LdvH0aNGtXi\n9gICvO7oyxYtfRvOHth7PsD+MzLfnbG3fLUGI0ou6lBysRqHTl+CTm9Ejd6Imlpjo8c1eiN0tXX1\n03ojdLX1B31nEtrRFwN7h7b5di1WAEaMGIFz5841ma/VahsNIOHt7Q2tVnvT7d3J16yDgnzuqCsJ\nS7P3fID9Z2S+O2OrfLpaI8oqdCgrr7nmPx3KKmpQoTXc8nbclXJ4eijh6a5AcIAKKvf6xw3z3BRy\nKBQyKORyKOSy+v+unVZcmSeXN/tYLpfVt5fLZJDL6oe8lDX8BNCunTcqKnSQX5kP4OrjK89p0DA6\nnsz8T8NjWcNko2VymQwBPh6t/v/TUmG3el9AarW60YDZ1dXVHFGIyIlpa+pQcklXf2Avr0FZxdWD\nvbam6XCZMhnQ3tcTPe4OQLC/CkEBKtwV7Is6fR08PRRXD+7uSqg8FPBwV0Aht+39LEFBPtCoHKZr\nNTOrJ46MjERRUREqKirg5eWF7OxsTJs2zdoxiMiCag1G7D+mwe68Ehw9U47rBx1RyGUI9Fch4i5f\n80E+JECF4AAvBPp5QqlofEC39zMoR2W1ArBhwwbodDqkpKRg7ty5mDZtGoQQSE5ORkhIiLViEJGF\nSJJA/plL2JVXggPHNTDUSQCAyLt8ERXqj6AAFYIDVAjxV6Gdryfk17aLkE1YtACEhoaab/McN26c\neX5iYiISExMt+auJyEqKS6uwK7cEWfmlqKyub7cP9ldhYM8QxPfqgJAALxsnpBtxvEYrIrK58io9\n9uSVYHdeCc5p6q/peXsqcX9sJ9zXswMiO/maL3aS/WIBIKJbUqM34sBxDXbllqCgqL5dXyGXIS46\nCPE9OyAmsj3clOxcwJGwABBRi/LPXMLPR35t1K7ftZMf4nt1wG+6BUOtcrNxQmotFgAiapYkBDK2\nFeK7vcUA6tv143t1QHzPEASzXd8psAAQURN1Rgl//W8+9h4tQ8f2XnhiVHe26zshFgAiakRXW4cP\n1h9BQXEFuob64YXkGDbzOCkWACIyu3S5FsvWHcJ5TTX6Rgfh6XE94O7mmAOe082xABARAOBcmRbL\n1h1CeZUeD/QNRerwKH5Zy8mxABARDp/UYOmX+1GjN2HysK4Y0b8z2/tdAAsAkYvbk1+Cv/33KIQA\npj/UAwN7dLB1JLISFgAiFyWEwHd7i7FuWyG8PZX4w4R70T08wNaxyIpYAIhckCQJ/GvLCWzZfw4B\nPh5Y9Mx98FayycfVsAAQuRhDnQmfbsjH/uMadAryxsxJvXF3R192t+yCWACIXIi2pg7Lvz6Mk+cq\n0S3MH889ci+8PHmPv6tiASByERcqavDntYdQckmH/t2DMW1MD3be5uJYAIhcQFFJFf6y7hAqqw0Y\nOSAME++PhJy3ebo8FgAiJ5d76iJWfpMLg8GEKQ9EIalfZ1tHIjvBAkDkxIpKqvB+xmHIZDI8O74X\n+nULtnUksiMsAEROSgiBtdtOwiQJvDjxXvTuGmjrSGRneAWIyEnlnb6Eo0Xl6BXRjgd/ahYLAJET\nkoTAup8KIQMwMSHS1nHITrEAEDmhrLxSnC3TYmDPDggL8bF1HLJTLABETqbOKGH9jlNQKmSYMLSL\nreOQHWMBIHIy2w6cw8XLtUiMC0Wgn8rWcciOsQAQORFdrREbdp2BykOJsffdbes4ZOdYAIicyKas\nIlTXGjF6YBjH8aWbYgEgchLlVXr8uO8sAnw8+G1fuiUsAERO4j8/n4LBKOHhwV04kDvdEhYAIidw\n/kI1dh7+FXcFemPQvRzSkW4NCwCRE1i/vRBCAMkJEVDI+WdNt4bvFCIHd+JcBQ6euICoUD/0YZcP\ndBssVgAkScL8+fORkpKCtLQ0FBUVNVr+7bffYsKECUhOTsZXX31lqRhETk0IgXXbCgEAk+7vChn7\n+KfbYLHeQDdv3gyDwYA1a9YgJycH6enp+Oijj8zL3377bWzcuBFeXl4YM2YMxowZAz8/P0vFIXJK\nB09cwMnzlYiLDkLXUP790O2xWAHYv38/hgwZAgDo06cPcnNzGy2/5557UFVVBaVSCSEEP7kQ3SaT\nJOHr7YWQy2RIToiwdRxyQBYrAFqtFmq12jytUChgNBqhVNb/yqioKCQnJ0OlUiEpKQm+vr4tbi8g\nwAtKZetvbQsKsu8Osew9H2D/GV0t3/d7zuDXizqMGBiOmG53fuePq+2/tmbv+ZpjsQKgVqtRXV1t\nnpYkyXzwLygowE8//YQtW7bAy8sLs2fPxqZNmzBq1Kgbbq+8XNfqLEFBPtBoqlr9fEuz93yA/Wd0\ntXz6OhP+ueko3N3kGNEv9I637Wr7r63Zc76WCpPFLgLHxcVhx44dAICcnBxER0ebl/n4+MDT0xMe\nHh5QKBRo164dLl++bKkoRE7nx31nUak14MHfdIa/2sPWcchBWewMICkpCZmZmUhNTYUQAkuWLMGG\nDRug0+mQkpKClJQUPProo3Bzc0NYWBgmTJhgqShETqVKZ8CmrCKoVW4YNSDc1nHIgVmsAMjlcixa\ntKjRvMjIqyMTTZkyBVOmTLHUrydyWht3FaFGb8KU4RFQeXBYb2o9fhGMyIFoKmqw9cA5BPp54v7Y\nTraOQw6OBYDIgfx75ymYJIFHhkbATck/X7ozfAcROYiikirsyStFWIga/XuE2DoOOQEWACIHkbH9\napcPcn5xktoACwCRA8g7cwl5py+h590B6Nmlna3jkJNgASCyc5IQyLjS4dvE+7vaOA05ExYAIju3\n92gpikqrMLBHCMI7OF53A2S/WACI7JjRJGH99lNQyGWYMJQdvlHbYgEgsmMHjmtwobIW98d2QpC/\nytZxyMmwABDZsaz8UgDgl77IIlgAiOxUdW0dDhdeROdgNToFets6DjkhFgAiO7X/mAYmSWAgv/RF\nFsICQGSnGpp/+ndnASDLYAEgskPlVXoUFJUjKtQP7f08bR2HnBQLAJEd2ne0FAJg8w9ZFAsAkR3a\nk18KhVyGft2CbR2FnBgLAJGdKb2kw5mSKvS4ux18vNxtHYecGAsAkZ1puPjL5h+yNBYAIjsihMCe\n/FK4K+XoExVo6zjk5FgAiOxIcakWJZd06BMVyPF+yeJYAIjsyJ78EgDAADb/kBWwABDZCUkI7D1a\nBi8PJXp1aW/rOOQCWACI7MSJsxUor9KjX7cgDvhOVsF3GZGd2HPl7p8BPTrYOAm5ChYAIjtgNEnI\nLiiDv9od93T2t3UcchEsAER2IPfUJVTXGtG/ewjkcpmt45CLYAEgsgNZRxuaf3j3D1kPCwCRjdUa\njDh4QoOQABXu5qDvZEUsAEQ2lnPiAgx1Egb0CIFMxuYfsh4WACIbu3r3D5t/yLpYAIhsSFtTh7zT\nlxAe4oOO7TnuL1kXCwCRDWUXlMEkCX76J5uwWG9TkiRhwYIFOHbsGNzd3bF48WKEh4eblx8+fBjp\n6ekQQiAoKAjvvPMOPDw8LBWHyC7tyS+FDED/7hz4hazPYmcAmzdvhsFgwJo1azBr1iykp6eblwkh\n8Prrr2Pp0qX417/+hSFDhuD8+fOWikJkly5drsXxsxW4J8wf7Xw57i9Zn8XOAPbv348hQ4YAAPr0\n6YPc3FzzstOnT8Pf3x//+Mc/cOLECSQkJCAiIqLF7QUEeEGpVLQ6T1CQfd9eZ+/5APvP6Gj5dubW\n9/w5vH+4XWS3hwwtYb62Z7ECoNVqoVarzdMKhQJGoxFKpRLl5eU4ePAg5s+fj7CwMPz+979Hr169\nEB8ff8PtlZfrWp0lKMgHGk1Vq59vafaeD7D/jI6Yb8u+YijkMtzTydfm2R1x/9kTe87XUmGyWBOQ\nWq1GdXW1eVqSJCiV9fXG398f4eHhiIyMhJubG4YMGdLoDIHI2f1yoRrFpVrcG9EeapWbreOQi7JY\nAYiLi8OOHTsAADk5OYiOjjYv69y5M6qrq1FUVAQAyM7ORlRUlKWiENmdLN77T3bAYk1ASUlJyMzM\nRGpqKoQQWLJkCTZs2ACdToeUlBS8+eabmDVrFoQQiI2Nxf3332+pKER2RQiBrPxSuLvJ0acrx/0l\n27FYAZDL5Vi0aFGjeZGRkebH8fHxyMjIsNSvJ7JbZ0qqUFZRg4E9QuDh3vobG4juFL8IRmRle/LY\n/EP2gQWAyIokSWDv0VKoVW7o2aWdreOQi2MBILKiguJyVFYb0O+eICgV/PMj2+I7kMiK2PMn2RMW\nACIrqTNK2H9MgwAfD0Rx3F+yAywARFZy5NRF1OiNGNAjBHIO/EJ2gAWAyEoamn8GsvmH7MQtfQ9g\n79692Lp1K86cOQO5XI7w8HAMHz4c/fr1s3Q+Iqegq63DoZMX0LG9FzoHq2/+BCIraPEM4OjRo0hL\nS8OXX36JTp06YdKkSUhJSUFoaCi++OILPPbYY8jLy7NWViKHtSe3BHVGjvtL9qXFM4Bvv/0Wy5cv\nR0BAQJNljz32GC5evIhPPvkEPXv2tFhAImew/eA5ALz7h+xLi2cAc+bMQUBAAHS6pl0xnz9/Hu3b\nt8e8efMsFo7IGVyuNiDnuAZdOvoiJMDL1nGIzG7pIvCECROQk5Njnv7qq6+QkpJisVBEzmRfQRkk\njvtLduiWLgK/+eabmDdvHhITE5Gfnw9PT0+sXbvW0tmInEJWfilkMo77S/bnls4A+vXrh8cffxxf\nffUVTp48iRkzZuCuu+6ydDYih3ehogYnz1cipmsg/NUeto5D1MgtnQE8/vjjUCgU2LBhA86fP49Z\ns2Zh2LBhmDt3rqXzETm0rKP19/4PjQ21cRKipm7pDGDEiBH4/PPPERoaigEDBmD9+vXQ6/WWzkbk\n8LLyS6FUyHBfDM+Yyf60WAC2bdsGAEhLS2s039vbG3/6058AAFu2bLFQNCLHdk6jxTlNNcf9JbvV\nYhPQuXPn8OSTT2LEiBHo168fOnToAKVSifPnzyMrKwv/+9//8MADD1grK5FDaRj3d2DPDjZOQtS8\nFgtAWloaRo8ejS+//BKzZs1CUVER5HI5wsLCMGzYMCxbtgyBgRzTlOh6DeP+ergr0Duyva3jEDXr\npheBDx8+jAkTJuCFF17Ajz/+iIyMDPTo0QN/+MMfoFRabEhhIodW+MtlXKisRXzPDnB347i/ZJ9a\nvAbw17/+FR988AH0ej0KCgowe/ZsPPDAA9DpdHjrrbeslZHI4WTlNTT/8MtfZL9a/Aj/n//8B2vW\nrIFKpcK7776LxMRETJo0CUIIjB492loZiRyKSZKwr6AUPl5u6B7etB8tInvR4hmATCaDSqUCAGRl\nZWHIkCHm+UTUvKNF5bisq8NvugVz3F+yay2eASgUCly+fBk6nQ5Hjx7FoEGDANR3BMf2f6LmNTT/\nsO8fsnctHsWnT5+O8ePHw2g0YuLEiQgODsb//vc/LFu2DDNmzLBWRiKHYagzYf9xDdr7eiKyk5+t\n4xC1qMUCMHLkSMTGxqK8vBzdunUDUP8lsMWLF2PAgAFWCUjkSA4XXkStwYTEuFCO+0t276btOCEh\nIQgJuXoqm5CQYNFARI4si+P+kgPhFSqiNqKrrcOhwovoFOiNUI77Sw6ABYCojew/roHRJPHiLzkM\nFgCiNtLQ/MMCQI6CBYCoDVRq9ThaVI7ITr4I8lfZOg7RLbFYAZAkCfPnz0dKSgrS0tJQVFTU7Hqv\nv/463n33XUvFILKKvQVlEAIY0J2f/slxWKwAbN68GQaDAWvWrMGsWbOQnp7eZJ3Vq1fj+PHjlopA\nZDUN4/7+hgWAHIjFCsD+/fvNXUf06dMHubm5jZYfOHAAhw4dQkpKiqUiEFlFWbkOp365jB53t4Of\nt7ut4xDdMov156DVaqFWX70VTqFQwGg0QqlUoqysDCtXrsQHH3yATZs23dL2AgK8oFS2vlvdoCCf\nVj/XGuw9H2D/GW2Vb2vOLwCApAHhLWbg/rszzNf2LFYA1Go1qqurzdOSJJn7D/ruu+9QXl6O6dOn\nQ6PRoLa2FhEREXjkkUduuL3ycl2rswQF+UCjqWr18y3N3vMB9p/RVvmEENiyrxhKhRxRHW+cgfvv\nzjBf67VUmCxWAOLi4rBt2zaMHj0aOTk5iI6ONi+bOnUqpk6dCgBYv349Tp061eLBn8henS3T4teL\nOvS9JwgqD3aQSI7FYu/YpKQkZGZmIjU1FUIILFmyBBs2bIBOp2O7PzkNdv1AjsxiBUAul2PRokWN\n5kVGRjZZj5/8yVFJQiDraClUHgrEcNxfckD8IhhRK508V4lLl/XoGx0Mtzu4QYHIVlgAiFqJXT+Q\no2MBIGoFo0nCvoIy+Hq7o1u4v63jELUKCwBRK+SfuQRtTR36dwuGQs4/I3JMfOcStYK5+acnm3/I\ncbEAEN0mfZ0JB45fQJC/JyI6+to6DlGrsQAQ3aZDJy9AX2fCgB4hkHHcX3JgLABEt2lPXsPdPx1s\nnITozrAAEN0GbU0djpy6iM7BanQK9LZ1HKI7wgJAdBsOHNfAJAne+09OgQWA6DbsySsBAPTvHmzj\nJER3jgWA6BaVV+lxrLgCUaF+CPTjuL/k+FgAiG7R3qOlEGDPn+Q8WACIblFWfinkMhn6dmPzDzkH\nFgCiW1BySYczJVXo2aUdfL047i85BxYAolvAgV/IGbEAEN2EEAJ78kvhrpSjT1SgreMQtRkWAKKb\nKC7VovSSDr27BnLcX3IqLABEN7Env/7efzb/kLNhASBqgSQJZOWXwstDiV4RHPeXnAsLAFELck9f\nQoXWgH7dguCm5J8LORe+o4luQBIC63cUAgAS40JtnIao7bEAEN3A3vxSFJdqMbBnCMJCfGwdh6jN\nsQAQNaPOKGH9jlNQKmR4ZEiEreMQWQQLAFEzfjp4HhcqazEsNhSB/uz4jZwTCwDRdXS1RmzYdQYq\nDwXG3hdu6zhEFsMCQHSd7/YWQVtTh1EDwuHDfn/IibEAEF2jvEqPH/aehZ/aHUm/6WzrOEQWxQJA\ndI1vM0/DYJQwfnAXeLgpbB2HyKJYAIiu+PViNXYe+hUd23thcExHW8chsjgWAKIrvt5+CpIQmJgQ\nCYWcfxrk/PguJwJw8lwlDhzXoGsnP3b5TC7DYn3bSpKEBQsW4NixY3B3d8fixYsRHn71lrqNGzfi\n888/h0KhQHR0NBYsWAA5P3WRDQghsO6nkwCAScMiIZPJbJyIyDosdsTdvHkzDAYD1qxZg1mzZiE9\nPd28rLa2Fn/5y1/wxRdfYPXq1dBqtdi2bZulohC1KOfkBZw4V4nYqEBEhfrbOg6R1VjsDGD//v0Y\nMmQIAKBPnz7Izc01L3N3d8fq1auhUtV/w9JoNMLDw6PF7QUEeEGpbP1dGUFB9t2Xi73nA+w/Y2vy\nmUwSvvl5H+Qy4OkJMRZ9jc64/6yJ+dqexQqAVquFWq02TysUChiNRiiVSsjlcgQG1rez/vOf/4RO\np8OgQYMEV4+PAAAPpElEQVRa3F55ua7VWYKCfKDRVLX6+ZZm7/kA+8/Y2nw7Dv2Cs6VVGNq7Izzl\nsNhrdNb9Zy3M13otFSaLFQC1Wo3q6mrztCRJUCqVjabfeecdnD59GitWrGC7K1mdvs6Eb3aegrtS\njocHs8M3cj0WuwYQFxeHHTt2AABycnIQHR3daPn8+fOh1+vx4YcfmpuCiKxpc/ZZVGgNSPpNZwT4\ntNwESeSMLHYGkJSUhMzMTKSmpkIIgSVLlmDDhg3Q6XTo1asXMjIy0K9fP/z2t78FAEydOhVJSUmW\nikPUiLamDv/bUwxvTyVGDWCHb+SaLFYA5HI5Fi1a1GheZGSk+XFBQYGlfjXRTW3cdQY1eiNSE7vC\ny9NifwZEdo033pPLuVBRg60HzqG9ryeGcahHcmEsAORy/r3zNIwmgUeGRnCgd3JpfPeTSykurcKe\nvBKEBasxoGeIreMQ2RQLALmUjO2FEAAmDouEnLcek4tjASCXcfTMJeSeuoTu4QHoeXc7W8chsjkW\nAHIJkhBY+1MhAHb4RtSABYBcQnZBGYpKqtC/ezDu7uBr6zhEdoEFgJye0SRh/fZTUMhleCQh8uZP\nIHIRLADk9Lbn/IKyihrcH9sJwf7sdoSoAQsAOTVdbR2+zTwNT3cFxg2629ZxiOwKCwA5rfIqPdK/\nPIgqXR1GDQiDr5e7rSMR2RV2gkJO6fyFaixbm4NLl/W4P7YTRsezwzei67EAkNM5VlyOFV8fgU5v\nRHJCBEYPDOdtn0TNYAEgp7KvoAyfbsiDEMBTY7vjvl4dbR2JyG6xAJDT+GHfWazZcgIe7grMmHAv\nenbht32JWsICQA5PEgJ//TYX32wvhJ/aHTMn9UZYiOMN0E1kbSwA5NDqjBL++t987D1aho7tvTBz\ncm8E+vFef6JbwQJADqu6tg4ffH0Ex85WoGdEezwzrgfUKjdbxyJyGCwA5JAuXa7FsrWHcP5CNfrd\nE4R5TwxAZYXO1rGIHAoLADmcs2VaLFubgwqtAQ/0C0Xq8Ci4uylsHYvI4bAAkEM5euYSPvj3EdTo\nTUhJ7IoR/cNsHYnIYbEAkMPYk1eCv/73KGQy4PcP90T/7hzSkehOsACQ3RNCYFNWMTJ+KoTKQ4nn\nH7kX3cIDbB2LyOGxAJBdqzUYkfFTIbYeOI8AHw/8cXJvdApS2zoWkVNgASC7I0kC+WcuYVdeCQ4c\n18BQJyE0yBszJ/dBgI+HreMROQ0WALILQgicLdNiV24JsvJLUVltAAAE+6sQ36sDkvp1hpcn365E\nbYl/UWRT5VV67Mkrwa68EpzXVAMAvD2VuD+2E+7r1QGRd/myJ08iC2EBIKur0Rtx4LgGu3JLUFBU\nDgFAIZchLjoI8T07ICayPdyUHKuIyNJYAMgqTJKE/DPl2J1bggMn6tv1AaBrJz/E9+qA33QLZjcO\nRFbGAkBtTgiBy9UGlJbXQFNRg6LSKuw7WtakXT++ZwiCA7xsnJbIdbEAUKtIQqD8sh5l5TqUVtRA\nU16DsvIa80FfX2dqtL63pxLDYjshnu36RHbDYgVAkiQsWLAAx44dg7u7OxYvXozw8Kvjsm7duhUr\nV66EUqlEcnIyJk+ebKkodBtMkoRagwk1eiNqDSbUGkzQ1dah5pgGhWfL6w/0FfUHeaNJNHm+h5sC\nwQGqq//5qxAS4IWuoX5QKtiuT2RPLFYANm/eDIPBgDVr1iAnJwfp6en46KOPAAB1dXVYunQpMjIy\noFKpMGXKFCQmJiIwMNBScaxCiPoDorjyj8CVaXF1uSTqf4qGnwA8qg24rDNcnXfdTwmAySTBJAmY\nTKL+pyTd4HHTaaNJunIwNzY5uNeaHxtRYzChzijd9HV6eyrROViN4AAvBPtfc7AP8IKvlxs/3RM5\nCIsVgP3792PIkCEAgD59+iA3N9e8rLCwEGFhYfDz8wMA9O3bF/v27cOoUaPaPEdBUTleXL4TesPV\nJonrP7eKJh9kxQ2Xi2sO7PUHecfm6a6Ap7sCXp5uaO/nCU935ZV5Snh6KKByV0LloUBE5wB4KmQI\n8lfxYi2Rk7BYAdBqtVCrr35lX6FQwGg0QqlUQqvVwsfn6pB93t7e0Gq1LW4vIMALSuXtd/lbbRQI\n6+ALw3Vt0td/SJWhyYwmk9d+sm14eP28hu1cu32ZrH49uUzWzOOmy2QyNJlWKuRQyGX1PxVyKBUy\nKOT1P+Xm+Vd+XpnfsJ5SIa8/kHsqofJQwuvKT093JeRy5/m0HhRk38NAMt+dYb62Z7ECoFarUV1d\nbZ6WJAlKpbLZZdXV1Y0KQnPKy1s32Ie3Uob0GYOh0VS16vnWEBTkY9V8ksGIaoMR1Tdf1czaGW8X\n890Z5rsz9pyvpcJksatycXFx2LFjBwAgJycH0dHR5mWRkZEoKipCRUUFDAYDsrOzERsba6koRETU\nDIudASQlJSEzMxOpqakQQmDJkiXYsGEDdDodUlJSMHfuXEybNg1CCCQnJyMkhH27ExFZk8UKgFwu\nx6JFixrNi4yMND9OTExEYmKipX49ERHdBG/MJiJyUSwAREQuigWAiMhFsQAQEbkoFgAiIhclE6Jp\nRwhEROT8eAZAROSiWACIiFwUCwARkYtiASAiclEsAERELooFgIjIRbEAEBG5KIv1BmorP/74I777\n7ju89957AOrHInjzzTehUCgwePBgPPfcc43Wr62txezZs3Hx4kV4e3vjrbfeQrt27Sya8ZNPPsHO\nnTsBAJcvX8aFCxeQmZnZaJ3FixfjwIED8Pb2BgB8+OGHNx00p60IITB06FDcfffdAOqH9Jw1a1aj\nddauXYvVq1dDqVTi2WefxbBhw6ySDQCqqqowe/ZsaLVa1NXVYe7cuU3Gk7DF/pMkCQsWLMCxY8fg\n7u6OxYsXIzw83Lx869atWLlyJZRKJZKTkzF58mSL5rleXV0dXn31VZw/fx4GgwHPPvsshg8fbl7+\nj3/8A+vWrTO//xcuXIiIiAirZpwwYYJ5JMHQ0FAsXbrUvMzW+2/9+vX497//DQDQ6/U4evQoMjMz\n4evrC8A+9t9tE07kjTfeECNGjBAvvfSSed5DDz0kioqKhCRJ4qmnnhJ5eXmNnvO3v/1NLF++XAgh\nxMaNG8Ubb7xh1czTp08XO3fubDI/NTVVXLx40apZGpw5c0Y888wzN1xeVlYmxo4dK/R6vbh8+bL5\nsbW8//774u9//7sQQojCwkIxfvz4JuvYYv99//33Ys6cOUIIIQ4ePCh+//vfm5cZDAbxwAMPiIqK\nCqHX68UjjzwiNBqNVfNlZGSIxYsXCyGEKC8vFwkJCY2Wz5o1Sxw5csSqma5VW1srHn744WaX2cP+\nu9aCBQvE6tWrG82z9f5rDadqAoqLi8OCBQvM01qtFgaDAWFhYZDJZBg8eDB27drV6DnXDl4/dOhQ\n7N6922p5f/jhB/j6+mLw4MGN5kuShKKiIsyfPx+pqanIyMiwWiYAyMvLQ2lpKdLS0vD000/j1KlT\njZYfPnwYsbGxcHd3h4+PD8LCwlBQUGC1fL/73e+QmpoKADCZTPDw8Gi03Fb779r3Up8+fZCbm2te\nVlhYiLCwMPj5+cHd3R19+/bFvn37rJKrwciRI/Hiiy8CqD/LUygaj7Gdl5eHTz75BFOmTMHHH39s\n1WwAUFBQgJqaGjz55JOYOnUqcnJyzMvsYf81OHLkCE6ePImUlJRG8229/1rDIZuA1q1bh88//7zR\nvCVLlmD06NHIysoyz7t+YHpvb2+cPXu20fOuHaDe29sbVVVtO67njbLGxMTg448/xp///Ocmz9Hp\ndHj88cfxxBNPwGQyYerUqejVqxe6devWptlulG/+/PmYPn06Ro0ahezsbMyePRtff/21efm1+wyo\n329arbbNs90oX8P+02g0mD17Nl599dVGy625/651/ftNoVDAaDRCqVRadZ/dSENzmFarxQsvvICX\nXnqp0fIxY8bg0UcfhVqtxnPPPYdt27ZZtWnP09MT06ZNw6RJk3DmzBk8/fTT+O677+xm/zX4+OOP\nMWPGjCbzbb3/WsMhC8CkSZMwadKkm67X3ODzDe11za3T3HJLZT158iR8fX0btRE3UKlUmDp1KlQq\nFQBg4MCBKCgosMgBrLl8NTU15k+H/fr1Q1lZGYQQkMlkAJrfr5ZqX7/R/jt27Bj++Mc/4pVXXkH/\n/v0bLbPm/rvW9ftFkiQolcpml1lyn7Xk119/xYwZM/Doo49i3Lhx5vlCCPz2t781Z0pISEB+fr5V\nD2BdunRBeHg4ZDIZunTpAn9/f2g0GnTs2NFu9t/ly5dx+vRpDBw4sNF8e9h/reFUTUDXU6vVcHNz\nQ3FxMYQQ+Pnnn9GvX79G68TFxWH79u0AgB07dqBv375WybZr1y4MHTq02WVnzpzBlClTYDKZUFdX\nhwMHDqBnz55WyQUAH3zwgflTd0FBATp27Gg++ANATEwM9u/fD71ej6qqKhQWFiI6Otpq+U6ePIkX\nX3wR7733HhISEpost9X+i4uLw44dOwDU33xw7T6JjIxEUVERKioqYDAYkJ2d3eTCtaVduHABTz75\nJGbPno2JEyc2WqbVajF27FhUV1dDCIGsrCz06tXLqvkyMjKQnp4OACgtLYVWq0VQUBAA+9h/ALBv\n3z7Ex8c3mW8P+681HPIM4HYsXLgQL7/8MkwmEwYPHozevXsDAJ588kmsWrUKU6ZMwZw5czBlyhS4\nubmZ7x6ytNOnT2PQoEGN5v39739HWFgYhg8fjocffhiTJ0+Gm5sbHn74YURFRVklFwBMnz4ds2fP\nxvbt26FQKMx3YlybLy0tDY8++iiEEJg5c2aTdnhLeu+992AwGPDmm28CqC/0H330kc33X1JSEjIz\nM5GamgohBJYsWYINGzZAp9MhJSUFc+fOxbRp0yCEQHJyMkJCQiye6VqrVq3C5cuX8eGHH+LDDz8E\nUH+GVVNTg5SUFMycORNTp06Fu7s74uPjmy2uljRx4kTMmzcPU6ZMgUwmw5IlS7Bp0ya72X9A/d9t\naGioefra/7+23n+twe6giYhclFM3ARER0Y2xABARuSgWACIiF8UCQETkolgAiIhcFAsAEZGLYgEg\nInJRLABErfTFF1/gsccegxAC2dnZePDBB23WPw1Ra/CLYEStJITA1KlTMXLkSPzzn//Em2++abWu\nRIjaAgsA0R04e/Ysxo0bZ+5ShMiRsAmI6A788ssvUKvVyM/PBz9LkaNhASBqperqarz++uv48MMP\noVKp8NVXX9k6EtFtYQEgaqV33nkHCQkJiImJMReC6wccIrJnvAZAROSieAZAROSiWACIiFwUCwAR\nkYtiASAiclEsAERELooFgIjIRbEAEBG5qP8HQPelmGQ4hzAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xba30e80>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Graphical representation of our sigmoid activation function\n",
    "for i in array([range(-10, 10)]):\n",
    "    x = append(array([]), i)\n",
    "    plt.plot(x, Sigmoid(x))\n",
    "    plt.title(\"Sigmoid activation function\")\n",
    "    plt.xlabel(\"x\")\n",
    "    plt.ylabel(\"S(x)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tan-h:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "An alternative to the logistic sigmoid is the [**hyperbolic tangent**](https://theclevermachine.wordpress.com/tag/tanh-function/), or tan-h function. Like the logistic sigmoid, the tan-h function is also sigmoidal (\"S\"-shaped), but instead outputs values that range [-1, 1]. Thus strongly negative inputs to the tan-h will map to negative outputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Equation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "$$tanh(x) = \\frac{2}{1 + e^{-2x}} - 1 = 2 \\times Sigmoid(2x) - 1$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Graphical Representation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://i.stack.imgur.com/Mg9s9.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Programmatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Tan-h activation function\n",
    "def Tanh(x):\n",
    "    return 2 / (1 + exp(-2*x)) - 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAETCAYAAADH1SqlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XtcVOW+BvBnrjDOjAqGiLlBtNiZVgjm0aPiTrO0G7UR\nYSypvFUnytJDZsfjtiK8lHV2FzR3n8x0l6a221I7K7cGajfF2CqKbeXiBQRElJnhMszMe/5ARkcu\no8CsmYHn+6mPzHrXmnlmOc6Ptd71rlcmhBAgIiJqhdzTAYiIyPuxWBARkUssFkRE5BKLBRERucRi\nQURELrFYEBGRSywWJKnU1FTExsYiNjYWQ4YMwd133+14XFtb2yGvYTAYsH379g55ristWLAAeXl5\njp9/+umnNj3PiRMnMGfOHABASUkJpk6d2mEZMzMz8Yc//AHx8fGwWCwd9rxAx71/8j1KTwegrmXh\nwoWOn8eNG4c33ngDt9xyiwcTXZs9e/YgKSkJALBkyZI2P8+pU6dQWFgIAAgJCcEnn3zSEfEAAF99\n9RWmTp2K2bNnd9hzNuqo90++h8WCvMpnn32GTZs2ob6+HhcuXMCTTz6JhIQEbNq0Cd9//z3sdjtO\nnjwJtVqN5cuX44Ybbmj2eb799lusWrUKFRUVGD16NF5++WXI5c4H0tnZ2XjzzTdRV1eH8vJyxMTE\n4NVXXwUA/POf/8Sf//xnCCGg1WrxyiuvYOvWrTh37hyee+45vPHGG0hLS8OMGTOwf/9+1NfX43/+\n538AADt37sT777+PDRs24L333sPOnTtRV1eHmpoaLFiwAGPGjMGf/vQnlJaWYtasWVi4cCHi4uKw\nb98+WCwWLFmyBL/88gvkcjkiIyPx4osvQqvVIiYmBlOmTMEPP/yAkpIS3HfffZg3b57Te1q9ejW+\n//57/Pjjj6iqqoJCoUB1dbUj21tvveV4bDAYcPvttyM7OxslJSUYPnw4lixZAplMdk3v/84778Q3\n33yDlStXwmazQa/XY8GCBbjlllvw1ltvoaysDKWlpTh9+jSCgoLw1ltvISgoqKM/OuRugshD7rjj\nDnHgwAHHY6PRKBISEkRlZaUQQoi9e/eKYcOGCSGE+Oyzz8Ttt98uzpw5I4QQYtGiRWLBggXNPm9i\nYqJITk4WNptNmM1mMXLkSLF///4m6z377LNi7969jte+/fbbxZEjR8SZM2dEdHS0OHLkiBBCiH/8\n4x/iiSeeEEIIMWbMGHH48GHH63z33XciPz9fjBw5UlgsFiGEEMnJyWLLli3ixIkT4tFHHxW1tbVC\nCCG++OILERsbK4QQYs+ePeKBBx4QQghRWFgooqOjhRBCvPnmm2LOnDmivr5eWK1W8cILL4iXX37Z\n8dqvv/66EEKI4uJiMXjwYFFcXNzkfc2bN0+sWbPG8XypqamOtssfJyYmirlz5wqbzSaqqqrEqFGj\nxN69e6/5/f/2229i1KhR4uTJk0IIIXbt2iVGjRolTCaTePPNN8WECROE0WgUQggxc+ZM8e677zb7\n90bejUcW5DV0Oh3S09Oxc+dOFBYW4siRI6iurna033LLLQgODgYADB48GJmZmS0+1z333AO5XI5u\n3bohNDQUFRUVTdZ5/fXXkZmZiZUrVyI/Px91dXUwm83Iz8/HoEGDcNNNNwEAJk2ahEmTJrX4WuHh\n4Rg4cCAyMzMRFRWFvXv3Yvny5dBoNEhLS8PWrVtRVFSEX3/91en9NCcrKwvz58+HUtnwT/Phhx/G\n3LlzHe3jx48H0HDqKiAgABcuXEBISEirz9maO+64A3K5HHq9Hv369cP58+dRVlZ2Te//xx9/xKhR\no9CvXz8AwOjRo9GjRw8cOXIEADBixAjodDoAwM0334wLFy60OS95Dju4yWucPn0aDz30EM6cOYNh\nw4Zhzpw5EJfduszPz8/xs0wmgxACJSUljg7y2NhYR1FQqVRO615JCIHExETs2rULN9xwA5555hkE\nBQVBCOH4om5kt9sdnbotmTx5Mr744gtkZGTg7rvvhkajwcGDB2EwGGA2mzF69GjMnDnT6f00x263\nN8lptVodj/39/Zvsg9ZcuU59fb1T+5XPB+Ca3/+VmRuXNb7W5X9vAFxmJu/EYkFe4+DBgwgKCsKT\nTz6JMWPGYOfOnc1+EV0uJCQEf//73x3/9+rV66peq7KyEnl5eUhJScGECRNw+vRpnDp1Cna7HZGR\nkfjtt99w/PhxAA39HwsWLADQ8EV65RcuAEycOBE5OTnYvHkzpkyZAgD45ZdfcNttt+Gxxx7D7bff\nju3bt8NmswEAFAqFUxFoNHr0aGzYsAFWqxU2mw1//etf8Z//+Z9X9Z6aExgYiNzcXAghYDab8cMP\nP7jc5lrf/8iRI5GVlYVTp04BAHbv3o2zZ8/61IUL5BpPQ5HXiImJweeff46JEydCo9HgtttuQ48e\nPXDixIkOf63AwEDMmDEDsbGx6NmzJ3r16oWhQ4eiqKgIw4cPx/Lly5GSkuLosH3jjTcANJwGmjNn\nDtLS0pyez8/PDxMnTkR2djYGDx4MALj//vuxfft23HvvvVAqlRg5ciQqKytRXV2NiIgI2Gw2JCQk\nYPny5Y7nSU5OxtKlSxEbGwur1YrIyEjHF3VbxMbGYvfu3bjrrrvQp08fDB061OU2vXv3vqb3//vf\n/x4LFy7E008/DZvNBo1Gg1WrVjlOPVHnIBM8JiQiIhd4GoqIiFxisSAiIpdYLIiIyCUWCyIicqnT\nXg1VXm5s87YBAd1QWdn64ClPYr72Yb72Yb728fZ8QUH6ZpfzyKIZSqXC0xFaxXztw3ztw3zt4+35\nWsJiQURELrFYEBGRSywWRETkEosFERG5xGJBREQusVgQEZFLHikW//rXvzBt2rQmy3fs2IG4uDgk\nJCTgs88+A9BwX/xFixYhISEB06ZNQ1FRkdRxiYi6PMkH5f3lL3/B1q1bodFonJbX19djyZIl2Lx5\nMzQaDQwGA8aNG4f9+/fDYrFg48aNyMnJwdKlS7Fy5UqpYxPRVairt+FcVS2sNgGrzd7wv9WO+ssf\n2+yw2gTqrc6PrTY76q122OwCEICAgAAu/nyRaFgmLjaIxraLKwgI+PmpUFfbdM4Rx1O4eA/uvg+3\nv78StbVN5zLpKD10asSNHQCFvGOPBSQvFqGhoXjnnXfwwgsvOC0/fvw4QkND0aNHDwBAdHQ09u7d\ni5ycHIwZMwZAw6Qshw4duqrXCQjo1q7BLy2NYvQWzNc+zNc+QUF6GKstyD91AcdPX0D+6QvILz6P\n02Um2DnpgUeplHJMnTQIAXp/1ytfA8mLxd133+2YUetyJpMJev2lfyBarRYmkwkmk8lpEpXGGcau\nnPrxSu0ZTh8UpG/X7ULcjfnah/mujRAClcY6nCg14USpESWVNTh2shIVVXVO6/mpFRh4fQ/0CewG\ntVIBpVIGpUIOlUIOhUIGlUIOpVIOpUIOpeJS25XLFHIZZDIZZAAgw8U/Lz5u+PHin03XAYBegVpU\nnjO3/qaazrR7RbOLFdohsJcW5ypc5GsHjZ8S1tp6lLdydNWaln5R8Zp7Q+l0OpjNl3ag2WyGXq9v\nstxut7ssFETUNkIInDlX7SgMJ0qNKCo1wVTj/MXTXavGkAGBCAvW43e9dQgL1iMoQAN5M/OdSy2o\nlxYKF9PxelJQQDfIrDZPx7hmXvOtO3DgQBQVFeH8+fPo1q0b9u3bhxkzZkAmk2Hnzp245557kJOT\ng4iICE9HJeqU7EJg9dZc/HKkzGn5dT388fvf9URoHz3CgnUYenMIbHVt+62VfJfHi0VGRgaqq6uR\nkJCAF198ETNmzIAQAnFxcQgODsaECROwZ88eJCYmQgjRZO5jIuoYX/5QiF+OlCGsjx4jbw5GaLAe\nocE6dPNXOa0X2N0f5eUsFl1Np52Duz3nfL3tnPGVmK99mK+pnGNn8c7mAwjs7o9Fjw2Dvpu6xXW5\n/9rHF/I1h4PyiLq4kgoz/pKRC5VSjuQ/3tJqoaCui8WCqAurqbPi3c8PoqbOhscm3YSwPt59yS55\nDosFURdlFwIffHkYJRXVuOv232HE4D6ejkRejMWCqIv6ck8hfv33WQwKC0D8HQM9HYe8HIsFUReU\n8++z+GJ3AXp198eTsYM7/NYQ1PnwE0LUxZRUmPGXL3OhVsrxTBw7tOnqsFgQdSFOHdr33ITQYHZo\n09VhsSDqIuxC4C8ZDR3adw//HUbczA5tunosFkRdRMaeQuQca+jQnvwHdmjTtWGxIOoCfv13Of6+\nuwDX9fDHUw8OYYc2XTN+Yog6uYYR2oehvjhCW6dRud6I6AosFkSdWHWtFe9sOYhaCzu0qX1YLIg6\nqcYR2mfOsUOb2o/FgqiT2rq7gB3a1GFYLIg6oV9/K8fWPYXs0KYOw08QUSfTMEKbHdrUsSSdKc9u\nt2Px4sU4evQo1Go1UlNTERYWBgAoLy/H3LlzHeseOXIE8+bNg8FgwEMPPQSdTgcA6NevH5YsWSJl\nbCKfUV1rxdsXO7SfeGAwO7Spw0haLLZv3w6LxYKNGzciJycHS5cuxcqVKwEAQUFBWLduHQDg119/\nxVtvvYUpU6agrq4OQghHGxG1bPP3x1B6rhoTh4fiP24O9nQc6kQkPQ2VnZ2NMWPGAAAiIyNx6NCh\nJusIIfDqq69i8eLFUCgUyMvLQ01NDaZPn46kpCTk5ORIGZnIpxwqOAedRoW4PwzwdBTqZCQ9sjCZ\nTI7TSQCgUChgtVqhVF6KsWPHDtx4440YMKDhw+7v748ZM2YgPj4ehYWFmDVrFrZt2+a0TXMCArpB\nqVS0OWtL89B6C+Zrn86Y74KpDmcv1GLYoGD0Ce7hhlSXdMb9JyVvz9ccSYuFTqeD2Wx2PLbb7U2+\n9Ldu3YqkpCTH4/DwcISFhUEmkyE8PBw9e/ZEeXk5QkJCWn2tysrqNuf0hQnVma/tOmu+fx07CwC4\nvlc3t76/zrr/pOIL+Zoj6WmoqKgoZGVlAQBycnIQERHRZJ1Dhw4hKirK8Xjz5s1YunQpAKC0tBQm\nkwlBQUHSBCbyIfnFVQCAAX27ezgJdUaSHllMmDABe/bsQWJiIoQQSEtLQ0ZGBqqrq5GQkIBz585B\np9NBJpM5tpk8eTIWLFgAg8EAmUyGtLQ0l6egiLqigpKGYhEewmJBHU/Sb125XI5XXnnFadnAgZdG\nlgYGBuLvf/+7U7tarcaKFSskyUfkq4QQKCipQu+eGo6rILfgoDyiTqCssgbmWitPQZHbsFgQdQKN\n/RU8BUXuwmJB1Ankl7Bzm9yLxYKoE8gvroJCLkNosM71ykRtwGJB5OPqrXacLDPid711ULVjICpR\na1gsiHzcyTITrDaBcJ6CIjdisSDycY3jKwawc5vciMWCyMflF18AwM5tci8WCyIfl19ihMZPieDA\nbp6OQp0YiwWRDzPX1qP0XDXCQ/SQX3abHKKOxmJB5MMKOL6CJMJiQeTDOHKbpMJiQeTDCop5JRRJ\ng8WCyEcJIZBfUoVe3f3QQ+fn6TjUybFYEPmoigu1MFbXI7yve6dQJQJYLIh8Vj4H45GEWCyIfBSn\nUSUpSTpTnt1ux+LFi3H06FGo1WqkpqYiLCzM0f7RRx9h06ZNCAwMBAC8/PLL6N+/f6vbEHVVBSVV\nkMtkCAvWezoKdQGSFovt27fDYrFg48aNyMnJwdKlS7Fy5UpH+6FDh7Bs2TIMGTLEsezbb79tdRui\nrshqs6PojBHXB2nhp+adZsn9JC0W2dnZGDNmDAAgMjIShw4dcmrPzc3F6tWrUV5ejj/84Q944okn\nXG5D1BUVnzXDYrVzfAVJRtJiYTKZoNNdmpxFoVDAarVCqWyIce+992Lq1KnQ6XRITk7Gzp07XW7T\nkoCAblC2497+QUHefWjPfO3j6/n2HasAANz2+94eeS++vv88zdvzNUfSYqHT6WA2mx2P7Xa740tf\nCIFHH30Uen3DThw7diwOHz7c6jatqaysbnPOoCA9ysuNbd7e3ZivfTpDvgO/lTWsq1NL/l46w/7z\nJF/I1xxJr4aKiopCVlYWACAnJwcRERGONpPJhPvuuw9msxlCCPz8888YMmRIq9sQdVUFxVXwUynQ\n9zqtp6NQFyHpkcWECROwZ88eJCYmQgiBtLQ0ZGRkoLq6GgkJCXj++eeRlJQEtVqNkSNHYuzYsbDb\n7U22IerKauqsKD5rRsTvekIu551mSRqSFgu5XI5XXnnFadnAgQMdPz/44IN48MEHXW5D1JUVnjFC\ngOMrSFoclEfkYxpvS84roUhKLBZEPoYjt8kTWCyIfExBSRV66NQI0PNOsyQdFgsiH1JprEOlsQ4D\nQrpDxmlUSUIsFkQ+hKegyFNYLIh8SH7JBQDs3CbpsVgQ+ZCC4irIAPTvw2JB0mKxIPIRdrtAwRkj\n+vTqhm7+kg6RImKxIPIVJRVm1Fls7K8gj2CxIPIRjs5t9leQB7BYEPmIxpHbA/r28HAS6opYLIh8\nRH5xFVRKOa4P4p1mSXosFkQ+oK7ehlPlZoQF66FU8J8tSY+fOiIfUHTGCLsQHF9BHsNiQeQDLvVX\nsFiQZ7BYEPmAxiuhwlksyENYLIh8QEFJFXQaFYJ6+Hs6CnVRkg4DtdvtWLx4MY4ePQq1Wo3U1FSE\nhYU52r/88kusXbsWCoUCERERWLx4MeRyOR566CHodDoAQL9+/bBkyRIpYxN5VJXZgrMXanHrwF68\n0yx5jKTFYvv27bBYLNi4cSNycnKwdOlSrFy5EgBQW1uL//u//0NGRgY0Gg3mzp2LnTt3YvTo0RBC\nYN26dVJGJfIa+SUcjEeeJ2mxyM7OxpgxYwAAkZGROHTokKNNrVZjw4YN0Gg0AACr1Qo/Pz/k5eWh\npqYG06dPh9Vqxdy5cxEZGenytQICukGpVLQ5a1CQvs3bSoH52seX8pVmnwIARA4K9prc3pKjJczX\n8SQtFiaTyXE6CQAUCgWsViuUSiXkcjmuu+46AMC6detQXV2NUaNG4bfffsOMGTMQHx+PwsJCzJo1\nC9u2bYNS2Xr0ysrqNucMCtKjvNzY5u3djfnax9fyHTp2FgAQ2E3lFbl9bf95G1/I1xxJi4VOp4PZ\nbHY8ttvtTl/6drsdr7/+OgoKCvDOO+9AJpMhPDwcYWFhjp979uyJ8vJyhISESBmdyCOEECgorkLv\nAA10GpWn41AXJunVUFFRUcjKygIA5OTkICIiwql90aJFqKurQ3p6uuN01ObNm7F06VIAQGlpKUwm\nE4KCgqSMTeQxpZU1qK6zcnwFeZykRxYTJkzAnj17kJiYCCEE0tLSkJGRgerqagwZMgSbN2/GsGHD\n8OijjwIAkpKSMHnyZCxYsAAGgwEymQxpaWkuT0ERdRYFjeMr2LlNHibpt65cLscrr7zitGzgwIGO\nn/Py8prdbsWKFW7NReStOOc2eQsOyiPyYvklVVDIZQjtrXO9MpEbsVgQeal6qx0ny4wIDdZB1Y7L\nwIk6AosFkZc6WWaC1cY7zZJ3YLEg8lL5xRcAsL+CvAOLBZGXarwtOY8syBu0ejVUfX09vvzyS+zY\nsQOFhYWQy+UICwvDuHHjcO+990Kl4iAhInfJL66Cxk+J4MBuno5C1HKx+P7777Fy5UpER0fjoYce\nQt++faFSqXDq1Cn89NNPWL9+PZ566imMHz9eyrxEXYKpph6llTUY3D8Act5plrxAi8WisLAQ69ev\nb3L0MHDgQIwdOxYWiwV//etf3R6QqCsqLOFkR+RdWuyzeOyxx6BSqVBaWtqk7cCBA1Cr1Xj88cfd\nGo6oq7p0W/IeHk5C1MBlB/eUKVPw9ddfA2jow3j99dfx3HPPuT0YUVfGaVTJ27i83cfHH3+Ml156\nCd988w3y8/MxfPhwbN26VYpsRF2SEAIFJVXo1d0fPbRqT8chAnAVRxYhISEYPnw4srOzUVVVhREj\nRjjNSUFEHav0XDWM1fU8qiCv4rJY3H///Thz5gz+8Y9/4MMPP8QHH3yA5ORkKbIRdUn/PnEeAKdR\nJe/i8jTU/PnzMW7cOACAXq/HJ598gg8//NDtwYi6qqMnKgFw5DZ5lxaPLNavXw+bzeYoFI4N5HLM\nnDkTNpsN69atc3tAoq7mtxOVkMtkCAv2vXmaqfNq8ciib9++ePjhhzF8+HAMGzYMffr0gUKhQHFx\nMX766Sf8/PPPePLJJ6XMStTpWW12HD91HtcHaeGn5p1myXu0WCzGjRuH0aNHIyMjAxs3bkRRURFk\nMhlCQ0Nxxx13YM6cOVCrr+1KDbvdjsWLF+Po0aNQq9VITU1FWFiYo33Hjh147733oFQqERcXhylT\nprjchqgzOV1uhsVq5yko8jqt9lmo1WrExcUhLi6uQ15s+/btsFgs2LhxI3JycrB06VKsXLkSQMMY\njiVLlmDz5s3QaDQwGAwYN24c9u/f3+I2RJ1NPm8eSF7KZQf3gQMH8OGHH6KyshJCCMfyjz/++Jpf\nLDs7G2PGjAEAREZG4tChQ46248ePIzQ0FD16NIxYjY6Oxt69e5GTk9PiNkSdTQGnUSUvdVVXQz3y\nyCO44YYbIGvnDc1MJpPTGA2FQgGr1QqlUgmTyQS9/lKHnlarhclkanWb1gQEdIOyHbOLBQV5d+ci\n87WPt+YrKjNB46fArTf1gULuvTcQ9Nb914j5Op7LYuHv74+HH364Q15Mp9PBbDY7HtvtdseX/pVt\nZrMZer2+1W1aU1lZ3eacQUF6lJcb27y9uzFf+3hrvpo6K06VGjFk4HU4V2HydJwWeev+a8R87dNS\nIWvx0tni4mIUFxdj0KBB+Oijj3Dy5EnHsuLi4jaFiIqKQlZWFgAgJycHERERjraBAweiqKgI58+f\nh8Viwb59+zB06NBWtyHqTArPGCEARIT29HQUoiZa/BX9kUcegUwmgxACP/30k1MfhUwmwz//+c9r\nfrEJEyZgz549SExMhBACaWlpyMjIQHV1NRISEvDiiy9ixowZEEIgLi4OwcHBzW5D1Bk1TqMaERrg\n4SRETcnE5b3WnUh7DvN84TCR+drOW/O9+/lB7P+tHGv+9y6Iequn47TIW/dfI+Zrn5ZOQ7k8+X/6\n9GmsX78eFy5ccLoaasmSJR2XjoiQX3wBPXVqXNdT49VfJtQ1uSwWzz33HIYNG4Zhw4a1+2ooImpe\npbEO500WDL3xOk9HIWqWy2JhtVoxf/58KbIQdVmN/RUcX0HeyuUtyqOjo7Fjxw5YLBYp8hB1SZem\nUWWxIO/k8shi27ZtWL9+vdMymUyGI0eOuC0UUVdTUFwFGYD+LBbkpVwWi927d0uRg6jLstsFCs4Y\nEXKdFho/1wNOiTzB5SezoqICGRkZMJvNEELAbrfj1KlTWL58uRT5iDq94goz6iw2noIir+ayzyI5\nORlHjhzB1q1bUVNTgx07dkAud7kZEV2lxpsHcs5t8mYuv/UrKyuxbNkyjBs3DnfddRfWrVuHf//7\n31JkI+oS2LlNvsBlsWi8ZXh4eDjy8vKg1+tRX1/v9mBEXUVBcRVUSjmuD9J6OgpRi1z2WYwYMQLP\nPvss5s+fj+nTpyM3NxcajUaKbESdXl29DafKzRjQtzuUCp7eJe/l8tOZn5+PlJQUXH/99XjzzTcx\nYMAA+Pv7S5GNqNMrOmOEXQgOxiOv1+KRxdNPP428vDyUlZXh8OHDjuVWqxV9+/aVJBxRZ5dfzGlU\nyTe0WCyWLVuG8+fP47XXXsPChQsvbaBUolevXpKEI+rsCko4jSr5hhaLhU6ng06nw8qVK6XMQ9Sl\n5BdXQadR4boePLVL3o09akQecsFsQUVVLQb07c47OpPXY7Eg8pDGwXgcX0G+QNIb0dTW1iIlJQUV\nFRXQarVYtmwZAgMDndb56KOP8NVXXwEAxo4di+TkZAghEBMTg/79+wMAIiMjMW/ePCmjE3W4fPZX\nkA+RtFh8+umniIiIwDPPPIOvvvoK6enpTp3nJ0+exNatW7Fp0ybI5XIYDAbceeed0Gg0GDx4MFat\nWiVlXCK3Krg4hwXvNEu+QNLTUNnZ2RgzZgwAICYmBj/++KNTe58+ffDBBx9AoVBAJpPBarXCz88P\nubm5KC0txbRp0zBr1izk5+dLGZuow9mFQEGJEcEBGug0Kk/HIXLJbUcWmzZtwtq1a52W9erVC3p9\nw2TgWq0WRqPzPMMqlQqBgYEQQmD58uW4+eabER4ejrNnz2L27NmYNGkS9u3bh5SUFGzZsqXV1w8I\n6AalUtHm/C1NWu4tmK99PJ3vVJkR1XVWDB/cp9ksns7nCvO1j7fna47bikV8fDzi4+OdliUnJ8Ns\nNgMAzGYzundvevhdV1eHl156CVqtFn/6058AAEOGDIFC0fDFP2zYMJSVlUEI0eoVJJWV1W3OHhSk\nR3m50fWKHsJ87eMN+bJzSwAAfQM1TbJ4Q77WMF/7+EK+5kh6GioqKgqZmZkAgKysLERHRzu1CyHw\nX//1X/j973+PV155xVEg3n33XcdRSl5eHkJCQnipIfm0fN6WnHyMpB3cBoMB8+fPh8FggEqlwooV\nKwAAa9asQWhoKOx2O3755RdYLBbs2rULADB37lzMnj0bKSkpyMzMhEKhwJIlS6SMTdThCkqqoJDL\nENrb905HUNckabHQaDR4++23myx//PHHHT8fPHiw2W1Xr17ttlxEUqq32nCi1ITQYB1USg51It/A\nTyqRxE6UmWCzCwwI6eHpKERXjcWCSGKXplHlKSjyHSwWRBK7NHKbRxbkO1gsiCRWUFyFbn5K9A7g\njJPkO1gsiCRkqqlHaWUNwvt2h5yXf5MPYbEgklBhCWfGI9/EYkEkocbBeLzTLPkaFgsiCTk6t3lk\nQT6GxYJIIkII5BdX4boe/uiuVXs6DtE1YbEgksjZC7Uw1dSzv4J8EosFkUQcNw9ksSAfxGJBJJEC\nTqNKPozFgkgi+cVVkMtkCOvD23yQ72GxIJKA1WZHUakR/YK08FO1fQZHIk9hsSCSwOlyM+qtdk52\nRD6LxYJIAhxfQb6OxYJIAvnFFwBwGlXyXZLOlFdbW4uUlBRUVFRAq9Vi2bJlCAwMdFonNTUV+/fv\nh1arBQDC0bPxAAAR5klEQVSkp6dDpVK53I7ImxWUGOGnVqBvL62noxC1iaRHFp9++ikiIiLwySef\n4MEHH0R6enqTdXJzc/HBBx9g3bp1WLduHfR6/VVtR+StauqsKDlrRngfPeRy3mmWfJOkRxbZ2dmY\nOXMmACAmJqbJl77dbkdRUREWLVqEs2fPYvLkyZg8ebLL7ZoTENANSmXbrzoJCvLuyxuZr32kzPev\nf5dDABg88Lqrfl3uv/Zhvo7ntmKxadMmrF271mlZr169oNc37CStVguj0ejUXl1djUceeQSPP/44\nbDYbkpKSMGTIEJhMpla3a05lZXWbswcF6VFe7vo1PIX52kfqfL8eOQMA6NPT/6pel/uvfZivfVoq\nZG4rFvHx8YiPj3dalpycDLPZDAAwm83o3t25s0+j0SApKQkaTcMMYiNGjEBeXh50Ol2r2xF5s0u3\nJec0quS7JO2ziIqKQmZmJgAgKysL0dHRTu2FhYUwGAyw2Wyor6/H/v37MXjwYJfbEXkrIQTyS6rQ\nU6dGgN7P03GI2kzSPguDwYD58+fDYDBApVJhxYoVAIA1a9YgNDQU48ePR2xsLKZMmQKVSoXY2Fjc\neOON6NevX7PbEXm7SmMdLpgsiIoI8nQUonaRCSGEp0O4Q3vOCfrCOUXmazsp8+3LK0P6F4cQN3YA\n7h3Z/6q24f5rH+Zrn5b6LDgoj8iNLt1plv0V5NtYLIjcKL+4CjIA/XmnWfJxLBZEbmK3CxSeMaLv\ndVpo/CTtHiTqcCwWRG5SXGFGXb2NM+NRp8BiQeQml8ZXsFiQ72OxIHKTxs5tHllQZ8BiQeQm+cVV\nUCvluD6Id5ol38diQeQGdRYbTpebEdpHD6WC/8zI9/FTTOQGRaVG2IXgzHjUabBYELkBO7eps2Gx\nIHIDzrlNnQ2LBZEbFBRXQd9NhV49/D0dhahDsFgQdbALZgsqqmoRHtIdMhmnUaXOgcWCqIMVsL+C\nOiEWC6IOll9yAQD7K6hzYbEg6mCNRxb9WSyoE2GxIOpAdiGQX2JEcIAGOo3K03GIOoyk902ura1F\nSkoKKioqoNVqsWzZMgQGBjrajxw5grS0NMfjnJwcvPfeexgzZgxiYmLQv39/AEBkZCTmzZsnZXSi\nq1J6rho1dVZE3tDL01GIOpSkxeLTTz9FREQEnnnmGXz11VdIT0/HwoULHe2DBg3CunXrAABff/01\nevfujZiYGBQVFWHw4MFYtWqVlHGJrlnjYDzePJA6G0mLRXZ2NmbOnAkAiImJQXp6erPrVVdX4513\n3sH69esBALm5uSgtLcW0adPg7++PBQsWYMCAAa2+VkBANyiVijZnbWkeWm/BfO3jrnxnzhcAAKIH\nh7TrNbrq/usozNfx3FYsNm3ahLVr1zot69WrF/T6hp2k1WphNDY/afnmzZsxceJExymqoKAgzJ49\nG5MmTcK+ffuQkpKCLVu2tPr6lZXVbc7uCxOqM1/buStf8Vkztu89AX+1AjqVvM2v0VX3X0dhvvZp\nqZC5rVjEx8cjPj7eaVlycjLMZjMAwGw2o3v35g/VMzIy8PbbbzseDxkyBApFw1HCsGHDUFZWBiEE\nBzyR16iuteKdzw+izmLDEw8MhkrJa0eoc5H0Ex0VFYXMzEwAQFZWFqKjo5usYzQaYbFYEBIS4lj2\n7rvvOo5S8vLyEBISwkJBXsMuBP6SkYvSc9WY+B+h+I+bgz0diajDSdpnYTAYMH/+fBgMBqhUKqxY\nsQIAsGbNGoSGhmL8+PEoKCjA9ddf77Td7NmzkZKSgszMTCgUCixZskTK2ESt+vuuAvzreAUG9w/A\n5LEDPR2HyC1kQgjh6RDu0J5zgr5wTpH52q4j82UfLcd7fzuI63r4Y9Fjt3fI2IqutP/cgfnap6U+\nC55YJWqj02fN+OCrw1Cr5Hgm7lYOwqNOjcWCqA2qa+vx7pYDqLPYMP2eQfhdb52nIxG5FYsF0TWy\nC4HVGYdRWlmDSf8RiuGD2KFNnR+LBdE1+mJXAQ4cr8Dg8EDEsUObuggWC6JrkH20DF/+UIignv54\n4oHBkMt5CTd1DSwWRFepoUP7SEOH9h/ZoU1dC4sF0VW4skO7Hzu0qYthsSBywalDewQ7tKlrYrEg\ncqGxQ3tIeCDiYtihTV0TiwVRKy7v0J7NDm3qwlgsiFrQ2KHtp1KwQ5u6PBYLomY4dWjfyw5tIhYL\noivY7Zc6tO8ZEYbbb+rt6UhEHsdiQXSFL3bnN3RoDwjEH2Nan76XqKuQdD4LIm9VZ7HhZLkJRwrP\n4csfitC7p4YjtIkuw2JBXU6V2YLcwnM4UWrEiVITTpQacaaiGo0Tu/ipFUiOuwVaf3ZoEzXySLH4\n7rvvsG3bNsdMeZf77LPPsGHDBiiVSjz11FO44447UFtbi5SUFFRUVECr1WLZsmUIDAz0QHLyJUII\nnKuqw4lSI4oaC0OZEeeq6pzW0/gpEfG7nggN1iM0WIdBYQEI7O7vodRE3knyYpGamordu3dj0KBB\nTdrKy8uxbt06bNmyBXV1dZg6dSpGjRqFTz/9FBEREXjmmWfw1VdfIT09HQsXLpQ6OnUgIUTDb/IC\nEBC4fL5Gm13AarPDahOwWu2w2uyot9lhswnU2+ytLrPaBCqNdReLgxHmWqvT6/bQqTFsUDD6BPgj\ntLceoX30COrhzzndiVyQvFhERUXhzjvvxMaNG5u0HThwAEOHDoVarYZarUZoaCjy8vKQnZ2NmTNn\nAgBiYmKQnp7utnzvb81FzrGz8ObZZmUyWev52hnd1eZXs2sa84mLxeDif5LqHaDBoP6BCAvWXTxq\n0KOHVu3101oSeSO3FYtNmzZh7dq1TsvS0tJwzz334Oeff252G5PJBL3+0vyvWq0WJpPJablWq4XR\n6PofekBANyiVimvOPaBfT1QY61yv6OXa+3uyq1+0Za5eQdaQofE3dpkMTr+9y2QNz9G4qPExZIBc\nLoNKIYdSKYdKKYdKcfFPpQJKhQwqpeLi44b/lQq50+PuWj+E9+2Obq30ObQ0z7C3YL72Yb6O57Zi\nER8fj/j4+GvaRqfTwWw2Ox6bzWbo9Xqn5WazGd27d3f5XJWV1dcW+KIJUddj6t03efVvnt7+m7E3\n5DMba2E21jbb5g35WsN87cN87dNSIfOqcRa33norsrOzUVdXB6PRiOPHjyMiIgJRUVHIzMwEAGRl\nZSE6OtrDSYmIuhavuHR2zZo1CA0Nxfjx4zFt2jRMnToVQgg8//zz8PPzg8FgwPz582EwGKBSqZq9\nioqIiNxHJry5J7cd2nOY5wuHiczXdszXPszXPr6QrzledRqKiIi8E4sFERG5xGJBREQusVgQEZFL\nLBZERORSp70aioiIOg6PLIiIyCUWCyIiconFgoiIXGKxICIil1gsiIjIJRYLIiJyicWCiIhc8opb\nlHvSd999h23btjlue56Tk4PXXnsNCoUCo0ePRnJystP6tbW1SElJQUVFBbRaLZYtW4bAwEC3Zly9\nejV27doFAKiqqsLZs2exZ88ep3VSU1Oxf/9+aLVaAEB6errTrIPuJIRATEwM+vfvDwCIjIzEvHnz\nnNb57LPPsGHDBiiVSjz11FO44447JMkGAEajESkpKTCZTKivr8eLL76IoUOHOq3jif1nt9uxePFi\nHD16FGq1GqmpqQgLC3O079ixA++99x6USiXi4uIwZcoUt+a5Un19PV566SWcPn0aFosFTz31FMaP\nH+9o/+ijj7Bp0ybH5//ll1/GgAEDJM340EMPQafTAQD69euHJUuWONo8vf8+//xz/O1vfwMA1NXV\n4ciRI9izZ49j8jZv2H/XRHRhr776qrj77rvFc88951j2wAMPiKKiImG328XMmTNFbm6u0zYffvih\nePvtt4UQQnz55Zfi1VdflTTz7Nmzxa5du5osT0xMFBUVFZJmaVRYWCieeOKJFtvLysrEfffdJ+rq\n6kRVVZXjZ6n8+c9/FmvWrBFCCHH8+HHx4IMPNlnHE/vvm2++EfPnzxdCCPHrr7+KJ5980tFmsVjE\nnXfeKc6fPy/q6urEH//4R1FeXi5pvs2bN4vU1FQhhBCVlZVi7NixTu3z5s0TBw8elDTT5Wpra0Vs\nbGyzbd6w/y63ePFisWHDBqdlnt5/16pLn4aKiorC4sWLHY9NJhMsFgtCQ0Mhk8kwevRo/PDDD07b\nZGdnY8yYMQCAmJgY/Pjjj5Ll/fbbb9G9e3eMHj3aabndbkdRUREWLVqExMREbN68WbJMAJCbm4vS\n0lJMmzYNs2bNQn5+vlP7gQMHMHToUKjVauj1eoSGhiIvL0+yfI899hgSExMBADabDX5+fk7tntp/\nl3+WIiMjcejQIUfb8ePHERoaih49ekCtViM6Ohp79+6VJFejiRMnYs6cOQAajh4VCuc57XNzc7F6\n9WoYDAa8//77kmYDgLy8PNTU1GD69OlISkpCTk6Oo80b9l+jgwcP4tixY0hISHBa7un9d626xGmo\nTZs2Ye3atU7L0tLScM899+Dnn392LDOZTI5DWgDQarU4efKk03Ymk8lxekKr1cJo7NhJTFrKeuut\nt+L999/Hm2++2WSb6upqPPLII3j88cdhs9mQlJSEIUOG4KabburQbC3lW7RoEWbPno1JkyZh3759\nSElJwZYtWxztl+8zoGG/mUymDs/WUr7G/VdeXo6UlBS89NJLTu1S7r/LXfl5UygUsFqtUCqVku6z\nljSekjOZTHj22Wfx3HPPObXfe++9mDp1KnQ6HZKTk7Fz505JTy/6+/tjxowZiI+PR2FhIWbNmoVt\n27Z5zf5r9P777+Ppp59ustzT++9adYliER8fj/j4eJfr6XQ6mM1mx2Oz2ew4v9jcOs21uyvrsWPH\n0L17d6dz2o00Gg2SkpKg0WgAACNGjEBeXp5bvuyay1dTU+P4rXPYsGEoKyuDEAIymQxA8/vVXf0B\nLe2/o0ePYu7cuXjhhRcwfPhwpzYp99/lrtwvdrsdSqWy2TZ37rPWlJSU4Omnn8bUqVNx//33O5YL\nIfDoo486Mo0dOxaHDx+W9MsuPDwcYWFhkMlkCA8PR8+ePVFeXo6QkBCv2X9VVVUoKCjAiBEjnJZ7\nw/67Vl36NNSVdDodVCoVTpw4ASEEdu/ejWHDhjmtExUVhczMTABAVlYWoqOjJcn2ww8/ICYmptm2\nwsJCGAwG2Gw21NfXY//+/Rg8eLAkuQDg3Xffdfw2n5eXh5CQEEehAIBbb70V2dnZqKurg9FoxPHj\nxxERESFZvmPHjmHOnDlYsWIFxo4d26TdU/svKioKWVlZABourLh8nwwcOBBFRUU4f/48LBYL9u3b\n16RT3t3Onj2L6dOnIyUlBZMnT3ZqM5lMuO+++2A2myGEwM8//4whQ4ZImm/z5s1YunQpAKC0tBQm\nkwlBQUEAvGP/AcDevXsxcuTIJsu9Yf9dqy5xZHEtXn75Zfz3f/83bDYbRo8ejdtuuw0AMH36dKxa\ntQoGgwHz58+HwWCASqVyXEXlbgUFBRg1apTTsjVr1iA0NBTjx49HbGwspkyZApVKhdjYWNx4442S\n5AKA2bNnIyUlBZmZmVAoFI4rUi7PN23aNEydOhVCCDz//PNN+g3cacWKFbBYLHjttdcANPxSsHLl\nSo/vvwkTJmDPnj1ITEyEEAJpaWnIyMhAdXU1EhIS8OKLL2LGjBkQQiAuLg7BwcFuz3S5VatWoaqq\nCunp6UhPTwfQcORWU1ODhIQEPP/880hKSoJarcbIkSObLcTuNHnyZCxYsAAGgwEymQxpaWn4+uuv\nvWb/AQ3/bvv16+d4fPnfr6f337XiLcqJiMglnoYiIiKXWCyIiMglFgsiInKJxYKIiFxisSAiIpdY\nLIiIyCUWCyIiconFgkgCH3/8MR5++GEIIbBv3z7cddddHrtXEVFbcFAekQSEEEhKSsLEiROxbt06\nvPbaa5LdKoaoI7BYEEnk5MmTuP/++x23jCHyJTwNRSSR4uJi6HQ6HD58GPwdjXwNiwWRBMxmM/73\nf/8X6enp0Gg0+OSTTzwdieiasFgQSeD111/H2LFjceuttzqKxpUTaxF5M/ZZEBGRSzyyICIil1gs\niIjIJRYLIiJyicWCiIhcYrEgIiKXWCyIiMglFgsiInLp/wFLGkFPJMgOpgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xba30da0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Graphical representation of our tan-h activation function\n",
    "for i in array([range(-10, 10)]):\n",
    "    x = append(array([]), i)\n",
    "    plt.plot(x, Tanh(x))\n",
    "    plt.title(\"Tan-h activation function\")\n",
    "    plt.xlabel(\"x\")\n",
    "    plt.ylabel(\"tanh(x)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Softmax:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Unlike the Sigmoid activation function, the [**Softmax activation function**](http://dataaspirant.com/2017/03/07/difference-between-softmax-function-and-sigmoid-function/) is used for multi-class classification. Softmax function calculates the probabilities distribution of the event over ‘n’ different events. In general way of saying, this function will calculate the probabilities of each target class over all possible target classes. Later the calculated probabilities will be helpful for determining the target class for the given inputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Equation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Softmax(x) = \\frac{e^j}{\\sum_i{e^i}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Programmatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.01165623,  0.03168492,  0.08612854,  0.23412166,  0.63640865])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from numpy import sum\n",
    "\n",
    "# Softmax activation function\n",
    "def Softmax(x):\n",
    "    return exp(x) / sum(exp(x))\n",
    "\n",
    "x = array([1, 2, 3, 4, 5])\n",
    "\n",
    "Softmax(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ReLU:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Instead of the sigmoid activation function, most recent artificial neural networks use [**rectified linear units**](https://github.com/Kulbear/deep-learning-nano-foundation/wiki/ReLU-and-Softmax-Activation-Functions) (ReLUs) for the hidden layers. A rectified linear unit has output 0 if the input is less than 0, and raw output otherwise. That is, if the input is greater than 0, the output is equal to the input. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Equation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ReLU(x) = max(0,x)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Graphical Representation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://i1.wp.com/laid.delanover.com/wp-content/uploads/2017/07/relu.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Programmatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# ReLU activation function\n",
    "def ReLU(x):\n",
    "    return x * (x > 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0xbfdb278>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAETCAYAAADNpUayAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlYVPXiP/D3LOw7Cu4o7gK531JDSdFABXFDQZPMzOV6\nv2U3Levr9bGrbTeta92rYGaWmfsC7lruS6mppSwaKigICIrAsM12fn/4dX6SArKcOcOZ9+t5fB6Y\nM3POew74nsNnznyOQhAEAUREJCtKqQMQEVH9Y7kTEckQy52ISIZY7kREMsRyJyKSIZY7EZEMsdyp\nWp06dUJ4eDgiIiIwcuRIhISEYMyYMbh06VK1j503bx6+/vrrJy4bNGjQY+u4dOkSBg0aVC+5i4qK\nEBMTY/o+IiIChYWFtVrXkSNHsGzZMgDATz/9hMWLF9dLRgBYsWIFXnjhBbz77rv1tk6gfp8/NTxq\nqQNQw/Dtt9/C09PT9P3XX3+NxYsXY+PGjRKmqlpBQUGFF4/4+Phar+vSpUsoKCgAAAQHByM4OLjO\n+R7asmULlixZgt69e9fbOoH6ff7U8PDInWpMr9cjKysLbm5upttWrFiBUaNGISIiAn/961+Rk5NT\n79vdsmULIiMjMXLkSAwcOBA//PCDaVlcXBxCQ0MRFhaGWbNmoaioCO+++y7KysoQEREBg8GATp06\n4d69e4iKisK+fftMj12yZAk+/fRTlJSU4O2338a4ceMQEhKC0aNH4/r16/jtt9+wYcMG7NmzB59/\n/jm2bduG6dOnAwCys7MxY8YMhIeHIywsDKtWrQIAZGRkYPDgwVi0aBHGjh2LIUOGYM+ePY89p9mz\nZyMnJwf/+7//iz179mDSpEkVsj36/TPPPIMvv/wSUVFRGDRoENasWVOr5w8A//3vfzFs2DCEh4fj\n9ddfR25urml7S5cuxcSJEzFo0CDMnTsXRqOxnn6CZE4sd3oqL7/8MkaMGIHAwECEhIQAAD766CMA\nwI4dO3D16lVs3rwZ8fHxCAoKwvz58+t1+8XFxdi8eTNWrlyJHTt24PPPP8enn34K4MEwybZt27Bx\n40bs2rULLVu2xPfff4+PPvoI9vb2iI+Ph0qlMq0rMjIS27dvBwAYDAYkJCQgMjISx44dg6urKzZt\n2oT9+/cjICAA69atQ7du3RAVFYVhw4bhzTffrJBrzpw5eO6557Bz506sX78eCQkJ2L17NwDg1q1b\nCAwMxJYtWzBnzhxT3kf9+9//hre3N5YsWYJhw4ZVuQ+0Wi08PDywYcMGfPHFF1i6dCnKy8tr/Py3\nbt2K48ePY8uWLdi5cyc6dOiAefPmmZbfvHkTa9euRUJCAn7++WecOXOmhj8tsgQclqGn8nBYJikp\nCa+99hp69OiBRo0aAQAOHz6MS5cuYcyYMQAAo9GI0tLSatepUCgeu00QBCiVjx9zODk5ITY2FkeP\nHkVaWhpSUlJQUlICADh9+jRCQ0NNf0k8HLvOyMh44naHDh2Kf/3rX8jNzUVSUhJat26NNm3aoE2b\nNmjVqhXWrl2L9PR0nDlzBj169Kg0f0lJCc6fP4/Vq1cDAFxcXDB69GgcO3YM3bp1g42NDYKCggAA\nfn5+uH//frX7pDoPh4P8/f2h1WpRUlJS4+d/7NgxjB49Go6OjgCAmJgYxMbGQqvVAgAGDhwIpVIJ\nZ2dntG7d2jQcRQ0Ly51qxM/PD++++y7mz5+Pbt26oWXLljAajZg6dSomTJgA4MER5tMUgoeHx2OF\nl5eXB3d398fum52djfHjx2PcuHHo1asXQkNDcfjwYQCASqWq8EJRWFhY5RuHjo6OCAkJwa5du3Dh\nwgVERkYCAH744Qds2rQJEydORHh4ONzd3SstSODBi9ifp2YyGo3Q6/UAABsbG9ML1ZNeyCrz6Dp1\nOl2FZXZ2dhXWJwhCjZ9/VZkBwN7e3vS1QqF47P7UMHBYhmosLCwM3bt3x4cffggApqEHjUYDAFi2\nbBnefvvtatczYMAArF+/3nTEWFpaio0bN5qOdh91+fJleHp64q9//Sv69+9vKnaDwYB+/frh4MGD\npu1/+eWXWLNmDdRqNQwGwxPLady4cdi2bRsuXLhgGmY6ceIERo0ahcjISPj6+uLQoUMwGAwAHryA\nPFqAAODs7Ixu3bph3bp1AB6cnbJjxw7069ev+p1YCU9PT1y+fBnAg+GRK1euVPuYmj7/wMBAbNu2\nzfSXz9q1a/GXv/wFtra2tc5NlodH7lQr//jHPzBixAgcP34ckZGRyMnJwbhx46BQKNCsWTN8/PHH\npvt+/vnn+M9//mP6fuDAgfjss88wY8YMfPbZZxg1apSpPIODg01vVj7q+eefx5YtWxAaGgoHBwd0\n7doVnp6eSE9PR1BQEFJTUxEdHQ0AaN++PRYtWgQHBwf4+flh6NChWL9+fYX1BQQEQK1WIyQkxHQ0\nPGXKFCxYsADbtm2DSqWCv78/rl69CgDo27cv/ud//gc2Njbw9/c3rWfJkiX45z//iW3btkGr1SI8\nPByjR49GZmZmrfbrzJkzMW/ePBw9ehRt27Z9qjNoavr8x44di6ysLERGRsJoNKJ169ZYsmRJrfKS\n5VJwyl8iIvnhsAwRkQyx3ImIZIjlTkQkQyx3IiIZspizZXJzi2r9WA8PR+Tnl9RjmvrFfHXDfHXD\nfHVj6fm8vFyeeLssjtzValX1d5IQ89UN89UN89WNpeerjCzKnYiIKmK5ExHJEMudiEiGWO5ERDLE\ncicikiGWOxGRDLHciYhkiOVORCSRnPwSbD6SCq3OUO/rZrkTEUmgXGfAl1svYe/PN5FXUFbv62e5\nExFJYMNPf+B2XjGCe7VE88ZO9b5+ljsRkZmdTbmDoxdvo5W3M8YNbCfKNljuRERmlFdQijV7U2Br\no8SMCH/YiDR3DcudiMhMDEYjViYkobRcj4mDO6JZo/ofjnmI5U5EZCbxJ24gNbMAz3bxRmDXZqJu\ni+VORGQGyWn3sPtUOhq72SMmpDMUCoWo22O5ExGJrKhEi5W7kqBUKjA9wh+O9uJfJ4nlTkQkIkEQ\nsHp3Mgo0Wowa0BbtmruZZbssdyIiEf14LgO/XbsLvzYeCH3Ox2zbZbkTEYkkPbsIm4+kwsXRBlPD\n/KAUeZz9USx3IiIRlGn1iE1IhN4gYGqYH9yd7cy6fZY7EZEI1h28ipx7JQh5thWeadvI7NtnuRMR\n1bOfE7Nx8lI2Wjd1wZggcaYXqA7LnYioHt3JL8F3+6/AzlaFGRH+UKukqVmWOxFRPdEbjIhLSESZ\n1oCYFzuhiYejZFlY7kRE9WTbseu4kVWEvv5N0TegqaRZWO5ERPXg8o272PfLTXh7OOClFztKHYfl\nTkRUVwXFWqzalQyVUoEZEf5wsBN/eoHqsNyJiOrAKAhYtSsJhcVajH2hHdo0dZU6EgBAtJcXnU6H\nefPmITMzE0qlEosWLUK7dtKcEkREJJYDZ24h8cY9PNO2EYb8pZXUcUxEO3I/evQo9Ho9NmzYgFmz\nZuHf//63WJsiIpLEjaxCbD16DW5Otnh1eBezTi9QHdGO3H19fWEwGGA0GqHRaKBWV70pDw9HqOtw\nuSkvL5daP9YcmK9umK9umK9unpSvpEyHr776GUZBwJyXeqFdG/N/CrUqopW7o6MjMjMzMXToUOTn\n5yM2NrbK++fnl9R6W15eLsjNLar148XGfHXDfHXDfHXzpHyCIOCrnUnIvluCYX1ao4WHg2TPobIX\nRtGGZdasWYPAwEDs378f8fHxmDdvHsrLy8XaHBGR2Zy6nI2fk3LQrrkrRvb3lTrOE4l25O7q6gob\nGxsAgJubG/R6PQwGg1ibIyIyi+x7Jfj+wFU42KkwbYR00wtUR7Rynzx5Mt577z1MmDABOp0Ob775\nJhwdpfsoLhFRXen0RsTGX0a5zoAZEf7wcneQOlKlRCt3JycnLFu2TKzVExGZ3ZYj13AzR4P+XZvh\n2S5NpI5TJcv8e4KIyML8lpqHg+duoVkjR0wYLP30AtVhuRMRVSO/qBxf706GWqXE9BH+sLOt/Wnb\n5sJyJyKqgsEo4KudidCU6jB+UHv4NLHsc/IfYrkTEVVh66E/kHLzPnp0aIxBPVtIHeepsdyJiCqR\nmlGAdftT4OFih1eGdYHCgqYXqA7LnYjoCUrKdIhLSAQEAdPC/eDsYCN1pBphuRMR/YkgCFiz7wru\nFpZh3OBO6OTjIXWkGmO5ExH9ybHfbuNcyh10aOmGqCGWf9rjk7DciYgekZlXjPU//gEnezWmhftD\nZaHTC1SnYaYmIhKBVmdAXPxlaPVGTB7aBY3c7KWOVGssdyKi/7PxcCoycosxsEcL9OrkJXWcOmG5\nExEB+PVKLg6fz0QLLyeMH9Re6jh1xnInIqt3t6AMa/Ymw1atxIwR/rC1sfzpBarDciciq2YwGvHV\nzkQUl+kRNbgDWng5Sx2pXrDciciq7TyZhqsZBejdyQtB3ZpLHafesNyJyGpduZmPnafS0MjVDi8P\n7dygpheoDsudiKySplSHlTuToIAC00cEwMm+YU0vUB2WOxFZHUEQ8M2eZOQXlSOivy/at3STOlK9\nY7kTkdU5dD4TF/7IQ2cfdwzv01rqOKJguRORVbmZU4SNh1Lh7GCD18L9oVTKZ5z9USx3IrIa5VoD\n4hISoTcYMWV4F3i42EkdSTQsdyKyGut/uoqsuyUY3LslurdvLHUcUbHcicgqnEnOwbHfsuDj7YzI\nFxr+9ALVYbkTkezl3i/Ft/tSYGejwvQIf9io5V998n+GRGTV9AYjViYkorTcgIlDOqJZIyepI5kF\ny52IZC3+xA1cu12IPn5N8PwzTaWOYzYsdyKSraS0e9hzOh1e7vaYFNJJVtMLVIflTkSyVFisxVc7\nk6BUPphewMFOLXUks2K5E5HsGAUBX+9ORkGxFqOD2qJtc1epI5kdy52IZOfHs7dw6fpd+Pt6IuRZ\nH6njSILlTkSykpZdiM1HrsHV0QZTh3eB0orG2R/Fcici2Sgt1yM2PhEGo4Cp4X5wc5bv9ALVYbkT\nkWysO3gVd/JLEfqcDwJ8G0kdR1IsdyKShdOXs3HqcjZ8m7lg9IC2UseRHMudiBq8nHsl+O7AFdjb\nqjB9hD/UKlabqCd+xsXF4dChQ9DpdIiOjkZkZKSYmyMiK6Q3GBGbkIhyrQHTwv3g7eEodSSLIFq5\n//LLL7hw4QLWr1+P0tJSrF69WqxNEZEV23r0GtKzi/D8M03Rx996pheojmjlfuLECXTs2BGzZs2C\nRqPB22+/LdamiMhKXbp+F/vP3EITT0dMHNJR6jgWRSEIgiDGiufPn4/bt28jNjYWGRkZmDlzJvbt\n21fp3A56vQFqtUqMKEQkQ/cKy/D60sMoLtVjyev90a6lu9SRLIpoR+7u7u5o27YtbG1t0bZtW9jZ\n2eHevXto1OjJpyfl55fUelteXi7IzS2q9ePFxnx1w3x1I8d8RkHAZxsvokCjRXRwB7jaqUR7jg1h\n/z2JaG8p9+rVC8ePH4cgCMjJyUFpaSnc3fnKSkR1t++Xm0hKy0fXdo0wuHdLqeNYJNGO3AcOHIiz\nZ89i7NixEAQBCxYsgErFYRciqptrtwuw/dh1uDnbYsrwLlY1jW9NiHoqJN9EJaL6VFKmR1x8IoxG\nAdPC/eHqaCt1JIvFM/2JqEEQBAHf7U9BXkEZhvdrjS6tPaSOZNFY7kTUIJz4PQtnku+gXQtXjHje\nV+o4Fo/lTkQWL+tuMdb9eBUOdmpMD+f0Ak+De4iILJpOb0BsfCK0OiMmD+2Mxu4OUkdqEFjuRGTR\nNh2+hlt3NBjQrTn+0tlb6jgNBsudiCzWxT/y8NOvGWje2AnRgztIHadBYbkTkUXKLyrH6j3JUKuU\nmDHCH3Y2/JxMTbDcicjiGI0CViYkQlOqQ1Rwe7T0dpY6UoPDcicii7P7dBqu3LqPnh29MLBHC6nj\nNEgsdyKyKH9k3Ef8iTR4uNhh8tDOnF6glp56+oErV64gPT0dSqUSPj4+6NiRcycTUf0qLtNhZUIi\nBAiYPsIfzg42UkdqsKosd0EQsH79enz77bdwcnJC8+bNoVarkZGRAY1Gg5iYGERFRUGp5B8ARFQ3\ngiBgzd4U3C0sR0SgLzq24iyydVFlub/++uvo168fNm3aBDc3twrLioqKsH37dsyaNQsrVqwQNSQR\nyd/Ri7fx65VcdGzljvB+baSO0+BVWe6ffPIJHB2ffLFZFxcXxMTEYOzYsaIEIyLrkZGrwfqf/oCT\nvRrTwv2gVHKcva6qHE95WOxLliyBXq833Z6bm4sZM2ZUuA8RUW2U6wyIi0+ETm/ElGFd4OlqL3Uk\nWXiqwfKCggJERkYiNTUV8fHxiIyMxHPPPSd2NiKyAl/HX0ZmXjEG9WyBHh29pI4jG091tsyiRYuw\ne/duREREwMPDA+vXr0erVq3EzkZEMvfrlTvYezoNLb2cMG5ge6njyMpTHblv3boV//rXvzB79mz0\n798fb7zxBpKTk8XORkQydregDN/sSYGtjQrTIwJgy+kF6tVTHblv2LABq1evRrt27QAAR44cwaxZ\ns3Do0CFRwxGRPBmMRsTtTERJuR5/i+yOFo2dpI4kO1WWe3l5Oezs7LBx48YK57K/8MIL6NWrV4X7\nEBE9rYQTaUjNKEDvzt548Tkf5OVppI4kO1UOy8yZMwebNm1CSUnJY8sUCgXWrVuHv//976KFIyL5\nSUnPx65TaWjkao/JoZ04vYBIqjxyX7ZsGdavX4+xY8fC1dUVTZs2hUqlQmZmJu7fv4+YmBgsW7bM\nXFmJqIHTlOrw1a4kKBQKTI/wh6M9pxcQS5XlrlQqMXHiREycOBEpKSlIS0szzS3TuXNnc2UkIhkQ\nBAGrdycjv6gcY4Laon0Lt+ofRLVWZbnfvn3b9LWrqyu6d+8OV1dXfnCJiGrs0PlMXEzNQ5fWHhj6\nXGup48heleX+0ksvQaFQQBAE0235+fno2LEjPvvsM7RowXmWiah6N3OKsPFQKpwdbDA1jNMLmEOV\n5V7ZqY47duzA+++/j5UrV4oSiojko1xrQGx8IvQGI6aGBcDDhWfXmUOt5uodOXIksrKy6jsLEcnQ\nuh+vIvteCYb0boWu7RpLHcdqcCJ2IhLNL0k5OPF7FnyaOGPsC+2kjmNVnvpKTA9pNBps3rwZrVvz\nDREiqlzu/VJ8tz8FdjYqzIgIgI2ax5LmVGW5d+5c8fqFCoUCrq6u6Nu3LxYuXCh2NiJqoPQGI+IS\nElFabsCrw7ugqSfPsDO3Kss9JSXFXDmISEZ2HL+B67cL0ce/CfoFNJU6jlWq9d9J06ZNq88cRCQT\niWn3sPfndHi7O2DSi5xeQCq1Lvdz587VZw4ikoHCYi1W7UyCUvlgegEHuxq/rUf1hO9wEFG9MAoC\nVu1OQkGxFmOC2sG3mavUkaway52I6sXBs7dw+fo9BPh64sVneaU2qVX5N9OgQYOeOF4mCALKyspE\nC0VEDUtadiG2HLkGVydbvBrmByXH2SVXZbmvXbu2Tiu/e/cuRo8eXeEqTkQkL6XlesTGJ8JgFDA1\nrAvcnGyljkSoZlimRYsWpn/nz5/Hpk2b4OnpibNnz1Y7aZhOp8OCBQtgb29fr4GJyLJ8f+Aq7uSX\nYuhzPgjwbSR1HPo/TzXmvmTJEhw9ehQHDhyAwWDA1q1b8fHHH1f5mE8++QRRUVHw9vaul6BEZHlO\nXc7C6cRs+DZzxagBbaWOQ494qvOUTpw4ge3bt2PUqFFwdnbGN998gxEjRmDevHlPvP+2bdvg6emJ\n/v37P/XMkR4ejlCra3/1cy8vl1o/1hyYr26Yr27EyHc7V4PvD1yFg50a773yLJo2qv1Frq1x/4nt\nqcr94cWxH765qtVqK1ww+8+2bt0KhUKB06dPIzk5Ge+88w5WrFgBLy+vSh+Tn//4dVqflpeXC3Jz\ni2r9eLExX90wX92IkU+nN+LDtb+iTGvAtBF+UBmNtd6GNe6/+lTZC89TlXtoaChmz56NgoICrFmz\nBvHx8QgLC6v0/uvWrTN9PWnSJCxcuLDKYieihmXr0WtIzylC4DPN0MeP0wtYomrL/fr164iIiECX\nLl3QvHlzZGdnY/LkyfyEKpGV+v3aXRw4ewtNPR0xcUhHqeNQJap8Q/XLL7/EmDFjEBoaCqVSiTlz\n5sDT0xPvv/9+heurVmXt2rU8DZJIJu5ryvH17iSoVQrMiPCHnW3t3ycjcVV55L5jxw7s378fd+7c\nwRdffIFVq1YhLy8Py5YtQ//+/c2VkYgsgFEQ8NXOJBSV6BA9uAN8mjS8NxmtSZXl7uTkBG9vb3h7\ne+P333/HyJEjsWrVKqhUfLUmsjZ7f05Hcno+urdvjMG9Wkodh6pRZbk/ekaMh4dHpac+EpG8Xcss\nwPZjN+DubItXhnXmNL4NQJVj7o/+APlJUyLrVFKmR1xCIgRBwGvh/nBx5PQCDUGVR+5//PEHgoOD\nAQA5OTmmrwVBgEKhwE8//SR+QiKSjCAI+G5/CvIKyhDWrw26tPaQOhI9pSrLff/+/ebKQUQW6Pjv\nWTiTfAftW7ghIrCN1HGoBqos9+omByMi+bqdV4wfDl6Fo536wadQq/hUOlke/rSI6DE6vQGx8YnQ\n6o2YPLQzGrs5SB2JaojlTkSP2XToGjJyNXihe3P07syZXRsiljsRVXDhai5+Op+BFo2dMD64g9Rx\nqJZY7kRkcq+wDKv3JMNGrcT0CH/Y2fADiw0Vy52IAABG44PpBYrL9IgK7oCWXs5SR6I6YLkTEQBg\n16k0XLl1H706euGF7s2ljkN1xHInIly9dR/xJ2/A09UOLw/l9AJywHInsnKaUh1W7kwEAEwL94ez\ng43Eiag+sNyJrJggCFizNwX3CssREeiLjq3cpY5E9YTlTmTFjlzIxPmruejUyh1hfdtIHYfqEcud\nyEpl3NFg/U+pcLJX47VwPyiVHGeXE5Y7kRUq1xkQm5AIvcGIKcO7wNOVU3rLDcudyApt+OkP3M4r\nRnCvlujRwUvqOCQCljuRlTmbcgdHL95GK29njBvIi9fLFcudyIrkFZRizd4U2NooMSPCHzZqTi8g\nVyx3IithMBqxMiEJpeV6TBzcEc0aOUkdiUTEcieyEvEnbiA1swDPdvFGYNdmUschkbHciaxActo9\n7D6VjsZu9ogJ4fQC1oDlTiRzBZpyrNyVBKVSgekR/nC0r/LqmiQTLHciGRMEAcs2XkCBRotRA9qi\nXXM3qSORmbDciWTsx3MZOJuUA782Hgh9zkfqOGRGLHcimUrPLsLmI6lwc7bF1DA/KDnOblVY7kQy\nVKbV/9/0AgLejO4Jd2c7qSORmbHciWRo3cGryLlXgpBnW6FX5yZSxyEJsNyJZObnxGycvJSN1k1d\nMCaI0wtYK5Y7kYzcyS/Bd/uvwM5WhRkR/lCr+F/cWvEnTyQTeoMRcQmJKNMaEPNiJzTxcJQ6EkmI\n5U4kE9uOXceNrCL0C2iKvgFNpY5DEmO5E8nA5et3se+Xm/D2cMDEIR2ljkMWQJTPIet0Orz33nvI\nzMyEVqvFzJkzERwcLMamiKxeQbEWq3YlQaVUYEaEPxzsOL0AiVTuCQkJcHd3x6effor79+9j5MiR\nLHciERgFAat2JaGwRIeoQe3Rpqmr1JHIQohS7qGhoQgJCQHwYG4LlYoXBCASw/4zN5F44x66tmuE\nIX9pJXUcsiAKQRAEsVau0Wgwc+ZMjBs3DuHh4VXeV683QM2rwhA9tas38/H2l8fh6mSLL94aCHcX\nfgqV/j/RBueysrIwa9YsTJgwodpiB4D8/JJab8vLywW5uUW1frzYmK9umO9xpeV6fPztGRiNAl4d\n3gW6Mi1yy7QWk68mmK9uvLxcnni7KOWel5eHKVOmYMGCBejbt68YmyCyWoIgYO3+K8i9X4bhfVvD\nr42n1JHIAolyKmRsbCwKCwuxfPlyTJo0CZMmTUJZWZkYmyKyOicvZePnpBy0a+6KiEBfqeOQhRLl\nyH3+/PmYP3++GKsmsmpZd4ux7uBVONipMG0EpxegyvE3g6iB0OkfTC9QrjPg5dDO8HJ3kDoSWTCW\nO1EDsflIKm7maDCgWzM824XT+FLVWO5EDcDF1Dz8eC4DzRo5IjqY0wtQ9VjuRBYuv6gcq3cnQ61S\nYkZEAOxs+XkQqh7LnciCGY0CvtqZCE2pDuMHtUcrb2epI1EDwXInsmC7f05Hys376NGhMQb1bCF1\nHGpAWO5EFio1owDxx2/Aw8UOrwzrAoVCIXUkakBY7kQWqKRMh7iERAgQMC3cD84ONlJHogaG5U5k\nYQRBwJq9KbhbWIbwfm3QycdD6kjUALHciSzM0d9u49yVXHRs6Ybw59tIHYcaKJY7kQXJzNVg/Y9/\nwMlejWkj/KFS8r8o1Q5/c4gshFZnQGxCInR6IyYP7QJPV3upI1EDxnInshAbD6UiM7cYA3u0QK9O\nXlLHoQaO5U5kAX69kovDFzLR0ssJ4we1lzoOyQDLnUhidwvKsGZvMmzVSkyPCICtDacXoLpjuRNJ\nyGA0YuXORBSX6RE9uANaNHaSOhLJBMudSEI7T6bhj4wC9O7sjQHdmksdh2SE5U4kkSs387HzVBoa\nudpjcmgnTi9A9YrlTiQBTakOK3cmQQEFpo/wh6M9pxeg+sVyJzIzQRCwency8ovKEdHfF+1bukkd\niWSI5U5kZofOZ+Jiah46+7hjeJ/WUschmWK5E5nRzZwibDyUCmcHG7wW7g+lkuPsJA6WO5GZlGsN\niEtIhN5gxKvDu8DDxU7qSCRjLHciM/nhx6vIuluCwb1bolv7xlLHIZljuROZwZnkHBz/PQs+TZwR\n+QKnFyDxsdyJRJZ7vxTf7kuBnY0KMyICYKPmfzsSH3/LiESkNxgRl5CI0nIDJg7piKaejlJHIivB\ncicSUfyJG7h+uxB9/Jrg+WeaSh2HrAjLnUgkSWn3sOd0Orzc7TEphNMLkHmx3IlEUFisxVc7k6BU\nKjB9RAAc7NRSRyIrw3InqmdGQcDXu5NRUKzF6KC2aNvcVepIZIVY7kT17Mezt3Dp+l0E+Hoi5Fkf\nqeOQlWJT0bLJAAAJuklEQVS5E9WjtOxCbD5yDa6ONng1zA9KjrOTRFjuRPWktFyP2PhEGIwCpob7\nwc3JVupIZMVY7kT1ZN3Bq7iTX4rQ53wQ4NtI6jhk5VjuRPXg9OVsnLqcDd9mLhg9oK3UcYgg2vlZ\nRqMRCxcuxJUrV2Bra4vFixejdWvOXU3ycztXg+8OXIG9rQrTR/hDreIxE0lPtN/CH3/8EVqtFhs3\nbsRbb72Fjz/+WKxNEUlGbzDi0+/PoVxrQExIJ3h7cHoBsgyiHbn/+uuv6N+/PwCge/fuuHz5sijb\nWXfgKk5czoIgCKKsvz4oFArmqwNLzmc0Pij4559pij7+nF6ALIdo5a7RaODs7Gz6XqVSQa/XQ61+\n8iY9PByhVqtqvB3fVu5Iz9XUOidRXbVo7IxZkd0s+lOoXl4uUkeoEvPVP9F+G52dnVFcXGz63mg0\nVlrsAJCfX1Kr7fTr4o2IAe2Qm1tUq8ebg5eXC/PVQUPJZ6mHGA1l/1mqhpDvSUQbc+/ZsyeOHTsG\nALh48SI6duwo1qaIiOhPRDtyHzJkCE6ePImoqCgIgoAPP/xQrE0REdGfiFbuSqUS//znP8VaPRER\nVYEn5BIRyRDLnYhIhljuREQyxHInIpIhljsRkQwpBEv9XDcREdUaj9yJiGSI5U5EJEMsdyIiGWK5\nExHJEMudiEiGWO5ERDLEcicikiHLvXRMJQ4ePIh9+/Zh6dKlAB7MFf/BBx9ApVIhMDAQf/vb3yrc\nv6ysDHPnzsXdu3fh5OSETz75BJ6enqJmXLlyJY4fPw4AKCwsRF5eHk6ePFnhPosXL8b58+fh5OQE\nAFi+fDlcXMxztRdBEDBgwAC0adMGwIPLIL711lsV7rNp0yZs2LABarUaM2fOxMCBA82SDQCKioow\nd+5caDQa6HQ6zJs3Dz169KhwHyn2X3UXfT906BD++9//Qq1WY8yYMRg3bpyoef5Mp9PhvffeQ2Zm\nJrRaLWbOnIng4GDT8jVr1mDz5s2m3//3338fbdu2NWvGUaNGma7Q1rJlS3z00UemZVLvv23btmH7\n9u0AgPLyciQnJ+PkyZNwdXUFYBn7r0aEBmTRokVCSEiIMHv2bNNtI0aMENLT0wWj0ShMnTpVSExM\nrPCY1atXC1988YUgCIKwa9cuYdGiRWbNPG3aNOH48eOP3R4VFSXcvXvXrFkeSktLE6ZPn17p8jt3\n7ghhYWFCeXm5UFhYaPraXJYtWyZ88803giAIwrVr14SRI0c+dh8p9t/+/fuFd955RxAEQbhw4YIw\nY8YM0zKtVisMHjxYuH//vlBeXi6MHj1ayM3NNWu+LVu2CIsXLxYEQRDy8/OFoKCgCsvfeust4dKl\nS2bN9KiysjIhIiLiicssYf89auHChcKGDRsq3Cb1/qupBjUs07NnTyxcuND0vUajgVarhY+PDxQK\nBQIDA3Hq1KkKj3n0Qt0DBgzA6dOnzZb3wIEDcHV1RWBgYIXbjUYj0tPTsWDBAkRFRWHLli1mywQA\niYmJyMnJwaRJk/Daa6/h+vXrFZb//vvv6NGjB2xtbeHi4gIfHx+kpKSYLd/kyZMRFRUFADAYDLCz\ns6uwXKr9V9VF369duwYfHx+4ubnB1tYWvXr1wtmzZ82S66HQ0FC88cYbAB78daZSVbwmcWJiIlau\nXIno6GjExcWZNRsApKSkoLS0FFOmTEFMTAwuXrxoWmYJ+++hS5cuITU1FePHj69wu9T7r6Ysclhm\n8+bN+Pbbbyvc9uGHH2LYsGH45ZdfTLf9+SLcTk5OuHXrVoXHaTQa05/rTk5OKCqq32shVpa1a9eu\niIuLw2efffbYY0pKSvDSSy/hlVdegcFgQExMDAICAtC5c+d6zVZZvgULFmDatGkYOnQozp07h7lz\n52Lr1q2m5Y/uM+DBftNoxLlCaFX7Lzc3F3PnzsV7771XYbk599+jqrrouzn3WWUeDlFpNBq8/vrr\nmD17doXlw4cPx4QJE+Ds7Iy//e1vOHz4sFmH2+zt7fHqq68iMjISaWlpeO2117Bv3z6L2X8PxcXF\nYdasWY/dLvX+qymLLPfIyEhERkZWe78/X4S7uLjYND72pPs8ablYWVNTU+Hq6lphTPYhBwcHxMTE\nwMHBAQDQp08fpKSkiFJOT8pXWlpqOqrr3bs37ty5A0EQoFAoADx5v4o1nl3Z/rty5Qr+/ve/4+23\n38azzz5bYZk599+jqrrouzn3WVWysrIwa9YsTJgwAeHh4abbBUHAyy+/bMoUFBSEpKQks5aTr68v\nWrduDYVCAV9fX7i7uyM3NxfNmjWzmP1XWFiIGzduoE+fPhVut4T9V1MNaljmz5ydnWFjY4ObN29C\nEAScOHECvXv3rnCfnj174ujRowCAY8eOoVevXmbJdurUKQwYMOCJy9LS0hAdHQ2DwQCdTofz58/D\n39/fLLkA4D//+Y/paDklJQXNmjUzFTsAdO3aFb/++ivKy8tRVFSEa9eumfUC56mpqXjjjTewdOlS\nBAUFPbZcqv1X1UXf27Vrh/T0dNy/fx9arRbnzp177E1gseXl5WHKlCmYO3cuxo4dW2GZRqNBWFgY\niouLIQgCfvnlFwQEBJg135YtW/Dxxx8DAHJycqDRaODl5QXAMvYfAJw9exZ9+/Z97HZL2H81ZZFH\n7jXx/vvvY86cOTAYDAgMDES3bt0AAFOmTEFsbCyio6PxzjvvIDo6GjY2NqazbMR248YNPP/88xVu\n++abb+Dj44Pg4GBERERg3LhxsLGxQUREBDp06GCWXAAwbdo0zJ07F0ePHoVKpTKdsfBovkmTJmHC\nhAkQBAFvvvnmY+PeYlq6dCm0Wi0++OADAA9exFesWCH5/nvSRd937tyJkpISjB8/HvPmzcOrr74K\nQRAwZswYNGnSRPRMj4qNjUVhYSGWL1+O5cuXA3jwl1FpaSnGjx+PN998EzExMbC1tUXfvn2f+MIp\nprFjx+Ldd99FdHQ0FAoFPvzwQ+zdu9di9h/w4P9ty5YtTd8/+vOVev/VFKf8JSKSoQY9LENERE/G\ncicikiGWOxGRDLHciYhkiOVORCRDLHciIhliuRMRyRDLnagS3333HSZOnAhBEHDu3Dm8+OKLks13\nQlRT/BATUSUEQUBMTAxCQ0Oxdu1afPDBB2abvoKorljuRFW4desWwsPDTdNYEDUUHJYhqsLt27fh\n7OyMpKQk8DiIGhKWO1EliouL8Y9//APLly+Hg4MDfvjhB6kjET01ljtRJT799FMEBQWha9euppL/\n88VgiCwVx9yJiGSIR+5ERDLEcicikiGWOxGRDLHciYhkiOVORCRDLHciIhliuRMRydD/AxwXH1Rz\nteQnAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xbff4438>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Graphical representation of our ReLU activation function\n",
    "x = array(range(-10, 10))\n",
    "plt.plot(x, ReLU(x))\n",
    "plt.title(\"ReLU activation function\")\n",
    "plt.xlabel(\"x\")\n",
    "plt.ylabel(\"ReLU(x)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Leaky ReLU:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### Definition:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **Leaky ReLU** activation function works the same way as the ReLU activation function except that instead of replacing the negative values of the inputs with 0 the latter get multiplied by a small alpha value in an attempt to avoid the \"dying ReLU\" problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Equation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "  LeakyReLU(x)=\\begin{cases}\n",
    "    x, & \\text{if x > 0}.\\\\\n",
    "    \\alpha x, & \\text{otherwise}.\n",
    "  \\end{cases}\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Graphical Representation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://cdn.tinymind.com/static/img/learn/leaky-relu.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Programmatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from numpy import where"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Leaky ReLU activation function\n",
    "def LeakyReLU(x):\n",
    "    return where(x > 0, x, x * 0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0xc078b70>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAETCAYAAADNpUayAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XdAFOfiPfyzy7K0pYtdESzBAhos0Yi9t4i9ol6NJooa\nTWKJr8k1zViiN4klmmZy/SUxV0FF89UYYzdq7BQBFRUFUarAAtuf9w/iRhREgd2F5Xz+ka1zdsDD\n8MzMMxIhhAAREVkVqaUDEBFRxWO5ExFZIZY7EZEVYrkTEVkhljsRkRViuRMRWSGWu5VKSkrCiy++\nWGHvd+bMGQwePLjMr1+8eDG6dOmCoUOHYujQoRgyZAh69eqFr7/+ulzLXrduHT744IMn7h88eDDO\nnDlT5ryPWr9+PQ4ePAgA+Pzzz7Fr164yvU9ubi4mTZpkvD106FDk5ORUSMbY2Fj07t0bw4YNQ1JS\nUoW850MV9fnJvGSWDkDVx5QpUzBt2jTj7bt372LgwIHo2bMnGjdubMFkT3fmzBk0adIEAPDGG2+U\n+X2ys7MRFRVlvL179+5yZ3vojz/+wEsvvYSPP/64wt7zoYr6/GReLPdqSKPR4NNPP8XZs2eh1+vR\nokULLF26FAqFAocPH8bmzZuh0WiQmZmJ4OBgzJs3r8jrz507hwULFmDNmjUIDw+Hh4cH3nzzTQBA\nREQEfvvtN2zYsKHUHPfu3QMAKBQKAMCFCxfw6aefoqCgABKJBHPmzEGPHj0q9LPfvHkTH3zwAfLz\n85Gamgo/Pz989tlnsLOzw+XLl/HRRx+hoKAAtra2WLhwIW7cuIHo6GisWrUKNjY2+OOPP9C0aVMo\nFAocOnQImzdvBgAkJCRgypQpOHLkCHbu3IlffvkFWq0W2dnZmD59OsaPH4933nkHKpUKQ4cORXh4\nOFq0aIFTp07Bw8MDGzZswK+//gobGxv4+Pjg3XffhZeXF0JCQtCmTRtcuHABKSkpaNu2LVauXAmp\n9J8/uiMiIvDzzz9Dr9dDpVKhc+fO+O2334zZwsPDjbcXL14MhUKB+Ph43Lt3D76+vli7di2cnJye\n6/NPmzYN586dw6pVq4zPnzdvHrp27Yrw8HD8/vvvkEqlSExMhK2tLVauXIlmzZpV6PeSSiHIKt25\nc0e0adOm2MfWrVsnVqxYIQwGgxBCiDVr1oh///vfwmAwiIkTJ4qbN28KIYS4d++eaN68ucjIyBCn\nT58WgwYNEqdOnRK9e/cWsbGxQgghrly5Ijp37iy0Wq0QQojx48eLY8eOPbHMRYsWiaCgIPHKK6+I\nXr16iQ4dOoiZM2eKU6dOCSGEePDggejbt6+4c+eOcdldu3YVycnJxmUX54svvhDvv//+E/cPGjRI\nnD59+on7V6xYIXbt2iWEEEKj0YjBgweL/fv3C41GIzp37iwOHz4shBAiKipKDB48WOj1ejFx4kSx\nb98+4+f45ptvRG5urmjbtq1ITU0VQgixatUqsXbtWqFUKsXo0aNFZmamEEKIixcvGr8Pj39PmjVr\nJjIyMsSOHTvEmDFjRF5envEzTZ06VQghxMSJE8XcuXOFXq8Xubm5IigoyLjOSloPYWFhYsaMGcbH\nHr29aNEiMWbMGKFWq4VGoxHBwcFix44dz/35MzMzRadOncSlS5eEEEJcvXpVdOjQQdy+fVuEhYWJ\ntm3bipSUFCGEEB988IFYuHBhsd8/Mh1uuVdDR44cQW5uLv78808AgFarhaenJyQSCTZt2oQjR45g\n7969SEhIgBACBQUFAAq3tF9//XWMGzcOfn5+AIDmzZujfv36OHLkCHx8fJCamoqgoKBil/twWCY/\nPx/z58+HVCpF+/btAQCXLl1CWloaQkNDjc+XSCSIj4+Ho6NjiZ9FIpEUe7/BYICNjc0T9y9YsAAn\nT57E119/jVu3biE1NRX5+fm4evUqpFIpunfvDgBo1aoV9uzZU+JyFQoF+vXrh4iICEyZMgURERH4\n6aef4OTkhE2bNuHo0aO4desW4uLikJ+fX+L7AMCxY8cwfPhw4+ecNGkSNm3aBI1GAwDo0aMHpFIp\nFAoFvL29kZ2d/dT3K02XLl0gl8sBAM2aNUN2dvZzf/7IyEg0bNgQrVu3BgA0bdoUgYGB+OuvvyCR\nSNCyZUvUrl0bANCiRQv8/vvv5cpMz4/lXg0ZDAYsWbIE3bp1AwDk5eVBrVYjPz8fw4YNQ+/evdGu\nXTuMGDECBw8ehPh7+iEbGxt89dVXmDVrFgYMGICAgAAAwIQJExAWFoZGjRph9OjRJRbuQ46Ojli1\nahUGDhyILVu24NVXX4Ver0fjxo2xfft24/Pu378PDw8PXLhwocT3cnd3x40bN564PyMjA25ubk/c\n/+abb0Kv12PAgAHo3r07UlJSIISAjY3NE7mvXr0KX1/fEpc9atQovPvuu2jcuDGaNGmCBg0a4N69\nexgzZgxGjx6Ntm3bon///jh8+PBT14d4bHong8EAnU5nvG1vb2/8WiKRPPH8xz3+HK1WW+Tx4t7v\neT+/wWAo9nPodDrY2to+d2aqeDxaphoKCgrCjz/+CI1GA4PBgHfffRdr165FYmIilEol5s2bh549\ne+Kvv/4yPgcAvLy8EBgYiEWLFmHBggXGLfp+/fohNjYWBw4cwIgRI54pg6urKxYtWoQNGzbg/v37\naNOmDRITE3H27FkAhUd/9OvXD6mpqU99n5dffhmnTp1CQkKC8b5du3ZBoVAUW0wnTpxAaGgoBg4c\nCIlEgsuXL0Ov18PX1xcSiQQnT54EAMTExGDy5MnGvwAeLduH2rRpAwDYsGEDRo0aBQCIjo6Gh4cH\nZs2ahS5duhiLXa/XQyaTQa/XP1F0QUFBCA8PN27hb926Fe3btzduXT8vDw8PXLt2DWq1GjqdrtRf\nLgCe+/O3bt0aN2/eRGRkJADg2rVrOHv2LDp06FCmzFTxuOVuxfLz8584HHLbtm2YNWsWVq5ciWHD\nhkGv16N58+ZYvHgxHB0d0b17dwwYMAAuLi5o2LAhmjRpgsTExCJFM2zYMPz2229YsWIF3n//fcjl\ncvTr1w/p6enw8PB45nyvvPIKtm/fjhUrVuA///kPvvjiC6xatQpqtRpCCKxatQr16tVDUlISEhIS\nnvgsx44dg4+PDz788EMsXLgQOp0OGo0G9evXx9dff11kp+ND8+fPR2hoKFxdXeHg4ID27dvj9u3b\nkMvlWLduHZYvX45Vq1bB1tYW69atg1wuR48ePbBy5contoCBwq33jRs3onfv3gCAzp07Y8eOHejf\nvz8cHBwQEBAADw8PJCYmwtvbGy1atMCAAQPw888/G99j5MiRSElJwahRo2AwGODt7Y1PP/30mdfj\n4zp37oz27dtjwIAB8PLywksvvYT4+PinvuZ5P7+Hhwc+//xzfPjhh1CpVJBIJPjkk0/g4+ODixcv\nljk7VRyJ4N9LVE75+fmYMGECli1bZhyDJSLL4rAMlcvx48fRvXt3dOzYkcVOVIlwy52IyApxy52I\nyAqx3ImIrFClOVomLS23zK91d3dEVtbTTxSxJOYrH+YrH+Yrn8qez8vLudj7rWLLXSZ78kzEyoT5\nyof5yof5yqey5yuJVZQ7EREVxXInIrJCLHciIivEciciskIsdyIiK8RyJyKyQix3IiIrxHInIrKQ\n+5n52H7kOjRafYW/N8udiMgClAVarP3fJew7fRvp2aoKf3+WOxGRmekNBmzaHY20ByoMfrkR6tZw\nqvBlsNyJiMxs++EEXLmVhTZNaiC4i49JlsFyJyIyo5NRKThw9g7qeDpi+pAWkJZyQfmyYrkTEZnJ\nzZQc/LA/Hg52MswZEQAHO9NNzMtyJyIyg2ylGuvDo6A3GPD60Jao7eFo0uWx3ImITEyrM2DDzmhk\n5aoxsltj+Pt6mnyZLHciIhMSQuDH3+NxPTkbL7Wohf4vNTTLclnuREQmdPhiMo5dTkHDWgpMGeAH\niYl2oD6O5U5EZCLxt7Pw88FrcHa0xZzhAbCzNd9VnVjuREQmkJ5dgA07owEAs4JbwdPV3qzLZ7kT\nEVUwtVaP9WFRUBZoMb53U7zQ0N3sGVjuREQVSAiBLf8Xi9upSnRrUxfdX6xnkRwsdyKiCrTvzG38\nFZuKJvVdMaFPM7PtQH0cy52IqIJEJqQj7EgC3J3tEDrMHzIby1Usy52IqAKkZORhc8QVyGRSzB7u\nD1cnuUXzsNyJiMopX6XDurAoFKh1mNLfDz51XCwdCSabtUar1WLx4sVITk6GVCrFhx9+iMaNG5tq\ncUREFmEQAl/vicG9zHz069AAnVrVtnQkACbccj969Ch0Oh22bduG0NBQfPbZZ6ZaFBGRxew6fgOX\nEzLQspE7RnavPBuwJit3Hx8f6PV6GAwGKJVKyGSmm9qSiMgSzsalYu+fiajp5oDXhraCjbTyjHRL\nhBDCFG+ckpKCWbNmIT8/H1lZWdi0aRMCAwNLfL5Op4dMZr5Tc4mIyuPm3WwsWHccUgmwem5XeNe2\n/Dj7o0xW7p988gnkcjneeustpKSkYPLkydizZw/s7OyKfX5aWm6Zl+Xl5Vyu15sa85UP85UP85VP\ncfly8zX44PtzyMhRYfZwfwQ287JQusJ8xTHZWImLiwtsbW0BAK6urtDpdNDr9aZaHBGRWej0Bny5\nKxoZOSoMDfKxaLE/jcnKfcqUKViyZAnGjx8PrVaL+fPnw9HRtFceISIytV8OXUfc7QcIbOaFIZ0b\nWTpOiUxW7k5OTvj8889N9fZERGZ3/PJd/HE+CfVqOGHaoOYmu7h1Rag8u3aJiCqxhORsbD0QDyd7\nGeaM8Dfpxa0rAsudiKgUWblqrN8ZBb1B4PXgVqjpXvmHmFnuRERPodHqsT48CtlKDcb0aIKWjTws\nHemZsNyJiEoghMCGHZdxMyUHnVrWRp/2DSwd6Zmx3ImISnDwXBIOnbsDnzrOmNz/BYvNzV4WLHci\nomJcuZWJXw5dh9vfc7PLzXhx64pQuXf3EhFZQOqDAny5KxoSCbBkcgd4KGwtHem5ccudiOgRKo0O\n68MikafSIaTfC2juUzV2oD6O5U5E9DchBL79NRZJaXnoGVgPXVvXtXSkMmO5ExH9be+ft3A+Pg0v\nNHDD2F5NLR2nXFjuREQALl5Lw87jN+HpYoeZw1pZ9OLWFaFqpyciqgDJ6Xn4es8VyGVSzB4eABdH\ny17cuiKw3ImoWstTabEuLBIqjR5TBzWHd+3i50evaljuRFRtGQwCmyNikJpVgIEdvdGheS1LR6ow\nLHciqrbCjiYg+kYm/H09Mbyrr6XjVCiWOxFVS6dj7mHfmduo5eGI115pAam06kwt8CxY7kRU7dy6\nl4Mt++LgYGeDuSP84Whf9c5ALQ3LnYiqlZw8DdaHR0GnM2D6kJao4+lk6UgmwXInompDpzdg484o\nZOaoMayrL9o0qWHpSCbDcieiauOng9dwNSkb7fxqYlAnb0vHMSmWOxFVC0cuJuPIxWQ0qKnAtIHN\nq9Tc7GXBciciq3f1zgP8+PtVKBxsMWe4P+zkVWtu9rJguRORVcvMUWHjzigIAcwKboUabg6WjmQW\nLHcisloarR7rwqOQk6/F2F5N4OftbulIZsNyJyKrJITA9/vjkHgvF0EBddCrbX1LRzIrljsRWaXf\n/rqD0zH30biuC0L6Vq2LW1cEljsRWZ3oGxnYfuQ63BRyhA73h62s+lVd9fvERGTV7mflY9PuGNhI\nJQgd7g83hZ2lI1kEy52IrEaBWod1YVHIV+swub8fGtd1tXQki2G5E5FVMAiBb/Zewd30PPRuVx+d\n/etYOpJFsdyJyCpEnLiJi9fS0dzbHWN6NrF0HItjuRNRlXc+PhURJ2+hhqs9Zga3go2U1SYz5Ztv\n3rwZhw4dglarxbhx4zBq1ChTLo6IqqGkVCW+2RsLua0Uc0YEQOFgfXOzl4XJyv3MmTO4ePEifv75\nZxQUFOC7774z1aKIqJpSFmixLjwSaq0es4JboUFNhaUjVRomK/cTJ06gWbNmCA0NhVKpxMKFC021\nKCKqhvQGAzbtjkbaAxUGv9wI7fxqWjpSpSIRQghTvPHSpUtx9+5dbNq0CUlJSZg5cyb2799f4lli\nOp0eMpn1z9RGRBXjm93R2H0sAR1a1Mb/968OVncN1PIy2Za7m5sbfH19IZfL4evrCzs7O2RmZsLT\n07PY52dl5Zd5WV5ezkhLyy3z602N+cqH+crHGvOdjErB7mMJqOPpiMn9miEjQ2midFVj/RXHZLuU\n27Zti+PHj0MIgfv376OgoABubm6mWhwRVRM37ubgh/3xcLSTYe6IADjYmfS4kCrLZGulR48eOHv2\nLEaOHAkhBN577z3Y2HDYhYjKLlupxoadUdAbDHhtqD9qeThaOlKlVWq537lzB0eOHEFiYiIkEgm8\nvb3Ro0cP1KtXr9Q3505UIqooWp0BG3ZGIytXjVHdG8Pft/ghXipUYrmnpqZi+fLluHv3LgIDA9Gw\nYUPIZDIkJSVh3rx5qFevHhYvXozatWubMy8RVUNCCPz4ezyuJ2fjpRa10P+lhpaOVOmVWO5r1qzB\n7Nmz0aRJ8afxxsXFYc2aNVi9erXJwhERAcDhi8k4djkFDWspMGWAX7Wbm70sSiz3lStXAij8jfn4\nilQqlfDz82OxE5HJxd/Ows8Hr8HZ0RZzhgfAzpb77p5FqUfLjBs3DklJScbbR48exSuvvGLSUERE\nAJCeXYANO6MBAKHD/OHpam/hRFVHqTtUJ0yYgEmTJuHVV19FTEwMYmNj8dlnn5kjGxFVY2qtHuvD\noqAs0CKk3wto1oCHUj+PUst9yJAhkEqlePvtt+Hp6Ylt27ahfv3qdaFZIjIvIQS2/F8sbqcq0a1N\nXfR4sfSj86ioUst94cKFiImJwbZt23Djxg2EhIRg6tSpCAkJMUc+IqqG9p25jb9iU9Gkvism9Glm\n6ThVUqlj7h4eHggPD0fr1q0xbNgwbNu2DSdOnDBHNiKqhiIT0hF2JAHuznYIHeYPmQ3nZi+LEtda\nXFwcAGDx4sWws/vnArO1atXC5s2bizyHiKgipGTkYXPEFchkUswe7g9XJ7mlI1VZJZZ7REQEFi5c\niBMnTkClUhnvLygowLFjx/DGG29g9+7dZglJRNYvX1V4cesCtQ5T+vvBp46LpSNVaSWOuS9cuBBx\ncXHYsmUL3nrrrcIny2QwGAzo2rUrZs6cCT8/P7MFJSLrZTAIfLUnBvcy89GvQwN0asUz38vrqTtU\n/fz8jCczZWZmQiqVcmZHIqpwO4/fQGRCBlr6eGBk98aWjmMVSiz3Xbt2FbktkUjg6uqKF198Ea6u\nriYPRkTVw4nLyfj1VCJqujng9aEteXHrClJiuZ85c6bIbSEEMjIysGTJEqxZswadOnUyeTgism63\n7+fis20XYSe3wZwR/nCy58WtK0qJ5f7JJ58Ue39CQgLeeecdljsRlUtuvgbrwqKg1ugxe7g/6nnx\n4tYV6bn//mncuHGRo2eIiJ6XTm/Al7uikZGjwvi+LyCwmZelI1md5y53vV4Pg8FgiixEVE38cug6\n4m4/QGAzL4zp84Kl41ilEodlzp49+8R9OTk52L17N7p06WLSUERkvY5fvos/ziehXg0nTBvUHFIp\n52Y3hRLL/YsvvihyWyqVwtXVFR07dsSYMWNMHoyIrE9Ccja2HoiHk70Mc0b48+LWJlTimt26das5\ncxCRlcvKVWP9zijoDQKvB7dCTXde3NqUynRAaWBgYEXnICIrptXpsT48CtlKDcb0aIKWjTwsHcnq\nlanchRAVnYOIrJQQAv/dH4+bKTno1LI2+rRvYOlI1UKZyp0XpyWiZ3XwXBJORt9Do9rOmNz/BfaH\nmZQ45n737t1i7xdCcMudiJ7JlVuZ+OXQdbg4yTF7uD/kvLi12ZRY7hMnToREIim2yN3d3U0aioiq\nvtQHBfhyVzQkEmD2MH94uPDi1uZUYrkfOnTInDmIyIqoNDqsD4tEnkqHyf1fQJP6nGzQ3Eodc8/O\nzsbSpUsxadIkZGVl4Z133kFOTo45shFRFSSEwLe/xiIpLQ89AuuhWxte3NoSSi33d999F/7+/njw\n4AGcnJxQs2ZNvP322+bIRkRV0N4/b+F8fBqaNXDDuF5NLR2n2iq13JOSkjBmzBhIpVLI5XLMnz8f\n9+7dM0c2IqpiLl5Lw87jN+HpYodZw1rx4tYWVOqat7GxQW5urvHwpVu3bkHKyfSJ6DHJ6Xn4es8V\nyGVSzB4eABdHXtzakkqd2GHOnDkICQlBSkoKZs2ahUuXLuHjjz82RzYiqiLyVFqsC4uESqPHa6+0\nhHdtZ0tHqvZKLfeuXbuiVatWiIyMhF6vxwcffAAPj2c7dTgjIwPDhw/Hd999h8aNeV1EImtkMAhs\n3h2D1KwCDOjYEC+1qGXpSISnDMuoVCps27YN+/btg4eHB7p3745evXohJiYGQ4YMKfWNtVot3nvv\nPdjb89hWImu242gCom9mwt/XEyO6ciOusihxy33RokW4e/cucnNzkZmZiZ49e2Lp0qW4dOkSpk+f\nXuobr1y5EmPHjsVXX31VoYGJqPI4HXMP+8/cRi0PR7z2SgvOzV6JlFjuUVFROHDgALKzszFjxgx8\n++23CAoKwurVq0sdlgkPD4eHhwe6dOnyzOXu7u4ImazspyZ7eVXuMT7mKx/mKx9T5Lt+5wG+3xcH\nBzsZ/v1qRzSoVfZlVMf1Z2ollruLiwtkMhk8PT1x7949/Pvf/0bfvn2f6U3DwsIgkUhw6tQpxMbG\nYtGiRfjyyy/h5VXydRKzsvKfP/3fvLyckZaWW+bXmxrzlQ/zlY8p8uXkafDhD2eh1RnwenAr2EtR\n5mVUx/VXkUr6xVNiuT86c5unp+czFzsA/Pjjj8avQ0JCsGzZsqcWOxFVHTq9ARt3RiEzR43hXX3R\npkkNS0eiYpRY7lqtFikpKTAYDDAYDEhJSSkyiVjdunXNEpCIKpefDl7D1aRstPOriUGdvC0dh0pQ\nYrnn5+dj4sSJxkKfMGGC8TGJRII//vjjmRbAy/URWY8jF5Nx5GIy6nspMG1gc87NXolxVkgieiZX\n7zzAj79fhcLBFnNH+MNOzrnZKzPOI0BEpcrMUWHjzigIAcwMboUabg6WjkSlYLkT0VNptHqsC4tC\nTr4WY3s1QXNvXqynKii13DUajTlyEFElJITA9/vjkHg/F0EBddCrbX1LR6JnVGq59+3bF++//z4i\nIyPNkYeIKpHf/rqD0zH30biuC0L68uLWVUmp5b5v3z60bt0aa9euxZAhQ/Dtt98iLS3NHNmIyIKi\nb2Rg+5HrcFXIMWuYP2xlHMWtSkr9bjk4OCA4OBjff/895s6di//+97/o06cPZs2ahcTERHNkJCIz\nu5+Vj027Y2AjlWD2cH+4O9tZOhI9p1Kn/E1MTERERAT27t2LunXr4u2330bfvn1x+vRpTJ8+HQcO\nHDBHTiIykwK1DuvCopCv1mHqwOZoXJcXt66KSi33f/3rX8Y52evV++dCt926dcPJkydNGo6IzMsg\nBL7ZewV30/PQu119BAXUsXQkKqNSy/27775Do0aNity3detWhISEYMmSJabKRUQWEHHiJi5eS0dz\nb3eM6dnE0nGoHEodc3/99deNY+vx8fEYNWoUDh48aPJgRGRe5+NTEXHyFmq42mNmcCvY8FrJVVqp\nW+7Lly/HzJkz8fLLL+PAgQN48803ERwcbI5sRGQmSalKfLM3FnJbKeaMCIDCwdbSkaicSv3VHBgY\niLVr12L//v1YvXo1i53IyigLtFgXHgm1Vo9XB7VAg5oKS0eiClDilrufn5/xhIWHM0NOnjwZQOGs\nkLGxsWaIR0SmpDcYsGl3NNIeqDD45UZo51fT0pGogpRY7nFxcebMQUQWsP1wAq7cykKbJjUQ3MXH\n0nGoApU65p6RkYE9e/YgLy8PQggYDAYkJSVh1apV5shHRCZyMioFB87eQR1PR0wf0gJSTi1gVUod\nc589ezZiY2MRERGBgoICHDp0CFLuRSeq0m7czcEP++PhaCfD3BEBcLArdTuPqphSWzorKwsrV65E\nz5490bdvX2zduhXXrl0zRzYiMoEHSjXWh0dCbzDgtaEtUcvD0dKRyARKLXdX18JTj318fBAXFwdn\nZ2fodDqTByOiiqfVGbBhZxQeKDUY2b0x/H09LR2JTKTUv8U6duyIuXPnYtGiRZg6dSpiYmJgZ8dJ\nhIiqGiEE/t+BeCQk5+ClFrXQv0NDS0ciEyq13OfPn4/bt2+jXr16WLt2Lc6ePYvQ0FBzZCOiCnTo\nQjKOR6bAu5Yzpgzw49zsVu6Z9oxevnwZ//nPf+Dr6ws3NzfUqlXL1LmIqALFJWZh2x/X4Oxoi9nD\n/WFny4tbW7tSy/3TTz/F0aNHceDAAej1eoSFhWHFihXmyEZEFSA1Mx8bd0UDAEKH+cPT1d7Cicgc\nSi33EydOYPXq1bCzs4NCocCWLVtw7Ngxc2QjonJSa/T4eMtfUBZoMb5PMzRr4GbpSGQmpZb7w2Pa\nH47PaTQaHudOVAUIIbBlXyxu3M1G9zZ10ePFeqW/iKxGqTtU+/fvj3nz5iE7Oxvff/89IiIiMGjQ\nIHNkI6Jy+L/TifgrNhUtfDwwvk8zS8chMyu13GfMmIHjx4+jbt26SElJwZw5c9CjRw9zZCOiMrp8\nPR3hR2/A3dkOiye3h06ltXQkMrNnOue4S5cu6NKli/H2smXLsGzZMlNlIqJySMnIw1d7YiCTSTFn\nhD/cne2RxnKvdso0eB4REVHROYioAuSrCi9uXaDWY8oAPzSq7WLpSGQhZSr3h/O7E1HlYTAIfLUn\nBvcy89G/Q0N0alnb0pHIgspU7jyzjajy2Xn8BiITMtDSxwMjuze2dByysBLH3ENCQootcSEE1Gq1\nSUMR0fP5K/Y+fj2ViJruDnh9aEtIpdwAq+5KLPc5c+aU+U21Wi2WLFmC5ORkaDQazJw5E7169Srz\n+xFRyW7fz8V3v8bCTm6DOcP94WTPi1vTU8q9Q4cOZX7TiIgIuLm5YfXq1Xjw4AGCg4NZ7kQmkJOv\nwbqwKGjGRJKvAAARnklEQVR0BswZ7o96Xry4NRUyyeVX+vfvj379+gEoHMaxseEkRUQVTac3YNOu\naGTkqBAc5IMXm3lZOhJVIhJhwkNflEolZs6cidGjR2PIkCFPfa5Op4dMxl8CRM9qc3gk9p68iU7+\ndbB4UnuOs1MRJrtwYkpKCkJDQzF+/PhSix0AsrLyy7wsLy9npKXllvn1psZ85cN8Tzp2+S72nryJ\nel5OCOnTFBkZyhKfy/VXPlUhX3FMUu7p6emYOnUq3nvvPXTq1MkUiyCqtq4nZ2Prb/FwspdhzogA\n2Mt5cWt6kkmmd9y0aRNycnKwceNGhISEICQkBCqVyhSLIqpWsnLV2BAeBYMQeD24FWq6OVg6ElVS\nJvmVv3TpUixdutQUb01UbWl1eqwPj0R2ngZjezVFy0Yelo5ElRgnZieqAoQQ+O/+eNxMyUXnVrXR\np119S0eiSo7lTlQF/H4uCSej78GnjjMm9X+BU4BQqVjuRJXclVuZ+N+h63B1kmP28ADY8pBhegYs\nd6JKLPVBAb7cFQ2JpPDi1u7OdpaORFUEy52oklJpdFgXFok8lQ4h/V5Ak/qulo5EVQjLnagSMgiB\nb3+NRXJaHnoG1kPX1nUtHYmqGJY7USW0989bOB+fhhcauGFsr6aWjkNVEMudqJK5eDUNu47fhKeL\nPWYOawWZDf+b0vPjTw1RJZKcnoev9l6B/O+LW7s4yi0diaooljtRJZGn0mJdWCTUGj2mDmqOhrWK\nnxCK6Fmw3IkqAYNBYNPuGKRmFWBQJ290aF7L0pGoimO5E1UCO44mIOZmJgIae2JYF19LxyErwHIn\nsrDTMfew/8xt1PJwxIwhvLg1VQyWO5EFJd7LxZZ9cXCws8HcEf5wtOfc7FQxWO5EFpKdp8EXYZHQ\n6QyYMaQl6ng6WToSWRGWO5EF6PQGbNwZhaxcNYZ380XrJjUsHYmsDMudyAJ+OngN15Ky0d6vJgZ2\n9LZ0HLJCLHciMztyMRlHLiajQU0Fpg5szrnZySRY7kRmdPXOA/z4+1UoHGwxZ7g/7OScm51Mg+VO\nZCaZOSps3BkFIYBZwa1Qgxe3JhNiuROZgVqrx7qwKOTkazGud1P4ebtbOhJZOZY7kYkJIfDDvjgk\n3s9Fl4A66BlYz9KRqBpguROZ2G9/3cHpK/fRuJ4LJvblxa3JPFjuRCYUfSMD249ch5tCjtBh/rCV\n8b8cmQd/0ohM5H5WPjbtjoGNVIrZwwPgpuDFrcl8WO5EJlCg1uGLHZHIV+swuf8L8K3rYulIVM2w\n3IkqmEEIfL3nClIy8tGnXQN09q9j6UhUDbHciSpYxImbuHQ9Hc293TG6Z2NLx6FqiuVOVIHOx6ci\n4uQt1HC1x8zgVrCR8r8YWQZ/8ogqSFKqEt/sjYWdrQ3mjgiAwsHW0pGoGmO5E1UAZYEWX4RFQq3V\nY9qg5qhfU2HpSFTNsdyJykmvN+DLXdFIz1ZhyMuN0M6vpqUjEcFk1/QyGAxYtmwZ4uPjIZfL8dFH\nH8Hbm/NWk/X5bm8MYhOz0KZJDQzt4mPpOEQATFjuBw8ehEajwS+//IJLly5hxYoV+PLLL021OCKT\n02j1yMhRISNHhfRsFTKyVbiXmY/z8Wmo4+mI6UNaQMqpBaiSMFm5nz9/Hl26dAEAtGnTBtHR0U99\nvru7I2Syss9t7eXlXObXmgPzlY858qnUOqRm5SM1q6Dw38y/v87MR2pWPrJy1cW+ztPVHsumd0Jd\nr8o7zs7vb/lU9nzFMVm5K5VKKBT//LDb2NhAp9NBJit+kVlZ+WVelpeXM9LScsv8elNjvvKpqHwF\nah0ysv/e6s5RIT27wHg7PVsFZYG22NfZSCXwdLFHc293eLrao4arPTxdCv+t4eqApj6eyMzMq7Tr\nsLp8f02lKuQrjsnKXaFQIC8vz3jbYDCUWOxE5SWEQP4j5f1w2CQ9u6BwKCVbhTyVrtjXymyk8HS1\nh3ctBTxdHQrL2/Wf8nZ1kkMqLXm4xcaGxyVQ5WOytg0MDMThw4cxcOBAXLp0Cc2aNTPVoqgaEEJA\nWaB9pLQL/zVugeeoUKDWF/tauaywvH3quqDGw/L+e8vb09UeLk5yjpWT1TFZuffp0wcnT57E2LFj\nIYTA8uXLTbUosgJCCOTkaZD+91b2o8MnWUoN7mfmQaM1FPtaO7lN4Va2y8Mtbod/hk9c7eHsYMs5\n1KnaMVm5S6VSfPDBB6Z6e6piDEIgW6kxDpX8M+79zxa4Vld8eTs52KKWu+M/wyUu9kWGT5zsZSxv\nosdwEJwqhN5gwINcjXGI5PHhk8xcFXR6UexrFQ62qFvDybjl/eh4t6eLPbwbuFfqHVpElRHLnZ6J\nTm9AVq76752VBcahk4dFnpWrht5QfHm7ONqiQU3nIjsqHx3ztpfzx5CoovF/FQEAtDoDMnNUxjHv\nwi3uf4ZPsnLVEMV3N1wVcjSq41zszkoPF3vY2Zb9/AUiKhuWezXx8OzK9Md2VqZnFyArV43MnOJP\n0JFIAHdnOzSt5/r3kEnRQwU9nO15XVCiSojlbiVUmseO8X50Z2V2AXLyiz9BRyqRoIa7A/wausGz\nmKNN3J3tIONx3ERVDsu9ishX6Z7YWfnoFvhTz650tUc9L8UTJ+d4utjDzVmO2rVcucOSyMqw3CsB\nIQTyVLp/zqh8bOs7PVuFAnXxZ1fayqTwcLFHo9rOj+2sLNz6dlXwBB2i6ojlbgYPT9ApOuZd9Fhv\ntab4syvtbG3g6WqPpvVdHznG+5/hExdHnqBDRE9iuVcAw8OzK4vMZ6L+55DBHDU02uLL215uA69H\nxrmNE1K5FX6t4NmVRFQGLPdnYDAIPFCqH9tR+ejwiRo6fQlnV9rLUL+mAq6Otv8cKvjIqfGOdjy7\nkogqHssdhWdXZuWoSzxUMDOn5BN0nB1tUd/L6ZHSdigyfOJgJ6v0U4YSkfWpFuWu0xeeoFPSoYJZ\nuWoYSjhDx9VJbtxZ+ei8Jg+/tpPzBB0iqnysotw1Wj3uZeYXe6RJRrYKD3LVKK66JQDcnO3gW9fl\nieGSwkMF7WBbjqtDERFZSpUv9/93IB6HLiQX+5hEAng426NpA7fHrp7zz6nxPEGHiKxRlS/3mu6O\nCGhSAy4Otk9MTOXGsyuJqJqq8uXet30DTBjYgjssiYgewc1aIiIrxHInIrJCLHciIivEciciskIs\ndyIiK8RyJyKyQix3IiIrxHInIrJCEiFKuqY9ERFVVdxyJyKyQix3IiIrxHInIrJCLHciIivEcici\nskIsdyIiK8RyJyKyQlXuYh2///479u/fjzVr1gAALl26hI8//hg2NjYICgrC7NmzizxfpVJhwYIF\nyMjIgJOTE1auXAkPDw+TZvzqq69w/PhxAEBOTg7S09Nx8uTJIs/56KOPcOHCBTg5OQEANm7cCGdn\nZ5PmekgIga5du6JRo0YAgDZt2uCtt94q8pz//e9/2LZtG2QyGWbOnIkePXqYJRsA5ObmYsGCBVAq\nldBqtVi8eDFefPHFIs+xxPozGAxYtmwZ4uPjIZfL8dFHH8Hb29v4+KFDh7BhwwbIZDKMGDECo0eP\nNmmex2m1WixZsgTJycnQaDSYOXMmevXqZXz8+++/x/bt240//++//z58fX3NmnHYsGFQKBQAgPr1\n6+OTTz4xPmbp9RceHo6dO3cCANRqNWJjY3Hy5Em4uLgAqBzr77mIKuTDDz8U/fr1E/PmzTPe98or\nr4jExERhMBjEq6++KmJiYoq85rvvvhNffPGFEEKIvXv3ig8//NCsmWfMmCGOHz/+xP1jx44VGRkZ\nZs3y0K1bt8Rrr71W4uOpqali8ODBQq1Wi5ycHOPX5vL555+LLVu2CCGESEhIEMHBwU88xxLr77ff\nfhOLFi0SQghx8eJF8frrrxsf02g0onfv3uLBgwdCrVaL4cOHi7S0NLPm27Fjh/joo4+EEEJkZWWJ\nbt26FXn8rbfeElFRUWbN9CiVSiWGDh1a7GOVYf09atmyZWLbtm1F7rP0+nteVWpYJjAwEMuWLTPe\nViqV0Gg0aNiwISQSCYKCgvDnn38Wec358+fRpUsXAEDXrl1x6tQps+U9cOAAXFxcEBQUVOR+g8GA\nxMREvPfeexg7dix27NhhtkwAEBMTg/v37yMkJATTp0/HjRs3ijweGRmJF198EXK5HM7OzmjYsCHi\n4uLMlm/KlCkYO3YsAECv18POzq7I45Zaf4/+LLVp0wbR0dHGxxISEtCwYUO4urpCLpejbdu2OHv2\nrFlyPdS/f3+88cYbAAr/OrOxsSnyeExMDL766iuMGzcOmzdvNms2AIiLi0NBQQGmTp2KSZMm4dKl\nS8bHKsP6eygqKgrXr1/HmDFjitxv6fX3vCrlsMz27dvxww8/FLlv+fLlGDhwIM6cOWO8T6lUGv/E\nAwAnJyfcuXOnyOuUSqXxz3UnJyfk5lbstVZLyhoQEIDNmzdj7dq1T7wmPz8fEydOxL/+9S/o9XpM\nmjQJrVq1gp+fX4VmKynfe++9hxkzZmDAgAE4d+4cFixYgLCwMOPjj64zoHC9KZXKCs9WUr6H6y8t\nLQ0LFizAkiVLijxuzvX3qMd/3mxsbKDT6SCTycy6zkrycIhKqVRi7ty5mDdvXpHHBw0ahPHjx0Oh\nUGD27Nk4fPiwWYfb7O3tMW3aNIwaNQq3bt3C9OnTsX///kqz/h7avHkzQkNDn7jf0uvveVXKch81\nahRGjRpV6vMUCgXy8vKMt/Py8ozjY8U9p7jHTZX1+vXrcHFxKTIm+5CDgwMmTZoEBwcHAEDHjh0R\nFxdnknIqLl9BQYFxq65du3ZITU2FEAISiQRA8evVVOPZJa2/+Ph4vPnmm1i4cCE6dOhQ5DFzrr9H\nPb5eDAYDZDJZsY+Zcp09TUpKCkJDQzF+/HgMGTLEeL8QApMnTzZm6tatG65cuWLWcvLx8YG3tzck\nEgl8fHzg5uaGtLQ01KlTp9Ksv5ycHNy8eRMdO3Yscn9lWH/Pq0oNyzxOoVDA1tYWt2/fhhACJ06c\nQLt27Yo8JzAwEEePHgUAHDt2DG3btjVLtj///BNdu3Yt9rFbt25h3Lhx0Ov10Gq1uHDhAlq2bGmW\nXACwfv1649ZyXFwc6tSpYyx2AAgICMD58+ehVquRm5uLhIQENGvWzGz5rl+/jjfeeANr1qxBt27d\nnnjcUusvMDAQx44dA1C4I//RddK4cWMkJibiwYMH0Gg0OHfu3BM7gU0tPT0dU6dOxYIFCzBy5Mgi\njymVSgwePBh5eXkQQuDMmTNo1aqVWfPt2LEDK1asAADcv38fSqUSXl5eACrH+gOAs2fPolOnTk/c\nXxnW3/OqlFvuz+P999/H22+/Db1ej6CgILRu3RoAMHXqVGzatAnjxo3DokWLMG7cONja2hqPsjG1\nmzdvonPnzkXu27JlCxo2bIhevXph6NChGD16NGxtbTF06FA0bdrULLkAYMaMGViwYAGOHj0KGxsb\n4xELj+YLCQnB+PHjIYTA/Pnznxj3NqU1a9ZAo9Hg448/BlD4S/zLL7+0+Prr06cPTp48ibFjx0II\ngeXLl2PPnj3Iz8/HmDFjsHjxYkybNg1CCIwYMQK1atUyeaZHbdq0CTk5Odi4cSM2btwIoPAvo4KC\nAowZMwbz58/HpEmTIJfL0alTp2J/cZrSyJEj8c4772DcuHGQSCRYvnw59u3bV2nWH1D4/7Z+/frG\n249+fy29/p4Xp/wlIrJCVXpYhoiIisdyJyKyQix3IiIrxHInIrJCLHciIivEciciskIsdyIiK8Ry\nJyrBf//7X0yYMAFCCJw7dw59+/a12HwnRM+LJzERlUAIgUmTJqF///7YunUrPv74Y7NNX0FUXix3\noqe4c+cOhgwZYpzGgqiq4LAM0VPcvXsXCoUCV65cAbeDqCphuROVIC8vD++++y42btwIBwcH/PTT\nT5aORPTMWO5EJVi9ejW6deuGgIAAY8k/fjEYosqKY+5ERFaIW+5ERFaI5U5EZIVY7kREVojlTkRk\nhVjuRERWiOVORGSFWO5ERFbo/wc84cSiTNpL+AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xc007978>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Graphical representation of our Leaky ReLU activation function\n",
    "x = array(range(-10, 10))\n",
    "plt.plot(x, LeakyReLU(x))\n",
    "plt.title(\"Leaky ReLU activation function\")\n",
    "plt.xlabel(\"x\")\n",
    "plt.ylabel(\"Leaky ReLU(x)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Why do we use activation functions ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are probably asking yourself after reading the whole Activation Function section about the useflness of the latter, why even bother using it ? why don't we just use the initial dot product output without feeding it to the activation function ?\n",
    "\n",
    "Well, without an activation function we will fail to introduce non-linearity into the network.\n",
    "\n",
    "An activation function will allow us to model a response variable (target variable, class label, or score) that varies non-linearly with its explanatory variables.\n",
    "\n",
    "non-linear means that the output cannot be reproduced from a linear combination of the inputs.\n",
    "\n",
    "another way to think of it: without a non-linear activation function in the network, an artificial neural network, no matter how many layers it has, will behave just like a single-layer perceptron, because summing these layers would give you just another linear function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Which activation function is the \"best\" to use ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many activation functions out there but which one is the best to use? This question has been asked by many beginners in the deep learning field (including myself).\n",
    "\n",
    "The short answer for this question is: There is no \"best\" activation function.\n",
    "\n",
    "The long answer is: The use of an activation function depends on the task you are dealing with, is the task a regression or a classification problem? if it's a classification problem then is it a binary classification or a multiclass classification task? it all comes down to the task you are dealing with.\n",
    "\n",
    "for example we can't possibly use the Sigmoid or tan-h activation functions for a multiclass classification task we would better be using the Softmax activation function instead. right? as it would output a vector of probabilities of each class. \n",
    "another example, which activation would be better to use in the hidden layer part of a multi-layered neural net? if you use the sigmoid activation function then you would be hit by the vanishing gradient problem (the vanishing gradient problem is when the gradient becomes so small in the earlier layers of a deep neural network to the point that it barely affects the weights of the earlier layers thus failing at optimizing our initial weights.), should we use the tan-h activation function then? not necessarily, it's true that the tan-h function is an enhanced version of the sigmoid function but the vanishing gradient problem is still persistently there. So, the solution to our persistent problem is the the ReLU activation function, but how does it solve the vanishing gradient problem? the fact that the derivative of the ReLU activation function is 1 if its inputs are positive will assure that the gradient will not get infinitesimally smaller passing from one layer to another but what if our inputs are negative? here we get stuck with what we call the \"dying ReLU\" problem (the dying ReLU problem is when the ReLU activation function always outputs the same value (zero) for any input. That means that it takes no role in discriminating between inputs. Once a ReLU ends up in this state, it is unlikely to recover, because the function gradient at 0 is also 0, so gradient descent learning will not alter the weights.), here comes the Leaky/Parametric ReLU to rescue and instead of outputting a flat out zero for the negative values the Leaky ReLU multiplies the negative values by an alpha parametric value ($\\alpha = 0.01$).\n",
    "\n",
    "As you can see from the previous explanation, the use of a certain activation function depends on the task you are dealing with and the problems you face while building your neural network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Forward Propagation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Definition:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Forward propagation** is the process of feeding the Neural Network with a set of inputs to get their dot product with their weights then feeding the latter to an activation function and comparing its numerical value to the actual output called \"the ground truth\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculations:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's demonstrate how the output gets calculated in a forward pass in a 3-4-1 Artificial Neural Network:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](http://neuralnetworksanddeeplearning.com/images/tikz10.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "****The initial weights of the first and second layer:****"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$W_{ij} =                      \n",
    "\\begin{bmatrix}\n",
    "    0.1 & 0.2 & 0.4 & 0.3 \\\\\n",
    "    0.5 & 0.4 & 0.7 & 0.9 \\\\\n",
    "    0.2 & 0.6 & 0.3 & 0.8 \\\\\n",
    "\\end{bmatrix}$\n",
    "$and$\n",
    "$W_{jk} = \n",
    "\\begin{bmatrix}\n",
    "    0.2 \\\\\n",
    "    0.3 \\\\\n",
    "    0.6 \\\\\n",
    "    0.1 \\\\\n",
    "\\end{bmatrix}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "****Input:****"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$i = \n",
    "\\begin{bmatrix}\n",
    "    0.1 \\\\\n",
    "    0.4 \\\\\n",
    "    0.5 \\\\\n",
    "\\end{bmatrix}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "****output:****"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$o = \n",
    "\\begin{bmatrix}\n",
    "    1 \\\\\n",
    "\\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The activation function of the hidden layer will be the ReLU function and the activation function of the output layer will be the Sigmoid function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "let's begin our calculculations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# import our dependencies\n",
    "from numpy import dot\n",
    "\n",
    "i = array([[0.1, 0.4, 0.5]])\n",
    "\n",
    "w_ij = array([[0.1, 0.2, 0.4, 0.3],\n",
    "              [0.5, 0.4, 0.7, 0.9],\n",
    "              [0.2, 0.6, 0.3, 0.8]])\n",
    "\n",
    "w_jk = array([[0.2],\n",
    "              [0.3],\n",
    "              [0.6],\n",
    "              [0.1]])\n",
    "\n",
    "z_ij = dot(i, w_ij) # calculating the dot product of the inputs and weights of the hidden layer\n",
    "a_ij = ReLU(z_ij) # applying the ReLU activation function\n",
    "\n",
    "z_jk = dot(a_ij, w_jk) # the dot product of the inputs in the hidden layer after activation and weights of the output layer\n",
    "a_jk = Sigmoid(z_jk) # applying the Sigmoid activation function "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.63807065]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The final output of the feed forward neural network\n",
    "a_jk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Cross entropy error:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[**Cross-entropy loss**](http://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html), or log loss, measures the performance of a classification model whose output is a probability value between 0 and 1. Cross-entropy loss increases as the predicted probability diverges from the actual label. So predicting a probability of .012 when the actual observation label is 1 would be bad and result in a high loss value. A perfect model would have a log loss of 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Equation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$L = - \\sum_{i}(y\\prime_{i} log(y_{i}) + (1 - y\\prime_{i}) log(1 - y_{i}))$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Where $y_{i}$ is the predicted probability value for class $i$ and $y\\prime_{i}$ is the true probability for that class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Programmatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from numpy import log10\n",
    "\n",
    "# cross entropy error function\n",
    "def crossentropyerror(a, y):\n",
    "    return - sum(y * log10(a) + (1 - y) * log10(1 - a))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a = a_jk # the output of the neural network\n",
    "y = array([[1]]) # the ground truth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.19513122954797735"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "crossentropyerror(a, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Conclusion:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is in a nutshell how a forward propagation works and how a neural network generates its predictions, the forward propagation step has always been less mathematically intense than the backpropagation step where most students and beginners find legitimately harder to grasp."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Back-propagation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Definition:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "[**Backpropagation**](https://en.wikipedia.org/wiki/Backpropagation) is a method used in artificial neural networks to calculate a gradient that is needed in the calculation of the weights to be used in the network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculations:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "first let's lay some important derivatives:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Cross entropy error derivative:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\frac{\\partial L}{\\partial y_{i}} = \\frac{-y\\prime_{i}}{y_{i}} + \\frac{(1 - y\\prime_{i})}{(1 - y_{i})}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sigmoid derivative:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "$$\\frac{\\partial Sigmoid(x)}{\\partial x} = Sigmoid(x) \\times (1 - Sigmoid(x))$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Graphical Representation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://i.stack.imgur.com/QdlcW.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Programmatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Derivative of the Sigmoid activation function\n",
    "def SigmoidDerivative(x):\n",
    "    return Sigmoid(x) * (1 - Sigmoid(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0xbf370f0>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAETCAYAAADZHBoWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlclWX+//HXWdgPIMoioCCguCsuaaZmaZaVS+aGlVY6\nWv1stWnSahhHDW1aHjMtOjbfpiab72hpi8ukZVom+lVEyRVUBFNBFmU7bIdzzv37AzmJAgeBswCf\n50Mfcu7lnLc3cD7nvq77um6VoigKQgghBKB2dAAhhBDOQ4qCEEIICykKQgghLKQoCCGEsJCiIIQQ\nwkKKghBCCAspCsJhLly4QM+ePZk0aRKTJk1iwoQJPPjgg3z99deNer558+Zx5syZRu175MgR4uLi\nADh69CjPPvtso57nZuzZs4c777yTKVOmUF5eXmNdWloa8+fPZ8KECUyYMIFHHnmEgwcPApCdnU1s\nbKzN89V3HJ544gm+/PJLm2cQDqAI4SDnz59XYmJiaiy7cOGCctdddynbtm2za5aNGzcq8+fPt+tr\nLlq0SPnggw9qXXffffcp3333neXxgQMHlIEDByr5+fn2ilev+fPnKxs3bnR0DGEDWkcXJSGuFRoa\nyrPPPstHH33EPffcg8Fg4K233iIxMRGTyUSvXr147bXX0Ol0jB49mn79+pGamsrChQtZsWIFf/vb\n3/jkk0/o1asXc+fOBeA///kP+/fv55133iE+Pp5ffvmFkpISFEVh+fLlhISE8O6771JcXMzixYt5\n4IEHWLZsGf/5z38YNWoU27dvJyAgAIDp06ezYMEChg0bVmeua1VWVrJy5Ur27duHRqOhX79+LF68\nmHXr1vHDDz/g5uZGcXExL7/8co39cnNzKS0ttTy+5ZZb+Otf/4pGo+HChQtMmDCBw4cPU1ZWxp/+\n9Cd++eUXvL296dq1KwArV65k9OjRjB8/nh9//JGCggKeeeYZDh06xPHjx9FqtaxevZqgoCBOnz7N\n0qVLKSgoQKVSMWfOHB544AH279/PsmXL2LJlC9nZ2SxatIicnBxCQkK4fPmyLX8MhANJ85FwOj16\n9ODUqVMAfPjhh2g0Gr788ks2bdpEYGAgb731lmXbbt268e233zJ27FjLsmnTptVogvryyy+ZPn06\nv/zyCzk5Oaxfv57//ve/TJ48mX/84x8EBwfz7LPPMnjwYFasWGHZz9vbm7Fjx7Jp0yagqkknNzeX\nkSNHWs1VbfXq1eTk5PDNN9/wzTffYDab+ctf/sLvfvc7Ro8ezWOPPXZDQQCIi4tj+fLljBgxguee\ne47PPvuMvn374u3tXWO7VatWYTKZ+Pbbb/nkk084ceJEjfUVFRVs2rSJRYsWERcXx6OPPsqmTZsI\nDg7mq6++wmg08tRTTzFr1iw2b97MP/7xD9555x0OHz5c43mWLl1K//792bp1K6+99hrp6elWv4+i\nZZIzBeF0VCoV7u7uAPz4448UFxezd+9eoOqTd4cOHSzbDh48+Ib9hw4dSkVFBUePHsXDw4MrV64w\nbNgwVCoVvr6+rFu3jvPnz7N//368vLzqzTJt2jT+/Oc/M3fuXDZu3MiDDz6IWq22mqva7t27eeGF\nF3BxcQFg1qxZLFiwwOoxGD9+PGPHjiUpKYnExEQ2btzI6tWrWb9+fY3tfvrpJxYvXoxarUan0zF5\n8mRSU1Mt6++++24AOnfujL+/Pz169AAgLCyMwsJCMjIyqKiosGwXFBTE3Xffzc8//8zQoUMtz7N3\n715L8QoPD6+xTrQuUhSE0zl69CjR0dEAmM1mXnnlFUaNGgVASUkJFRUVlm09PT1v2F+lUjF16lS+\n+eYbXFxcmDp1KiqVih9//JHXX3+dxx9/nDFjxhAZGWk5C6jL4MGDMRqNHDlyhC1btrBu3boG5apm\nNptveFxZWVnva6alpfHVV1/x+9//nttuu43bbruN5557jscff5zt27dzzz33WLbVarUo10xfplbX\nPPl3dXW1fF1dmOrLB6AoCkajscYylUpV43W0WnnraK2k+Ug4lfT0dFatWsWcOXMAGDFiBP/+978x\nGAyYzWb++Mc/8s4771h9nsmTJ7Nz5062b9/Ogw8+CEBCQgJ33nknDz30EH379mXHjh2YTCYANBrN\nDW+E1aZNm8ayZcvo3r07ISEhN5Vr5MiRrFu3jsrKSsxmM//+978ZPnx4vdn9/f35/PPP2bZtm2VZ\nQUEBeXl59OrVq8a2o0aNYuPGjZjNZsrKytiyZQsqlcrq8akWERGBi4sL3333HVB1ZdP27du57bbb\nbvh/VJ+lZGZmsn///ga/hmhZpCgIhyovL7dckjp58mQWL17MwoULueOOOwD4f//v/xEaGsrkyZO5\n7777UBSFRYsWWX3egIAAevXqRXR0NEFBQQDExsaSmJjIhAkTmDFjBp07d+bChQuYzWYGDBjA2bNn\na23aeeCBBzh58iTTpk2zLGtorqeeegp/f38eeOAB7r33XoxGI6+++mq92X19ffnXv/7Fhg0bGD16\nNPfffz+PP/44c+fOZdiwYTW2feKJJ3Bzc2PChAk8/vjjdOjQwdL01hAuLi6sWrWKTz/91PIcCxYs\n4NZbb62x3Z/+9CfS0tK49957efXVVy3NUKL1USmKTJ0tREu1detWdDodo0aNwmw288wzzzB8+HAe\neughR0cTLZQUBSFasFOnThEXF0dZWRmVlZUMHTqUV155pdb+AyEaQoqCEEIIC+lTEEIIYSFFQQgh\nhEWLv9g4N7e40fv6+XmSn19qfUMHkXxNI/maRvI1jbPnCwjwrnV5mz5T0Go1jo5QL8nXNJKvaSRf\n0zh7vrq06aIghBCiJikKQgghLKQoCCGEsJCiIIQQwkKKghBCCAspCkIIISxsNk7BbDazZMkSUlNT\ncXV1Zfny5YSHh1vWb9myhX/9619oNBqio6NZsmQJarWayZMnW25p2KlTpxp3whJCCGFbNisKO3bs\nwGAwsH79epKTk1m5ciWrV68GqqZL/utf/8rmzZvx8PBg4cKF7Nq1ixEjRqAoCmvXrrVVLCGcwq/Z\nxew4nMmd/TuiUcsJu3AeNisKSUlJjBw5EoCYmBiOHTtmWefq6sq6devw8PAAwGg04ubmRkpKCmVl\nZcyZMwej0cjChQuJiYmp93X8/DybNEikrlF9zkLyNY2z5nv78184fvYyXYJ9GNY32NFx6uSsx6+a\n5Gt+NisKer3e0gwEv93ZSqvVolar8ff3B2Dt2rWUlpYyfPhwTp06xdy5c5k2bRoZGRnMmzePbdu2\n1Xvrv6YMIw8I8G7SNBm2Jvmaxlnz5eSXcvzsZQD+u+csXTvqrOzhGM56/KpJvqapq2DZrCjodDpK\nSkosj81mc403d7PZzJtvvkl6ejrvvfceKpWKiIgIwsPDLV+3a9eO3NxcgoOd95OUEDdrz9FLALho\n1RxJu0xhiQFfL1crewlhHzZrzBw4cCC7d+8GIDk52XIj9mpxcXFUVFSwatUqSzPShg0bWLlyJVB1\nr1i9Xk9AQICtIgphd2azwt5jWXi4aXhkXA/MisK+Y5ccHUsIC5udKYwdO5aEhARiY2NRFIX4+Hg2\nb95MaWkpffr0YcOGDQwePJhHH30UgNmzZzN16lQWL17MzJkzUalUxMfH19t0JERLc/JcPleKKri9\nfwhjh4az9tuT7DmaxT1DOqNSqRwdTwjbFQW1Ws3SpUtrLIuKirJ8nZKSUut+b7/9tq0iCeFwPx/J\nBGBEv2C8PV2J6RbAwZQc0rOKiQzxcXA6IWTwmhB2U1JeyaFTeXRs70nU1QIw4uqVR3uOZjkymhAW\nUhSEsJMDJ7IxmsyM6BdsaSrqE9GedjpX9p/IxlBpcnBCIaQoCGE3e45moVLBsN4dLcvUahW39Qmm\nrMLIodO5DkwnRBUpCkLYwcVcPelZxfSN7ICft1uNdSP6VTUhJRyRJiTheFIUhLCD6j6DEbWMXu7Y\n3pOunXw5kZHP5cJye0cTogYpCkLYmNFkZt+xS+g8XIjp5l/rNiP6BqMAe4/J2YJwLCkKQtjY0bOX\nKSqt5NZeQWg1tf/K3dIjEFcXNXuOZmFWFDsnFOI3UhSEsLE9V/sKqvsOauPhpmVw90ByC8o5fb7A\nXtGEuIEUBSFsqKjEwJG0y4QF6ggLqn/GTMuYBelwFg4kRUEIG9p3/BIms8Lwes4SqkWHtcPf153E\n1BzKKox2SCfEjaQoCGEjiqKw52gWWo2qxtiEuqhVKkb0C8ZQaeZgSo4dEgpxIykKQthIxqViLuaW\nENPVH52HS4P2Gd4nGBUy7YVwHCkKQtiIZWxCA5qOqnXwdadnFz9OXygk+0rjbyAlRGNJURDCBiqN\nJvYfz8ZX50rviPY3ta9MkiccSYqCEDZw6FQepRVGbuvTEY365n7NBkYH4OGmZe+xS5jNMmZB2JcU\nBSFsoL5pLaxxddEwtGcg+cUVHM+40tzRhKiXFAUhmtmVonJOpF8hKtSH4A5ejXqO6ktYZcyCsDcp\nCkI0s4Rjl1CAkf1CGv0ckcE+hPh7cfh0LvqyyuYLJ4QVUhSEaEaKopBwNAtXrZpbegQ2+nlUKhUj\n+gZjNCnsP5HdjAmFqJ8UBSGa0ekLheTklzGoeyAebk27Bfqw3kGoVSq5CknYlRQFIZpRQya/ayhf\nnRv9ojpw7lIxF3L0TX4+IRpCioIQzaTcYCQxJQd/X3e6h7VrluccLmMWhJ1JURCimSSm5FBRaWJ4\n32DUKlWzPGf/rh3Qebiw99gljCZzszynEPWRoiBEM6m+x/LwPtYnv2sorUbNsN4d0ZdV8suZy832\nvELURYqCEM0gO7+UUxcK6Rnuh387j2Z97pFX+ycSpAlJ2IEUBSGaQUIjJr9rqE6BOsI7enMk7TKF\n+opmf34hriVFQYgmMpsVEo5ewsNNw8DoAJu8xoi+wZgVhX3HZcyCsC0pCkI00YlzV8gvrmBIzyDc\nXDQ2eY2hvYLQaqrGLCiKTJInbEeKghBNZBmb0IjJ7xpK5+HCgG4BZOaVkJ5VbLPXEUKKghBNUFJe\nyaFTeQR38CQyxMemr1XdXyFjFoQtSVEQogn2n8jGaDIzom8wqmYam1CX3l3a4+ftxv4T2RgqTTZ9\nLdF22awomM1m4uLimDFjBrNmzeLcuXM11m/ZsoVp06YRGxtLXFwcZrPZ6j5COJs9R7JQq1QMa8ax\nCXVRq1Xc1qcjZRVGDp3KtfnribbJZkVhx44dGAwG1q9fz4svvsjKlSst68rLy/nrX//Kp59+yrp1\n69Dr9ezatavefYRwNhdy9WRcKqZvZHva6dzs8ppyq05hazYrCklJSYwcORKAmJgYjh07Zlnn6urK\nunXr8PCoGuRjNBpxc3Ordx8hnE1zTn7XUEHtPenWyZeTGflcLiy32+uKtqNpc/vWQ6/Xo9PpLI81\nGg1GoxGtVotarcbf3x+AtWvXUlpayvDhw/n222/r3Kcufn6eaLWNvwwwIMC70fvag+RrGlvlM5rM\nHDiZg4+XK2NujcBF27jPV43Jd+9tEbz7eTLJ6VeIHdu9Ua/bUG31+9tcnD1fbWxWFHQ6HSUlJZbH\nZrO5xpu72WzmzTffJD09nffeew+VSmV1n9rk55c2OmNAgDe5uc57eZ/kaxpb5jt8KpcCfQV3De5E\nQX6J9R1q0dh83UN9cHVR893/ZXBn/+abfO96bfn72xxaQr7a2Kz5aODAgezevRuA5ORkoqOja6yP\ni4ujoqKCVatWWZqRrO0jhLNITM0BYHgf+zUdVfNw0zK4eyC5BeWcu+S8bzqiZbLZmcLYsWNJSEgg\nNjYWRVGIj49n8+bNlJaW0qdPHzZs2MDgwYN59NFHAZg9e3at+wjhjM5cKMTLXUtYkM76xjbQM9yP\nvccukXaxkIhg246PEG2LzYqCWq1m6dKlNZZFRUVZvk5JSal1v+v3EcLZFJYYyCssp19UB5uPTahL\nVKgvAGmZRdzlkASitZLBa0LcpLMXCwGIsvEI5voE+Xng5a4l7WoWIZqLFAUhbtKZzKo34sirn9Yd\nQaVSERXqS15hOYUlBoflEK2PFAUhbtLZi0WogEgHt+VXn6mclbMF0YykKAhxE0xmM+mXiggN8MLD\nzWZdcg1S3a9QfeYiRHOQoiDETbiQU4Kh0kxkiOOajqpFBPugAtIuFjk6imhFpCgIcRPSrn4qjwp1\n/GWgHm5aQgO8yMgqwmQ2OzqOaCWkKAhxE9IsVx45/kwBIDLEF4PRzIWcxo2qFuJ6UhSEuAlpmUV4\numnp2MHT0VGA385Y0qRfQTQTKQpCNFBRqYGc/DIiQ31sNt/QzepaPYhNrkASzUSKghANdPZqh66z\nNB1B1VTanm5a6WwWzcZqUTAYDKxevZo//OEP6PV63n//fQwGGSwj2h5n6mSuplapiAz1IaegjKJS\n+b0UTWe1KCxdupSysjJOnDiBRqPh119/5dVXX7VHNiGcSnUTjaMHrV2v+szlrJwtiGZgtSgcP36c\nhQsXotVq8fDw4I033uDkyZP2yCaE0zCbFdKzignx98LT3cXRcWqQzmbRnKwWBZVKhcFgsMwGmZ+f\n77CZIYVwlAu5eioqTQ6dBK8ukcG+VwexSVEQTWd1nP7s2bN5/PHHyc3N5fXXX2fHjh0sWLDAHtmE\ncBppmVc7mR04CV5dPN21BPt7kZ5VjMlsRqOW60dE41ktCg888AB9+vRh//79mEwmVq9eTY8ePeyR\nTQin4QzTZdcnKsSHzLwSLuaWEBbU8u4LLJyH1aIwceJEJk2axPjx4wkKCrJHJiGczpnMIjzcNAT7\nezk6Sq2iQn35+UgWaZlFUhREk1g9z3z77bcpKipi9uzZPPbYY3z55ZeUlMiQetF26Msqyb5SSmSw\n8wxau55Moy2ai9Wi0K1bN1544QW2b9/O008/zWeffcbw4cPtkU0Ip3C2+qY6TjRo7XrB/l54uGk4\nkymXpYqmsdp8ZDKZ2LNnD1u3biUxMZERI0bwyiuv2CObEE7hzEXn7WSuplapiAz24XhGPvqySnQe\nznXZrGg5rBaFUaNG0b9/fyZOnMjy5ctxdXW1Ry4hnMZvZwrO2clcLSrUl+MZ+ZzNLKRflL+j44gW\nympR2LJlC+3atbNHFiGcjtmscDaziI7tPZ3+03d189aZi0VSFESj1VkUnnjiCdasWcODDz5YY7Ca\noiioVCp++OEHuwQUwpEyL5dQbjA51XxHdak+kzkrI5tFE9RZFJYtWwbA2rVr7RZGCGfjbDfVqY/O\nw4WO7T05m1mE2aygVjvnlVLCudV59VFgYCAAK1euJDQ0tMZf6WgWbUVaC+hkvlZUqA/lBhOZeXLZ\nuGicOs8UFixYQEpKCjk5OYwZM8ay3GQy0bFjR7uEE8LR0jILcXPVEOqkg9auFxXqS8LRS6RlFtIp\nUOfoOKIFqrMovPHGGxQUFPD666/z2muv/baDVkuHDh3sEk4IRyopryTrcik9w/1aTFNMdTNX2sUi\nRsWEOjiNaInqbD7S6XR06tSJ1atXU1hYSFZWFpmZmaSnp/P111/bM6MQDpFumQTP+TuZq4X6e+Hm\nqpFptEWjWb0k9eWXX+bw4cMUFhYSGRlJSkoKAwcOZOrUqfbIJ4TDnLno/COZr6dWVw1iO3kun5Ly\nSryc7N4PwvlZneYiMTGRrVu3cs8997Bs2TI+//xzuR2naBMs02U7+aC161Wf2ZyVKS9EI1gtCoGB\ngbi4uBAVFUVqairdunWTCfFEq2dWqgatBfl54O3Zskbx/9avIE1I4uZZbT4KCgpizZo1DBs2jDff\nfBOA0tJSmwcTwpGyLpdSVmEkpmvLGxlcPYgtTc4URCNYPVN4/fXX6dSpE/369ePuu+9my5YtLFmy\nxOoTm81m4uLimDFjBrNmzeLcuXM3bFNWVkZsbCxpaWmWZZMnT2bWrFnMmjWLxYsX39z/RohmUj0F\nddcW1MlczdvTlSA/j6pBbIri6DiihbF6pvC3v/2NiRMnAljerBtix44dGAwG1q9fT3JyMitXrmT1\n6tWW9UePHuVPf/oT2dnZlmUVFRUoiiKjqIXDpbWA6bLrExniy77jl8i6XNpixlgI52C1KHTp0oX4\n+HgKCwsZP348EydOpFOnTlafOCkpiZEjRwIQExPDsWPHaqw3GAx88MEH/OEPf7AsS0lJoaysjDlz\n5mA0Glm4cCExMTH1vo6fnydarcZqnroEBDj3XaokX9M0Nl9Gth53Vw0DenVEo7HdPY9tdfxiegSy\n7/glcooqiOnZ+MGmrfX7ay/Onq82VovCww8/zMMPP0xmZibffvstCxYswNPTk//85z/17qfX69Hp\nfhtRqdFoMBqNaLVVLzlo0KAb9nF3d2fu3LlMmzaNjIwM5s2bx7Zt2yz71CY/v/H9GwEB3uTmFjd6\nf1uTfE3T2Hyl5UbOXyqme1g7rlyx3UUVtjx+QT5uAPySms2AyPaNeo7W+v21l5aQrzZWiwJAcXEx\ne/fuJSEhAZPJxIgRI6zuo9PpalylZDab631zB4iIiCA8PByVSkVERATt2rUjNzeX4ODghsQUolmk\nZxWh0HKbjgBCA7xwc9FY5m4SoqGsnhc/+eST3H///aSkpPDcc8+xZcsWFixYYPWJBw4cyO7duwFI\nTk4mOjra6j4bNmxg5cqVAGRnZ6PX6wkICLC6nxDNqbo/oSWNZL6eRq0mItibzLwSSsuNjo4jWhCr\nZwrTp0/n9ttvt/op/3pjx44lISGB2NhYFEUhPj6ezZs3U1payowZM2rdZ+rUqSxevJiZM2eiUqmI\nj4+/6dcVoqksM6O24DMFqDrTSfm1gPSsInpHNK4JSbQ9db7jvvfeezzzzDN8//33fP/99zesX7Fi\nRb1PrFarWbp0aY1lUVFRN2x37ZVGrq6uvP3221ZDC2EriqJwNrOQgHbu+Hi1rEFr16s+00nLLJSi\nIBqszqLQu3dvAIYMGWK3MEI42qUrpZSUG+kb1fJnAr52xlQhGqrOojB69GgANm/ezD//+U+7BRLC\nkVpL0xGAj5crAe3cOZtZiFlRUKtaxvTfwrGsdjRXVFSQlZVljyxCONzZVtDJfK2oUF9Kyo1kX5Gp\naUTDWO3FvXLlCqNHj6ZDhw64ubmhKAoqlYoffvjBHvmEsKszF4tw1arpFNA67loWFeLL/x3PJu1i\nEcEdZGSzsM5qUfif//kfe+QQwuHKKoxczNPTLdQXrQ1HMdvTb9NoFzKin4z3EdZZ/ckPDQ3l0KFD\nfP7557Rv357ExERCQ+U2f6L1ycgqQlEgMrTl9ydU6xSgw1Wr5ox0NosGsloU3nrrLX766Se+++47\nTCYTGzdutAwwE6I1OZPZejqZq2k1arp09OZinp6yChnEJqyzWhT27NnDm2++iZubGzqdjo8//tgy\nUlmI1qR6uuzW0slcLSrUF0WpOhMSwhqrRUGtrtpEdfVyNoPBYFkmRGuhKAppmUV08HGnnc7N0XGa\nVfUcTmfkpjuiAax2NI8bN47nn3+ewsJCPvnkEzZt2sT48ePtkU0Iu8kpKENfVkmvLn6OjtLsLJ3N\ncntO0QBWi8L8+fP5+eefCQkJISsri2eeeYY777zTHtmEsJvq+xm3pv6Eau10bnTwcScts8hySbkQ\ndbFaFE6dOkVJSQlDhw4lKiqKzp072yOXEHZlGcnciq48ulZUqA8HTuaQk19GUHtPR8cRTqzOonD5\n8mWeffZZTp8+bbnHQXp6OgMGDOCtt97Cx6d1dcaJti0tsxCtRk1YUOsYtHa9qFBfDpzMIS2zUIqC\nqFedPcbLli1j0KBBJCQk8MUXX/D555+TkJBA9+7diY+Pt2dGIWyqwmDiQk4JXTp6t5pBa9eTyfFE\nQ9X5G5CamsrChQtxcXGxLHN1dWXhwoWcOHHCLuGEsIeMS0WYFaXVXYp6rbAgHVqN2nIDISHqUmdR\ncHOr/bI8lUoll6SKVuVMK+5krlY9iO1CTgkVBpOj4wgnVue7e31XKMjVC6I1ae2dzNWiQn0wKwrp\nMohN1KPOjubTp08zZsyYG5YrikJubq5NQwlhL9V3Wmvv44afd+satHa9qjOh86RlFtIjvPWNxxDN\no86isH37dnvmEMIhcgvLKSqtZHCPQEdHsbnqMyHpbBb1qbMoyEyooi2oHuXbNaT1djJX8/N2o72P\nG2czC2UQm6iT9BiLNq36U3Nrmi67PpEhvhSVVpJbWO7oKMJJSVEQbdqZzEK0GhXhQd6OjmIX1WdE\naTIPkqhDnc1HiYmJ9e54yy23NHsYIeypotLEhRw9XTp646JtG5+PqvsVzl4sYljvjg5OI5xRnUXh\n3XffBaCgoIBff/2VgQMHolarOXz4MNHR0axbt85uIYWwhXOXijGZFcvU0m1BWJA3Wo2KMzKITdSh\nzqKwdu1aAObNm8f7779PeHg4ABcvXiQuLs4+6YSwoerRva15JPP1XLRqwoO8ybhUTEWlCTcXjaMj\nCSdj9Zw5MzPTUhAAQkJCyMzMtGkoIezBMmitDZ0pQFVns8mscO5SsaOjCCdkders3r178/LLL3Pv\nvfdiNpvZsmULgwcPtkc2IWxGURTSLhbSTudKe5/WPWjtelGhPnx/sKqzObpzO0fHEU7GalFYvnw5\nn332maUP4bbbbuOhhx6yeTAhbOlyUTmFJQYGdQ9oc9frd60exCa35xS1qLMo5ObmEhAQQF5eHuPG\njWPcuHGWdTk5OYSEhNgloBC20FabjqBqEFs7nStpF2UQm7hRnUXhtddeY82aNTzyyCOoVCoURbGs\nU6lU/PDDD3YJKIQtWGZGbUOdzNVUKhVRob4kpeaSV1hOQDsPR0cSTqTOorBmzRoAdu7cabcwQtiD\noigkn87Fw01Dl45trygA9OrSnqTUXA6fyuXuIWGOjiOciNWrj65cucLzzz/P0KFDGTx4ME8//TR5\neXlWn9hsNhMXF8eMGTOYNWsW586du2GbsrIyYmNjSUtLa/A+QjTV2cwiLhdVMKBbQJsZtHa9QdEB\nqFSQmJLj6CjCyVj9jYiLi6Nv37788MMP7Ny5k/79+/Pqq69afeIdO3ZgMBhYv349L774IitXrqyx\n/ujRozxhu9JhAAAeJklEQVT88MOcP3++wfsI0RwOnKx6IxzSs/XPjFoXHy9Xeob7kZZZRF5hmaPj\nCCditSicP3+euXPnotPp8PHxYd68eQ0ap5CUlMTIkSMBiImJ4dixYzXWGwwGPvjgAyIjIxu8jxBN\nZVYUDqbm4OWupVeX9o6O41C3XJ0uXM4WxLWsXpKqUqnIysoiODgYqBrMptVa3Q29Xo9Op7M81mg0\nGI1Gy76DBg266X1q4+fniVbb+FGZAQHOPRGa5Gua6/MdP3uZ/OIKxg4JI7ij4688cuTxu/u2SD77\n7hSHz1xm9vg+tW7T0r6/zsbZ89XG6rv7c889x4wZM+jfvz+KovDLL7+wbNkyq0+s0+koKSmxPDab\nzVaLSWP2yc8vtZqlLgEB3uTmOu+oTsnXNLXl+35fBgB9I/wcnt0Zjl/PLn4cO3uF46eyCfTzrLHO\nGfLVR/I1TV0Fy2rz0Z133snXX3/NlClTePDBB/n666+54447rL7gwIED2b17NwDJyclER0fbZB8h\nGspsrmo60nm40CNMbkcJ0oQkbmT1TOHKlSts3bqVwsKq67pPnDgBwNNPP13vfmPHjiUhIYHY2FgU\nRSE+Pp7NmzdTWlrKjBkzGryPEM3l1PkCCksMjIoJQatpm1cdXW9gdACfbksl8WQO9w/r4ug4wglY\nLQrz5s0jOjr6pm/PqVarWbp0aY1lUVFRN2xXPRtrXfsI0VwOXP00PKQN3I+5obzcXegT0Z5f0i5z\n6UopHdt7Wt9JtGrWe4yBFStW2DqHEDZlMptJSs3Bx9OF6DCZBO5at/QM5Je0yxw4mc3E4RGOjiMc\nzOo59F133cUXX3zB+fPnyczMtPwVoiVJOVdAcWklg3oEolFL09G1BnQLQKtRk3hS+hVEA84UiouL\n+fDDD/Hz+61jTuY+Ei1NYko2IE1HtfFw09I3sj2HT+dxMVdPaIDO+k6i1bJaFL777jv27duHu7u7\nPfII0eyMJjNJqbn46lzp1kmajmozpGcQh0/nkZiSI0WhjbN6Ht25c2fLlUdCtEQnMvIpKTdyS/dA\n1GqZJro2/bt2wFWr5sDJnBozIou2p0Ejmu+//366deuGi4uLZfmnn35q02BCNBdL01HPIAcncV7u\nrlr6RXXgYGouF3JL6BwoZwttldWi8OSTT9ojhxA2UWk0c+hUHn7ebkS2wXsn3IwhPYM4mJrLgZPZ\nUhTaMKvNRyqVqsZftVqNh4cHRUVyKz/h/I6nX6GswsgtPQJRyx3G6tU3qgNuLhoSpQmpTbN6pvDB\nBx9w7Ngxhg0bhqIoHDhwgNDQUPR6Pc899xzjx4+3R04hGkWajhrOzUVD/64dOHAyh1+z9QQGyplV\nW2T1TEFRFDZt2sR7773H+++/z+bNm2nfvj1fffUVH330kT0yCtEohkoTh0/n4e/rTkRwy5ut0hGq\ni+eBk9kOTiIcxWpRyMnJISQkxPI4KCiInJwcdDqdnGIKp5aUkk25wcQtPQLl5vQN1DeyPe6uGhJT\npAmprbLafDRgwABefPFFJkyYgNlsZuvWrQwYMIAff/wRT0+ZJ0U4rz3JVSPvpemo4Vy0GgZ082ff\n8WxOny/Az6NBM+GIVsTqmcLSpUsZMGAA69ev58svv2TQoEHExcWhUqn4y1/+Yo+MQty0ikoTB05c\nIrCdB2FBciXNzbjlahH9Ofmig5MIR6jzY0Bubi4BAQHk5OQwevRoRo8ebVmXk5PDqFGj7BJQiMY4\nknaZcoOJMYOk6ehm9e7SHg83LXuSLzL+1jC5aquNqbMovPbaa6xZs4ZHHnnE8ktV3cYocx8JZ1fd\nUSpNRzfPRatmYLQ/CUcvkXaxUKYGaWPqLApr1qwBYOfOnXYLI0RzKDcYOZJ2mdAAHZ0CvBwdp0Ua\n0jOIhKOXSDyZI0Whjam3T2HXrl2cP38egB07dvDkk0/y7rvvYjQa7RJOiMZIPpNHpdHMyJhQaTpq\npJ7hfnh7upCYmoPZLFchtSV1FoWPPvqI999/n4qKClJSUvj973/PmDFjKCkp4Y033rBnRiFuSvV9\nAUbEhFjZUtRFq1EzrG8IhXoDpy8UODqOsKM6i8I333zDZ599RteuXdmyZQujR49m2rRpLFq0iD17\n9tgzoxANVlZh5OjZy4T6exHeUUbkNsXIq0W1+jamom2osyioVCo8PDwA2L9/PyNHjrQsF8JZHT6d\ni9GkcEtPuZlOU/WN8sfb04WklBxMZrOj4wg7qbMoaDQaioqKuHTpEidPnmT48OEAXLx4Ea1WBrQI\n53TgatPRLXKHtSbTaNQM6h5IUWklqb9KE1JbUWdRmD9/Pg888ADTp09n6tSpBAYG8t///pfHHnuM\nuXPn2jOjEA1SUl7J8fQrdA7UEdxBrjpqDtW3L02UJqQ2o86P/OPGjWPAgAHk5+fTo0cPALy8vFi+\nfDlDhw61W0AhGurQqVxMZoUh0nTUbKI7t8PXy5Wk1FweHhuNVmN1EgTRwtX7HQ4KCrIUBIBRo0ZJ\nQRBOK1GajpqdWq1icPdA9GWVpJzLd3QcYQdS9kWroC+r5ERGPuEdvQn0k4kam1N1p71chdQ2SFEQ\nrUJSag5mRZqObKFrJ1/8vN04lJqL0SRXIbV2UhREqyBXHdmOWlXVhFRaYeR4+hVHxxE2JkVBtHiF\nJQZSfs0nMsQHf18PR8dplarPwKqLr2i9pCiIFu9Qag6K8tvlk6L5RYb40MHHneQzuVQaTY6OI2xI\nioJo8ao/vQ6WomAzKpWKW3oGUlZh4thZaUJqzaQoiBYtv7iCU+cL6NrJl/Y+7o6O06pV99fIVUit\nmxQF0aIlpeagIE1H9tClozcB7dxJPp2HoVKakForm01iZDabWbJkCampqbi6urJ8+XLCw8Mt63fu\n3MkHH3yAVqtlypQpTJ8+HYDJkyej01XdU7dTp06sWLHCVhFFK3AgJQcV0nRkDyqViiE9g9i67xxH\n0i7LMW+lbFYUduzYgcFgYP369SQnJ7Ny5UpWr14NQGVlJStWrGDDhg14eHgwc+ZMRo8ejbe3N4qi\nsHbtWlvFEq3IlaJyzlwopHvndrTTuTk6TptwS49Atu47x4GUHCkKrZTNikJSUpJluu2YmBiOHTtm\nWZeWlkZYWBi+vr4ADBo0iMTEREJCQigrK2POnDkYjUYWLlxITExMva/j5+eJVqtpdM6AAO9G72sP\nkq9uCSeq2rZHDwmrM4ccv6a5Pp+/v47QAC+OpF3G28cDdzfHzpjc0o5fS2Cz76her7c0A0HVVNxG\noxGtVoter8fb+7eD5eXlhV6vx93dnblz5zJt2jQyMjKYN28e27Ztq3eq7vz80kZnDAjwJje3uNH7\n25rkq9+ug7+iUkH3EJ9aczg6nzUtNd/AbgFs3pvBD/szGNIzyAHJqrTU4+cs6ipYNuto1ul0lJSU\nWB6bzWbLm/v160pKSvD29iYiIoKJEyeiUqmIiIigXbt25Obm2iqiaMHyCso4m1lEjzA/fLxcHR2n\nTblFBrK1ajYrCgMHDmT37t0AJCcnEx0dbVkXFRXFuXPnKCgowGAwcPDgQQYMGMCGDRtYuXIlANnZ\n2ej1egICAmwVUbRg1ZdFylxH9tcpQEeIf1UTUmm50dFxRDOzWfPR2LFjSUhIIDY2FkVRiI+PZ/Pm\nzZSWljJjxgwWLVrE3LlzURSFKVOmEBQUxNSpU1m8eDEzZ85EpVIRHx8vd3kTN8gpKGPrvgzcXDQM\n6i5FwRFu69ORDT+m8b87TvG78b0cHUc0I5u946rVapYuXVpjWVRUlOXr0aNHM3r06BrrXV1defvt\nt20VSbQCRpOZNd8cp6zCxNz7e6LzcHF0pDbp7ls6k5Saw95jl+gZ7sfwvsGOjiSaiQxeEy3Kl7vP\nkp5VxLDeHeWNyIG0GjVPTOqDh5uGz747xaUrjb/gQzgXKQqixTiSdplt+38lyM+DR+6Otr6DsKnA\ndh48Oq4HFZUm/v71MZkor5WQoiBahPziCv5nywm0GhVPTuqDh4OvjxdVhvQM4vb+wfyao+fzXWmO\njiOagRQF4fTMZoV/bD6OvqyS6Xd2JbxjyxsQ1JrNvCuaEH8vfki6wKFTcgl5SydFQTi9rfsySPm1\ngAHd/BkzqJOj44jruLloeHJSb1y0aj7+70kuF5Y7OpJoAikKwqmdOl/A13vSae/jxuP39USlUjk6\nkqhFpwAdD93VjZJyI2s2H8dklns5t1RSFITT0pdVsmbTcVSomD+ht1x+6uRu7x/CLT0COXOhkG/2\npDs6jmgkKQrCKSmKwj+3niS/uIJJIyOI7tzO0ZGEFSqVikfH9cDf152te89xPEPu0NYSSVEQTmlH\n0gWSz+TRM9yP+28Nt76DcAqe7lqenNQHtVrF/2w+QWGJwdGRxE2SoiCczrlLxXyx6wzeni7Mm9AL\ntVr6EVqSyBAfpoyKorDEwEdbTmBWFEdHEjdBioJwKmUVRlZ/cwyjSeF343vJzXNaqLuHdKZvZAeO\npV9h+/5fHR1H3AQpCsJpKIrC2u9SyckvY9zQMPpGdnB0JNFIapWKueN74qtz5cvdZ0m7WOjoSKKB\npCgIp5Fw9BL/dzybyBAfHrw90tFxRBP5eLoyf0JvzGaFNZuOU1pe6ehIogGkKAinkHW5hM++T8XD\nTcsTE3uj1ciPZmvQM9yP8bd1Ia+wnI+/TUGR/gWnJ795wuEMlSZWf30cQ6WZx+7tQUA7D0dHEs1o\n4oguRHfyJSk1lx+TMx0dR1ghRUE43PpdZ7iQq+eOmKrBT6J10ajVzJ/YGy93Lf/ZcZrzOXpHRxL1\nkKIgHCopNYddhy4SGuBF7Jhujo4jbKS9jztz7++F0WTm798co8Ig02w7KykKwmHyCsr4+L8puGrV\nPDmpD64uGkdHEjYU082fuwZ3IutyKf/eccrRcUQdpCgIhzCazKzZfJzSCiMPj40m1N/L0ZGEHUy7\noyvhQd7sOZLF/x2/5Og4ohZSFITd5RWWseab46RdLGJoryBG9JPbarYVLlo1T07qjZurhn9tT2X3\nL5kyo6qTkaIg7EZfVsn6nad55cP9JJ3KJSLYm9n3dJfpsNuYoPaezL2vJ4pZ4ZNvU4j76ACHT+fK\n5apOQu5pKGyu0mhiR9IFtu49R2mFkQ4+bky+PZJbe3dELQWhTRrcI5CoUF++2ZPOz0cyeW/jUbp1\n8mX6nV2JCvV1dLw2TYqCsBmzWWHf8Ut89fNZrhRV4OWuZfqdXRkzKBQXrXQqt3V+3m48dm8Pxt7S\nmY0/ppF8Jo/X1yYxKDqAKXdE0bG9p6MjtklSFESzUxSFY+lX+GLXGS7klqDVqBk3NIz7h4Xj5S43\nyhE1hfp78ezUfpw6X8AXu86QdCqXw6fzGBUTwsQREfh6uTo6YpsiRUE0q4xLRXyxK42T5/JRAcP7\ndOSBkZF08HV3dDTh5KI7t+OVWYM4dCqXDT+dZdfhi+w9dol7hnRm3NAw3F3l7coe5CiLZpFTUMaX\nP6Vx4GQOAH0i2zPtjq50DtQ5OJloSVQqFYO6B9K/qz8/H8nimz3pbErI4MfkTCYN78LI/iEyL5aN\nSVEQTVJcamDz3gx2HbqIyawQ3tGb6XdE0bNLe0dHEy2YVqPmzgGhDOsdxPYD59m2/1fWfneK7w5e\nYMrtkQzqHuDoiK2WFAXRKCXllezacYoNO09RVmHC39edB0dFMqRnkFxRJJqNu6uWSSMiuGNAKJsS\n0vnpcCarvj5GVIgPcyb1IdDbFY1azhyakxQFYVWFwcS57GIysopIv1RMelYROfllAOg8XJg5JpI7\nBoTiopVfTmEbvl6uzLq7O2MHd2bjT2kkpeby6uq9uLqoCQvyJqKjDxHB3kQE+xDo5yFjX5pAioKo\nwWgycz5HX1UAsopJv1REZl4J144r8nTT0quLHwN7BnFr90A83eXHSNhHx/aeLJjcl7SLhRxIzeVk\n+mXSLhZy5sJvd3bzdNPSJdibLtcUCj9vNykUDSS/zW2Y2ayQebmE9KwiMrKqzgAu5Ooxmn6rAK4u\narqG+hIR7EOX6k9i7ao+iQUEeJObW+zA/4Foq6JCfbk1phO5ucW/ncleuno2m1XEiYx8TmTkW7b3\n8XIloqO35ee4S7APPp5yqWttbFYUzGYzS5YsITU1FVdXV5YvX054eLhl/c6dO/nggw/QarVMmTKF\n6dOnW91H1E9RFMoqTOjLDOjLjOjLDBSXVlJSVklxWSX6skr0pVf/Laskr7CcisrfpjDWqFV0DtT9\nVgA6+hDs7ylttsKpublqiO7cjujO7SzLSsorrykSVR94fkm7zC9ply3b+Hm74evlis7DBZ2nS9W/\nHi54e7ig83S1PK7+21aaR21WFHbs2IHBYGD9+vUkJyezcuVKVq9eDUBlZSUrVqxgw4YNeHh4MHPm\nTEaPHs2hQ4fq3MfRFEVBAVBAQUFRQFHArCgoioLZfM3XStWn8Kqvqx5XbXP1a3PVcpNZwWgyYzSa\nqTQpVBrNVY9NZipNZtzcXSgoLLOsr/r36nqjmQqDyfIGX1xW9eZvMjds/hgvdy3+7dzpcvXTU0Sw\nD50CdG3mB1+0bl7uLvTu0p7e11wFV6ivIP2aQnExT8/FvBIqjQ2bkM/NVVNVMK7+9bpaKFw0arQa\nNVqtCheNGhdt1WM/Xw/Kyyur1mvVaDWqmttr1KjVKtSqqktxa3x99bFKheXrmuuqbl6kVjd/k5jN\nikJSUhIjR44EICYmhmPHjlnWpaWlERYWhq9v1RwngwYNIjExkeTk5Dr3aW5f/HiGH5IuVr3ZX323\nr243V66+8V/94/S83LXoPFwIaOeOt4crXh7a3/69/hOPpwte7lr59C/aHF+dGzFd3Yjp6l9jeUWl\nqcYZdHGZocZj/bVn2eWVN1VIbMnNVcOrswbRKaB5xwLZrCjo9Xp0ut/CajQajEYjWq0WvV6Pt7e3\nZZ2Xlxd6vb7eferi5+eJthHz6ER08iPsYhEqQKUCFSqu/rF0SKlU131NVeW+dl1VBf+teldVdxWa\nayu9WnVNtf9tuUatxtWl6pODi4um6lOE9sbHrlo1Ltpr1l997O6qQefhgsaBg3kCArytb+RAkq9p\nJF/tyg1GSsoqMVSaqTSaMBivnvEbzRiMJiqvfl3197fH1euqtzWbf2tNqG5dMFUvM1d9UDVf8/ja\n5R5uWiLD2uOrc2vW/5vNioJOp6OkpMTy2Gw2W97cr19XUlKCt7d3vfvUJT+/tFH5BnftwL3Dujh1\nR2m9HbkmE4YyE1fKDPYNdQ1n72iWfE0j+azTAlqNCg+NBtxqfji1Rz5DmYHcRr4H1FVQbfYRc+DA\ngezevRuA5ORkoqOjLeuioqI4d+4cBQUFGAwGDh48yIABA+rdRwghhO3Z7Exh7NixJCQkEBsbi6Io\nxMfHs3nzZkpLS5kxYwaLFi1i7ty5KIrClClTCAoKqnUfIYQQ9qNSWvjtjppyeuYMp5/1kXxNI/ma\nRvI1TUvIVxu5BEUIIYSFFAUhhBAWUhSEEEJYSFEQQghhIUVBCCGERYu/+kgIIUTzkTMFIYQQFlIU\nhBBCWEhREEIIYSFFQQghhIUUBSGEEBZSFIQQQlhIURBCCGFhs6mznc3333/Ptm3bePvtt4Gq+zW8\n/vrraDQaRowYwdNPP11j+/Lycl566SUuX76Ml5cXb7zxBu3bt6/tqZvNhx9+yM8//wxAUVEReXl5\nJCQk1Nhm+fLlHDp0CC8vLwBWrVpV4y52tqQoCrfffjtdunQBqm6Z+uKLL9bY5vPPP2fdunVotVqe\neuop7rzzTrtkAyguLuall15Cr9dTWVnJokWLGDBgQI1tHHH8zGYzS5YsITU1FVdXV5YvX054eLhl\n/c6dO/nggw/QarVMmTKF6dOn2zTP9SorK3nllVe4ePEiBoOBp556ijFjxljWf/LJJ3zxxReWn/8/\n//nPREZG2jXj5MmTLXdl7NSpEytWrLCsc/Tx+/LLL/nqq68AqKio4OTJkyQkJODj4wM4x/G7KUob\nsGzZMuWee+5Rnn/+ecuyiRMnKufOnVPMZrPyu9/9Tjl+/HiNff75z38q7777rqIoirJlyxZl2bJl\nds08f/585eeff75heWxsrHL58mW7ZqmWkZGhPPHEE3Wuz8nJUcaPH69UVFQoRUVFlq/t5W9/+5vy\n8ccfK4qiKGlpacoDDzxwwzaOOH7bt29XXn75ZUVRFOXw4cPKk08+aVlnMBiUu+66SykoKFAqKiqU\nBx98UMnNzbVrvg0bNijLly9XFEVR8vPzlVGjRtVY/+KLLypHjx61a6ZrlZeXK5MmTap1nTMcv2st\nWbJEWbduXY1ljj5+N6tNNB8NHDiQJUuWWB7r9XoMBgNhYWGoVCpGjBjB3r17a+yTlJTEyJEjAbj9\n9tvZt2+f3fJ+9913+Pj4MGLEiBrLzWYz586dIy4ujtjYWDZs2GC3TADHjx8nOzubWbNmMW/ePM6e\nPVtj/ZEjRxgwYACurq54e3sTFhZGSkqK3fI99thjxMbGAmAymXBzq3nvWkcdv2t/lmJiYjh27Jhl\nXVpaGmFhYfj6+uLq6sqgQYNITEy0S65q48aN47nnngOqzgY1mpq3lTx+/DgffvghM2fOZM2aNXbN\nBpCSkkJZWRlz5sxh9uzZJCcnW9Y5w/GrdvToUc6cOcOMGTNqLHf08btZrar56IsvvuBf//pXjWXx\n8fHcd9997N+/37JMr9dbTkUBvLy8OH/+fI399Hq9pVnBy8uL4uLmvVlGXVn79evHmjVreOedd27Y\np7S0lEceeYTHH38ck8nE7Nmz6dOnDz169GjWbHXli4uLY/78+dx7770cPHiQl156iY0bN1rWX3vM\noOq46fX6Zs9WV77q45ebm8tLL73EK6+8UmO9PY/fta7/edNoNBiNRrRarV2PWV2qm9L0ej3PPvss\nzz//fI31999/Pw899BA6nY6nn36aXbt22bVZ0N3dnblz5zJt2jQyMjKYN28e27Ztc5rjV23NmjUs\nWLDghuWOPn43q1UVhWnTpjFt2jSr2+l0OkpKSiyPS0pKLO1/tW1T23pbZT1z5gw+Pj412pyreXh4\nMHv2bDw8PAC49dZbSUlJscmbWm35ysrKLJ8iBw8eTE5ODoqioFKpgNqPq63a6+s6fqmpqSxcuJA/\n/OEPDBkypMY6ex6/a11/XMxmM1qtttZ1tjxm9cnKymLBggU89NBDTJgwwbJcURQeffRRS6ZRo0Zx\n4sQJu76pRUREEB4ejkqlIiIignbt2pGbm0twcLDTHL+ioiLS09O59dZbayx3huN3s9pE89H1dDod\nLi4u/PrrryiKwp49exg8eHCNbQYOHMhPP/0EwO7duxk0aJBdsu3du5fbb7+91nUZGRnMnDkTk8lE\nZWUlhw4donfv3nbJBfD+++9bPp2npKQQHBxsKQgA/fr1IykpiYqKCoqLi0lLSyM6Otpu+c6cOcNz\nzz3H22+/zahRo25Y76jjN3DgQHbv3g1UXeBw7TGJiori3LlzFBQUYDAYOHjw4A2d47aWl5fHnDlz\neOmll5g6dWqNdXq9nvHjx1NSUoKiKOzfv58+ffrYNd+GDRtYuXIlANnZ2ej1egICAgDnOH4AiYmJ\nDBs27IblznD8blarOlO4GX/+85/5/e9/j8lkYsSIEfTv3x+AOXPm8Pe//52ZM2fy8ssvM3PmTFxc\nXCxXLdlaeno6w4cPr7Hs448/JiwsjDFjxjBp0iSmT5+Oi4sLkyZNolu3bnbJBTB//nxeeuklfvrp\nJzQajeUKkGvzzZo1i4ceeghFUXjhhRduaNe3pbfffhuDwcDrr78OVBX/1atXO/z4jR07loSEBGJj\nY1EUhfj4eDZv3kxpaSkzZsxg0aJFzJ07F0VRmDJlCkFBQTbPdK2///3vFBUVsWrVKlatWgVUnYmV\nlZUxY8YMXnjhBWbPno2rqyvDhg2rteDa0tSpU1m8eDEzZ85EpVIRHx/Pt99+6zTHD6p+bzt16mR5\nfO3319HH72bJ1NlCCCEs2mTzkRBCiNpJURBCCGEhRUEIIYSFFAUhhBAWUhSEEEJYSFEQQghhIUVB\nCCGEhRQFIZrZp59+ysMPP4yiKBw8eJC7777bYfPxCHGzZPCaEM1MURRmz57NuHHjWLt2La+//rrd\npkkRoqmkKAhhA+fPn2fChAmW6VKEaCmk+UgIG8jMzESn03HixAnkc5doSaQoCNHMSkpK+OMf/8iq\nVavw8PDgf//3fx0dSYgGk6IgRDN78803GTVqFP369bMUh+tv4iSEs5I+BSGEEBZypiCEEMJCioIQ\nQggLKQpCCCEspCgIIYSwkKIghBDCQoqCEEIICykKQgghLP4/uazVa4Ta3gwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xba77198>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Graphical representation of the Sigmoid derivative\n",
    "x = array(range(-10, 10))\n",
    "plt.plot(x, SigmoidDerivative(x))\n",
    "plt.title(\"Derivative of Sigmoid\")\n",
    "plt.xlabel(\"x\")\n",
    "plt.ylabel(\"Sigmoid Derivative\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ReLU derivative:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "   \\frac{\\partial ReLU(x)}{\\partial x}=\\begin{cases}\n",
    "    1, & \\text{if $x>0$}.\\\\\n",
    "    0, & \\text{otherwise}.\n",
    "  \\end{cases}\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Graphical Representation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://cdn-images-1.medium.com/max/1600/1*g0yxlK8kEBw8uA1f82XQdA.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Programmatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Derivative of the ReLU activation function\n",
    "def ReLUDerivative(x):\n",
    "    return 1. * (x > 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0xc118d30>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAETCAYAAAA/NdFSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlUVeX+P/D3GQCBg4qG3hywZIl2Mwd0Vf4USbzkLCoi\nYEEOaYNpDlFmV5cTqDmsUtMcblpeE5xS0etsieJdfkEzpzA1IyuXoKGeATjT/v3hZesJDkeBvc+G\n836t1bqc/Zy9+fBc5H32s/d+HpUgCAKIiMjjqN1dABERuQcDgIjIQzEAiIg8FAOAiMhDMQCIiDwU\nA4CIyEMxAMhtfvvtNzzzzDOIjo5GdHQ0BgwYgCFDhmDHjh2VOt6YMWNw5cqVSu179uxZzJgxAwBw\n7tw5TJgwoVLHeRzHjx9Hjx49EBMTg+LiYoe2yMhI9OrVC9HR0Rg0aBD69u2L/v37IzMz0+Vxly1b\nhtmzZ5fblpiYiH379jls+/PPP9G6devK/yBUY2ndXQB5tjp16mDnzp3i699//x0jRoyAr68vevXq\n9VjHWrNmTaXruHLlCm7evAkAeO6557B06dJKH+tR7dmzB7GxsXj77bfLbV+0aBGee+458fW+ffsw\nbdo0HD9+XPLayDPwDIAUpWnTppgwYQL+9a9/AQDMZjNSU1MxePBgDBw4EFOnToXBYABw/1PyxIkT\n0adPHxw8eBCRkZE4d+4cpkyZIu4PAJs2bcLEiRNht9sxd+5cxMbGom/fvujTpw9OnTqFGzduYOnS\npcjJycGHH36IkydPon///tDr9QgLC0NBQYF4rGHDhuHo0aMV1vUwi8WCOXPmoG/fvhgwYAA++ugj\nGAwGrF27FocPH8amTZuwYMECl/0iCAJ+++031KtXT9y2ZcsWDBkyBIMGDcKIESNw9erVSvc7eSYG\nAClOmzZt8NNPPwEAVq9eDY1Gg+3bt2PXrl1o1KgRFi1aJL63VatW2Lt3L6KiosRtsbGxDsNI27dv\nx7Bhw/DDDz8gPz8f6enp+M9//oPBgwdjzZo1ePLJJzFhwgR07twZ8+bNE/cLCAhAVFQUdu3aBQC4\nevUqCgoKEB4e7rKuUitXrkR+fj527tyJnTt3wm634+OPP8brr7+OyMhIjBgxAh988EG5/fDee+8h\nOjoaERERiIiIwM8//4zPP/8cAPB///d/2LFjBzZu3IgdO3bg9ddfx/jx46vQ6+SJOAREiqNSqVCn\nTh0AwHfffQe9Xo8TJ04AuP+JumHDhuJ7O3fuXGb/F154ASUlJTh37hx8fX3x559/okuXLlCpVKhX\nrx7S0tJw/fp1nDx5Ev7+/hXWEhsbi1mzZmH06NHYtm0bhgwZArVa7bKuUpmZmZg0aRK8vLwA3B+D\nHzdu3CP1Q+kQ0PXr1zFy5EiEhISgefPmYr/k5eUhPj5efP/du3dx586dCo+pUqnKbBMEAWo1Pwt6\nIgYAKc65c+cQGhoKALDb7Zg2bRoiIiIAAEajESUlJeJ7/fz8yuyvUqkwdOhQ7Ny5E15eXhg6dChU\nKhW+++47pKSkYOTIkejZsydatmwpfrp3pnPnzrBarTh79ix2796NtLS0R6qrlN1uL/PaYrE8Rm8A\nzZs3x8cff4zExER07twZ7du3h91uR3R0NJKTk8Xj5ufnOwwRlScwMLBMSNy6dQv169d/rJqodmDs\nk6Jcu3YNK1aswKhRowAA3bp1w8aNG2E2m2G32zF9+nQsWbLE5XEGDx6MI0eOYP/+/RgyZAgAICsr\nCz169MDw4cPx3HPP4dChQ7DZbAAAjUYDq9Va7rFiY2MxZ84ctG7dGk2aNHmsusLDw5GWlgaLxQK7\n3Y6NGzeia9euj90vYWFhGDx4MGbNmgW73Y6uXbtiz549yM/PB3D/Osdrr73m8jjdu3fH9u3bodfr\nAQBWqxUbN24Ug4w8C88AyK2Ki4sRHR0NAFCr1fDx8cHkyZPx0ksvAQDefvttLFiwAIMHD4bNZsMz\nzzyDqVOnujxuUFAQ/v73v8NqtaJx48YAgPj4eLz33nsYMGAANBoNOnfujAMHDsBut6Njx4745JNP\nMG7cOCQlJTkca9CgQViyZInDH/hHreutt97CggULMGjQIFitVrRr1w7Tp0+vVF9NnjwZffr0QXp6\nOhISEjBmzBiMGjUKKpUKOp0Oy5cvF4d4Nm/ejG+++Ubct3Xr1khLS8OQIUOQn5+P4cOHQ61Wo7i4\nGC+88AL++c9/VqomqtlUnA6aiMgzcQiIiMhDMQCIiDwUA4CIyEMxAIiIPFSNuQuooEBf6X0DA/1Q\nWGiqxmqql9LrA5RfI+urGtZXNUquLygowGmbR5wBaLUad5dQIaXXByi/RtZXNayvapRenzMeEQBE\nRFQWA4CIyEMxAIiIPBQDgIjIQzEAiIg8FAOAiMhDSRoAP/zwAxITE8tsP3LkCGJiYhAXF4fNmzdL\nWQIRETkh2YNga9aswa5du+Dr6+uw3WKxYN68edi6dSt8fX2RkJCAyMhIPPHEE1KVQlRjWax2bDt6\nFXrT4y0i87jq1NGiuLj89RCUwJPrU6uAl8KaIqRJxYv9VIZkARAcHIxly5bh/fffd9h+9epVBAcH\niysXderUCdnZ2ejTp0+FxwsM9KvSwxYVPQ2nBEqvD1B+jbWxvjM/5eNA9nUJqqGapNmTdfFi+2bV\nflzJAqBXr1747bffymw3GAwICHjwD8Hf3x8Gg8Hl8arymHVQUECVppKQmtLrA5RfY22t7/ofdwEA\nMREt8eLf/1bdZYkaNPTHn7eNkh2/qjy5PpUKCAzwqfTvd0UfPGSfC0in08FofNBRRqPRIRCI6IHS\noZ/GgX5oWK+OZN8nKNAPKqtNsuNXFeuThux3AYWEhCAvLw937tyB2WxGTk4OOnbsKHcZRDWCvsgM\nAAjw83JzJVQbyXYGkJGRAZPJhLi4OEydOhWjR4+GIAiIiYkR12wlIkelZwA6P283V0K1kaQB0KxZ\nM/E2zwEDBojbIyMjERkZKeW3JqoVDP8LAJ4BkBT4IBiRgulNZqgA6OowAKj6MQCIFExfZIG/rxfU\napW7S6FaiAFApGB6k4XDPyQZBgCRQtntAoxFFgT4MgBIGgwAIoUyFFsgAAjgHUAkEQYAkULpeQcQ\nSYwBQKRQBtP9h8B0DACSCAOASKHEMwBfDgGRNBgARAqlL+IQEEmLAUCkUHpT6TxAPAMgaTAAiBRK\nnAeIt4GSRBgARApl4BAQSYwBQKRQHAIiqTEAiBRKb7KgjrcGXlr+MyVp8DeLSKH0JjPH/0lSDAAi\nBRIEAYYiC4d/SFIMACIFKjbbYLUJvABMkmIAECnQgwvADACSDgOASIE4DQTJgQFApECcBoLkwAAg\nUiA9ZwIlGTAAiBTIIK4FwCEgkg4DgEiBHlwD4BkASYcBQKRA+iLeBUTSYwAQKZCeQ0AkAwYAkQLp\nTRZoNSrU8da4uxSqxRgARApUOg+QSqVydylUizEAiBSI8wCRHBgARApjsdpRbLbxAjBJjgFApDBc\nCIbkwgAgUhiuBUxykSwA7HY7ZsyYgbi4OCQmJiIvL8+hfdeuXRg8eDBiYmLw9ddfS1UGUY3DtYBJ\nLlqpDnzo0CGYzWakp6fjzJkzmD9/PlauXCm2f/zxx9i9ezf8/PzQr18/9OvXD/Xq1ZOqHKIag0NA\nJBfJAuDUqVMIDw8HAHTo0AHnz593aG/dujX0ej20Wi0EQeDtbkT/w2kgSC6SBYDBYIBOpxNfazQa\nWK1WaLX3v2WrVq0QExMDX19fREVFoW7duhUeLzDQD1pt5R+KCQoKqPS+clB6fYDya6wt9dn+92Go\nWZN6sv5MtaX/3EXp9ZVHsgDQ6XQwGo3ia7vdLv7xz83NxXfffYfDhw/Dz88PycnJ2Lt3L/r06eP0\neIWFpkrXEhQUgIICfaX3l5rS6wOUX2Ntqi//lgEAYDNbZfuZalP/uYOS66somCS7CBwWFobMzEwA\nwJkzZxAaGiq2BQQEoE6dOvDx8YFGo0GDBg1w7949qUohqlEezAPEISCSlmRnAFFRUcjKykJ8fDwE\nQUBqaioyMjJgMpkQFxeHuLg4DB8+HF5eXggODsbgwYOlKoWoRtGbzFAB0NVhAJC0JAsAtVqN2bNn\nO2wLCQkRv05ISEBCQoJU356oxtIXWeDv6wW1mjdGkLT4IBiRwuhNFg7/kCwYAEQKYrcLMBZZeAso\nyYIBQKQghmILBPAhMJIHA4BIQcR5gDgERDJgABApiMHEtYBJPgwAIgV5MA0Eh4BIegwAIgXRcyZQ\nkhEDgEhBSmcC5TUAkgMDgEhBDBwCIhkxAIgUhENAJCcGAJGC6HkXEMmIAUCkIHqTBT7eGnhVYe0L\nokfFACBSEAOngSAZMQCIFEIQBOhNZk4DQbJhABApRLHZBqtN4Pg/yYYBQKQQ4gVgDgGRTFwGgNls\nxsqVK/H+++/DYDBg+fLlMJvNctRG5FEe3ALKISCSh8sAmD17NoqKinDx4kVoNBr8+uuv+Oijj+So\njcijcC1gkpvLALhw4QImT54MrVYLX19fLFiwAD/++KMctRF5FE4DQXJzGQAqlQpmsxkq1f31SQsL\nC8Wviaj6cBoIkpvLReGTkpIwcuRIFBQUICUlBYcOHcK4cePkqI3Io3AaCJKbywAYNGgQ2rZti5Mn\nT8Jms2HlypVo06aNHLUReRROA0FycxkAAwcORHR0NPr374/GjRvLURORR3pwEZhDQCQPl9cAFi9e\njHv37iEpKQkjRozA9u3bYTQa5aiNyKPoTRZo1CrU8eY8QCQPlwHQqlUrTJo0Cfv378c777yDf//7\n3+jatasctRF5FEORGQF+XrzJgmTjcgjIZrPh+PHj2LNnD7Kzs9GtWzdMmzZNjtqIPIreZEFQfV93\nl0EexGUAREREoH379hg4cCDmzp0Lb2+OTxJVN4vVjmKzjReASVYuA2D37t2oX7++HLUQeSzxITDO\nA0QychoAb7zxBlatWoUhQ4Y4jEkKggCVSoXDhw/LUiCRJzBwHiByA6cBMGfOHADAhg0bZCuGyFNx\nHiByB6d3ATVq1AgAMH/+fDRt2tThP14EJqpeDx4C4xkAycfpGcC4ceOQm5uL/Px89OzZU9xus9nw\nt7/9TZbiiDyFeAbAawAkI6cBsGDBAty5cwcpKSn45z//+WAHrRYNGzZ0eWC73Y6ZM2fi0qVL8Pb2\nxty5c9GiRQux/ezZs5g/fz4EQUBQUBAWLlwIHx+fKv44RDUT5wEid3A6BKTT6dCsWTOsXLkSd+/e\nxY0bN/DHH3/g2rVr2LFjh8sDHzp0CGazGenp6ZgyZQrmz58vtgmCgOnTp2PevHnYtGkTwsPD8fvv\nv1fPT0RUAxnEqaA5BETycXkb6AcffIDvv/8ed+/eRcuWLZGbm4uwsDAMHTq0wv1OnTqF8PBwAECH\nDh1w/vx5se3atWuoX78+1q9fj8uXLyMiIgItW7as8HiBgX7Qaiv/iHxQUECl95WD0usDlF9jTa6v\nxCYAAJ5uHoh6OvecCdfk/lMCpddXHpcBkJ2djf3792POnDlISkqCIAiYPXu2ywMbDAbodDrxtUaj\ngdVqhVarRWFhIb7//nvMmDEDwcHBePPNN9G2bVt06dLF6fEKC02P+COVFRQUgIICfaX3l5rS6wOU\nX2NNr+92oQkqAMXGEpiL5F9ytab3n7spub6KgsnlXECNGjWCl5cXQkJCcOnSJbRq1eqRJoPT6XQO\n77Pb7dBq7+dN/fr10aJFC4SEhMDLywvh4eEOZwhEnkZfZIG/rxfUas4DRPJxGQCNGzfGqlWr0LFj\nR6SlpWHPnj0wmVx/Gg8LC0NmZiYA4MyZMwgNDRXbmjdvDqPRiLy8PABATk4OWrVqVdmfgajG05ss\nvABMsnM5BJSSkoKjR4+iXbt2ePnll7F7927MnDnT5YGjoqKQlZWF+Ph4CIKA1NRUZGRkwGQyIS4u\nDikpKZgyZQoEQUDHjh3x0ksvVcOPQ1Tz2O0CjEUWNGno5+5SyMO4DIBPP/0UAwcOBAAkJiYiMTHx\nkQ6sVqvLXCsICQkRv+7SpQu2bt36OLUS1UqGYgsE8A4gkp/LAHjqqaeQmpqKu3fvon///hg4cCCa\nNWsmR21EHsHAaSDITVxeA3jllVewadMmrF27Fj4+Phg3bhwSEhLkqI3II3AtYHIXlwEAAHq9HidO\nnEBWVhZsNhu6desmdV1EHuPBNBAcAiJ5uRwCevPNN3Hx4kW8/PLLePfdd9G+fXs56iLyGKXTQOh4\nBkAycxkAw4YNQ/fu3cV7+Imoehk4BERu4vSv+rJlyzB+/HgcPHgQBw8eLNM+b948SQsj8hQcAiJ3\ncRoAzz77LADg+eefl60YIk/EmUDJXZwGQGRkJAAgIyMDX3zxhWwFEXka3gVE7uLyLqCSkhLcuHFD\njlqIPJLBZIGPtwZeVZjtlqgyXF7Z/fPPPxEZGYmGDRvCx8eHi8ITVTN9kYUrgZFbuAyAtWvXylEH\nkUcSBAF6kxnNG9W8ueSp5nM5BNS0aVOcPn0amzdvRoMGDZCdnY2mTZvKURtRrVdstsFqEzj+T27h\nMgAWLVqEo0eP4sCBA7DZbNi2bZvD8o5EVHniHUAcAiI3cBkAx48fFxds1+l0WLdunTjPPxFVzYM7\ngPgMAMnPZQCo1fffolLdX6nIbDaL24ioavScCZTcyOVF4N69e2PixIm4e/cu1q9fj127dqF///5y\n1EZU65WeAeg4BERu4DIAxo4di2PHjqFJkya4ceMGxo8fjx49eshRG1GtZxCfAuYQEMnPZQD89NNP\nMBqNeOGFFxASEoLmzZvLUReRR+AQELmT0wC4ffs2JkyYgMuXL6NFixZQqVS4du0aOnbsiEWLFqFu\n3bpy1klUK3EaCHInp1dz58yZg06dOiErKwtbtmzB5s2bkZWVhdatWyM1NVXOGolqrdIzAB1nAiU3\ncBoAly5dwuTJk+Hl9eCTibe3NyZPnoyLFy/KUhxRbWcoskCjVsHXh/MAkfycBoCPj0+521UqFW8D\nJaomepMZAX5e4m3WRHJy+pe8ol9I/rISVQ+9ycI7gMhtnF4Evnz5Mnr27FlmuyAIKCgokLQoIk9g\nsdpRbLbxGQByG6cBsH//fjnrIPI4Bq4ERm7mNAA44yeRtDgPELkbr+YSuQkfAiN3YwAQuYl4BsBr\nAOQmToeA/vjjD4fXarUadevWhZ+fn+RFEXkCPecBIjdzGgCvvvoqVCoVBEEQtxUWFiI0NBRLlizh\nNQKiKuIQELmb0wA4cuRIudt37NiBWbNmYfXq1ZIVReQJDKVTQfMMgNzksa8BDBo0CDdu3HD5Prvd\njhkzZiAuLg6JiYnIy8sr933Tp0/HokWLHrcMohpPPAPgNQByE8kuAh86dAhmsxnp6emYMmVKuesI\np6Wl4aeffpKqBCJF0xdZoAIXgyH3eawAMBgMWLduHVq0aOHyvadOnUJ4eDgAoEOHDjh//rxD++nT\np/HDDz8gLi7ucUogqjX0JjP8fb2gVnNqFXIPp9cA2rRp4zDnj0qlQt26ddGlSxfMnDnT5YENBgN0\nOp34WqPRwGq1QqvVIj8/H5999hmWL1+OvXv3PlKhgYF+0GorP2NiUFBApfeVg9LrA5RfY02rz1hs\nRf0AH8XUrZQ6nGF91c9pAOTm5lbpwDqdDkajUXxtt9uh1d7/dvv27UNhYSHGjh2LgoICFBcXo2XL\nlhgyZIjT4xUWmipdS1BQAAoK9JXeX2pKrw9Qfo01rT67XYDeaEbjQF9F1F3T+k9plFxfRcFUqWsA\nY8eOdfmesLAwZGZmAgDOnDmD0NBQsS0pKQnbt2/Hhg0bMHbsWPTv37/CP/5EtY2x2AIBfAaA3Mvl\nmsDlycnJcfmeqKgoZGVlIT4+HoIgIDU1FRkZGTCZTBz3J4/HZwBICSoVAI9CrVZj9uzZDttCQkLK\nvI+f/MkTcS1gUgLOBUTkBlwLmJTA6RlAZGRkuSt/CYKA4uJiSYsiqu24FgApgdMA2LBhg5x1EHkU\nDgGREjgdAmratKn43+nTp7F582Y0aNAA2dnZnAiOqIoeTAPBISByH5fXABYtWoSjR4/iwIEDsNls\n2LZtW7nTOhDRo9NzCIgUwGUAHD9+HAsXLoSPjw90Oh3WrVsn3t9PRJVj4BAQKYDLAFCr77+l9IKw\n2WwWtxFR5ehNFvh4a+BVhelNiKrK5XMAvXv3xsSJE3H37l2sX78eu3btQr9+/eSojajW0hdZOA00\nuZ3LABg7diyOHTuGJk2a4MaNGxg/fjx69OghR21EtZIgCNCbzGjeSOf6zUQSqjAAfv75Z/j7+yM8\nPFyc2vn27duYPn065syZI0uBRLVNsdkGq03gPEDkdk4H85ctW4aYmBj07t0bJ06cgM1mw+rVqxEV\nFVVmwXgienTiHUAcAiI3c3oGsGPHDuzfvx/5+flYunQp1qxZg1u3buHTTz8VzwaI6PE9eAiMZwDk\nXk4DwN/fH40aNUKjRo1w9uxZDBo0CGvXroVGw7sWiKpCnAeIt4CSmzkNgIdv9QwMDMTUqVNlKYio\ntjNwMXhSCKfXAB6eCK5OnTqyFEPkCfRFHAIiZXB6BnD58mX07NkTAHDz5k3xa0EQoFKpcPjwYXkq\nJKpluBgMKYXTANi/f7+cdRB5jNKLwLwGQO7mNAA44yeRNAycCZQUgpP6EMlMX2SBRq2Crw/vqCP3\nYgAQyUxvMiPAz6vcFfeI5MQAIJKZ3mThWsCkCAwAIhlZrHYUm228A4gUgQFAJCMuBk9KwgAgkhHn\nASIlYQAQyUjPaSBIQRgARDJ6MA0EA4DcjwFAJKMH00BwCIjcjwFAJCPOA0RKwgAgkpGhdB4gXgMg\nBWAAEMlIXA6SQ0CkAAwAIhnpTRaowDMAUgYGAJGM9CYz/H29oFZzHiByP6fTQVeV3W7HzJkzcenS\nJXh7e2Pu3Llo0aKF2L579258+eWX0Gg0CA0NxcyZMx2WoSSqje7PA8RP/6QMkv3FPXToEMxmM9LT\n0zFlyhTMnz9fbCsuLsYnn3yCr776CmlpaTAYDPj222+lKoVIEex2AcZiC+8AIsWQ7Azg1KlTCA8P\nBwB06NAB58+fF9u8vb2RlpYGX19fAIDVaoWPj0+FxwsM9INWW/n504OCAiq9rxyUXh+g/BqVXl8d\nfx8IAvBEoJ8ia1ViTQ9jfdVPsgAwGAzQ6XTia41GA6vVCq1WC7VajSeeeAIAsGHDBphMJnTt2rXC\n4xUWmipdS1BQAAoK9JXeX2pKrw9Qfo01ob5frhcCALw1KsXVWhP6j/VVTkXBJFkA6HQ6GI1G8bXd\nbodWq3V4vXDhQly7dg3Lli3j4hhU6+n5DAApjGTXAMLCwpCZmQkAOHPmDEJDQx3aZ8yYgZKSEqxY\nsUIcCiKqzQx8BoAURrIzgKioKGRlZSE+Ph6CICA1NRUZGRkwmUxo27Yttm7dis6dO+O1114DACQl\nJSEqKkqqcojcjtNAkNJIFgBqtRqzZ8922BYSEiJ+nZubK9W3JlKkB2sBMABIGXjjPZFMHqwFwCEg\nUgYGAJFMuBwkKQ0DgEgmHAIipWEAEMlEb7LAx1sDryo80EhUnRgARDLRF1m4FjApCgOASAaCIEBv\n4jxApCwMACIZFJVYYbXZ+RAYKQoDgEgG94z/uwDMISBSEAYAkQzuGkoAADoOAZGCMACIZCCeAXAI\niBSEAUAkg7sGDgGR8jAAiGRwz3h/CIhnAKQkDAAiGZSeAfAaACkJA4BIBg+uATAASDkYAEQyuFs6\nBMSZQElBGABEMrhnMEOjVsHXh/MAkXIwAIhkcNdYAp2fF9e+JkVhABDJ4J7RzOEfUhwGAJHELFY7\nTMVWXgAmxWEAEEmMK4GRUjEAiCQmrgTGISBSGAYAkcT0PAMghWIAEEmMawGTUjEAiCSmN90/A9Bx\nHiBSGAYAkcRKA4AzgZLSMACIJMa7gEipGABEEntwDYBDQKQsDAAiiZUOAfn7at1cCZEjBgCRxPQm\nMwL8vKBR858bKQt/I4kkZiiyoK6/j7vLICqDAUAkIbsgwFBkQT0dx/9JeSQLALvdjhkzZiAuLg6J\niYnIy8tzaD9y5AhiYmIQFxeHzZs3S1UGkVsZiywQBKCuPwOAlEeyADh06BDMZjPS09MxZcoUzJ8/\nX2yzWCyYN28evvjiC2zYsAHp6em4deuWVKUQuU3pBeB6Og4BkfJIdlvCqVOnEB4eDgDo0KEDzp8/\nL7ZdvXoVwcHBqFevHgCgU6dOyM7ORp8+faq9jty8Qry79DhKLNZqP3Z1UalUEATB3WVUSOk1KrU+\nu/3+//IMgJRIsgAwGAzQ6XTia41GA6vVCq1WC4PBgICAALHN398fBoOhwuMFBvpBq3385fSMVgHB\nfwtAicX22PsSVQcvjRr/77kmCAoKcP1mN2J9VaP0+sojWQDodDoYjUbxtd1uh1arLbfNaDQ6BEJ5\nCgtNlarDX6vC/HHdUFCgr9T+cggKClB0fYDya2R9VcP6qkbJ9VUUTJJdAwgLC0NmZiYA4MyZMwgN\nDRXbQkJCkJeXhzt37sBsNiMnJwcdO3aUqhQiIiqHZGcAUVFRyMrKQnx8PARBQGpqKjIyMmAymRAX\nF4epU6di9OjREAQBMTExaNy4sVSlEBFROSQLALVajdmzZztsCwkJEb+OjIxEZGSkVN+eiIhc4INg\nREQeigFAROShGABERB6KAUBE5KEYAEREHkolKPH5eSIikhzPAIiIPBQDgIjIQzEAiIg8FAOAiMhD\nMQCIiDwUA4CIyEMxAIiIPJRks4G6y8GDB7Fv3z4sXrwYwP21CFJSUqDRaNCtWze88847Du8vLi5G\ncnIybt++DX9/fyxYsAANGjSQtMbVq1fj2LFjAIB79+7h1q1byMrKcnjP3Llzcfr0afj7+wMAVqxY\n4XLRnOoiCAK6d++Op556CsD9JT2nTJni8J7NmzcjLS0NWq0Wb731Fnr06CFLbQCg1+uRnJwMg8EA\ni8WCqVOnlllPwh39Z7fbMXPmTFy6dAne3t6YO3cuWrRoIbYfOXIEn332GbRaLWJiYjBs2DBJ6/kr\ni8WCadNZvqh4AAAHDklEQVSm4ffff4fZbMZbb72Fnj17iu3r16/Hli1bxN//WbNmoWXLlrLWOHjw\nYHElwWbNmmHevHlim7v7b/v27fjmm28AACUlJfjxxx+RlZWFunXrAlBG/z02oRaZM2eO0KtXL2Hi\nxInitoEDBwp5eXmC3W4XXn/9deHChQsO+3zxxRfC0qVLBUEQhN27dwtz5syRteaxY8cKx44dK7M9\nPj5euH37tqy1lPrll1+EN954w2l7fn6+0L9/f6GkpES4d++e+LVcPv30U2HdunWCIAjC1atXhUGD\nBpV5jzv6b//+/cIHH3wgCIIgfP/998Kbb74ptpnNZuEf//iHcOfOHaGkpEQYMmSIUFBQIGt9W7du\nFebOnSsIgiAUFhYKERERDu1TpkwRzp07J2tNDysuLhaio6PLbVNC/z1s5syZQlpamsM2d/dfZdSq\nIaCwsDDMnDlTfG0wGGA2mxEcHAyVSoVu3brhxIkTDvs8vHh99+7d8d///le2eg8cOIC6deuiW7du\nDtvtdjvy8vIwY8YMxMfHY+vWrbLVBAAXLlzAzZs3kZiYiDFjxuDnn392aD979iw6duwIb29vBAQE\nIDg4GLm5ubLVN2LECMTHxwMAbDYbfHx8HNrd1X8P/y516NAB58+fF9uuXr2K4OBg1KtXD97e3ujU\nqROys7NlqatU79698e677wK4f5an0TiusX3hwgWsXr0aCQkJWLVqlay1AUBubi6KioowatQoJCUl\n4cyZM2KbEvqv1Llz53DlyhXExcU5bHd3/1VGjRwC2rJlC7788kuHbampqejbty9OnjwpbvvrwvT+\n/v64fv26w34PL1Dv7+8Pvb561/V0Vmu7du2watUqLFmypMw+JpMJr776KkaOHAmbzYakpCS0bdsW\nbdq0qdbanNU3Y8YMjB07Fn369EFOTg6Sk5Oxbds2sf3hPgPu95vBYKj22pzVV9p/BQUFSE5OxrRp\n0xza5ey/h/31902j0cBqtUKr1craZ86UDocZDAZMmDABEydOdGjv168fhg8fDp1Oh3feeQfffvut\nrEN7derUwejRoxEbG4tffvkFY8aMwb59+xTTf6VWrVqFcePGldnu7v6rjBoZALGxsYiNjXX5vvIW\nny8dryvvPeW1S1XrlStXULduXYcx4lK+vr5ISkqCr68vAODFF19Ebm6uJH/AyquvqKhI/HTYuXNn\n5OfnQxAEqFQqAOX3q1Tj687679KlS5g8eTLef/99PP/88w5tcvbfw/7aL3a7HVqtttw2KfusIjdu\n3MC4ceMwfPhwDBgwQNwuCAJee+01saaIiAhcvHhR1j9gTz/9NFq0aAGVSoWnn34a9evXR0FBAZ58\n8knF9N+9e/dw7do1vPjiiw7bldB/lVGrhoD+SqfTwcvLC7/++isEQcDx48fRuXNnh/eEhYXh6NGj\nAIDMzEx06tRJltpOnDiB7t27l9v2yy+/ICEhATabDRaLBadPn8azzz4rS10AsHz5cvFTd25uLp58\n8knxjz8AtGvXDqdOnUJJSQn0ej2uXr2K0NBQ2eq7cuUK3n33XSxevBgRERFl2t3Vf2FhYcjMzARw\n/+aDh/skJCQEeXl5uHPnDsxmM3JycspcuJbarVu3MGrUKCQnJ2Po0KEObQaDAf3794fRaIQgCDh5\n8iTatm0ra31bt27F/PnzAQA3b96EwWBAUFAQAGX0HwBkZ2ejS5cuZbYrof8qo0aeATyOWbNm4b33\n3oPNZkO3bt3Qvn17AMCoUaPw+eefIyEhAR988AESEhLg5eUl3j0ktWvXrqFr164O29atW4fg4GD0\n7NkT0dHRGDZsGLy8vBAdHY1WrVrJUhcAjB07FsnJyTh69Cg0Go14J8bD9SUmJmL48OEQBAGTJk0q\nMw4vpcWLF8NsNiMlJQXA/aBfuXKl2/svKioKWVlZiI+PhyAISE1NRUZGBkwmE+Li4jB16lSMHj0a\ngiAgJiYGjRs3lrymh33++ee4d+8eVqxYgRUrVgC4f4ZVVFSEuLg4TJo0CUlJSfD29kaXLl3KDVcp\nDR06FB9++CESEhKgUqmQmpqKvXv3Kqb/gPv/bps1aya+fvj/X3f3X2VwOmgiIg9Vq4eAiIjIOQYA\nEZGHYgAQEXkoBgARkYdiABAReSgGABGRh2IAEBF5KAYAUSV99dVXeOWVVyAIAnJycvDyyy+7bX4a\nosrgg2BElSQIApKSktC7d29s2LABKSkpsk0lQlQdGABEVXD9+nUMGDBAnFKEqCbhEBBRFfzxxx/Q\n6XS4ePEi+FmKahoGAFElGY1GTJ8+HStWrICvry++/vprd5dE9FgYAESVtHDhQkRERKBdu3ZiEPx1\nwSEiJeM1ACIiD8UzACIiD8UAICLyUAwAIiIPxQAgIvJQDAAiIg/FACAi8lAMACIiD/X/ARknbHVT\n1kwPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xc0ec828>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Graphical representation of the ReLU derivative\n",
    "x = array(range(-10, 10))\n",
    "plt.plot(x, ReLUDerivative(x))\n",
    "plt.title(\"Derivative of ReLU\")\n",
    "plt.xlabel(\"x\")\n",
    "plt.ylabel(\"ReLU Derivative\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now demonstrate the calculations that go underneath the Backpropagation process:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An illustration of how a neural network backpropagates its error:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://matthewmazur.files.wordpress.com/2018/03/output_1_backprop-4.png?w=525)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Programmatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Backpropagation from the output layer to the hidden layer\n",
    "dl_jk = -y/a_jk + (1 - y)/(1 - a_jk) # the derivative of the cross entropy loss wrt output\n",
    "da_jk = SigmoidDerivative(z_jk) # the derivative of Sigmoid wrt the input (before activation) of the output layer\n",
    "dz_jk = a_ij # the derivative of the inputs of the hidden layer (before activation) wrt weights of the output layer  \n",
    "gradient_jk = dot(dz_jk.T , dl_jk * da_jk) # chain rule"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.1121981 ],\n",
       "       [-0.17372609],\n",
       "       [-0.17010679],\n",
       "       [-0.28592418]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gradient_jk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Backpropagation from the hidden layer to the output layer\n",
    "dl_ij = dot(da_jk * dl_jk, w_jk.T) # the derivative of the cross entropy loss wrt hidden layer input (after activation)\n",
    "da_ij = ReLUDerivative(z_ij) # the derivative of the ReLU function wrt the input (before activation) of the hidden layer\n",
    "dz_ij = i # the derivative of the inputs of the hidden layer (before activation) wrt weights of the hidden layer \n",
    "gradient_ij = dot(dz_ij.T , dl_ij * da_ij) # chain rule"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.00723859, -0.01085788, -0.02171576, -0.00361929],\n",
       "       [-0.02895435, -0.04343152, -0.08686304, -0.01447717],\n",
       "       [-0.03619293, -0.0542894 , -0.1085788 , -0.01809647]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gradient_ij"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# The new weights we get after the backpropagation process\n",
    "w_ij = w_ij - gradient_ij \n",
    "w_jk = w_jk - gradient_jk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New synaptic weights after the backpropagation process: \n",
      "w_ij:  [[ 0.10723859  0.21085788  0.42171576  0.30361929]\n",
      " [ 0.52895435  0.44343152  0.78686304  0.91447717]\n",
      " [ 0.23619293  0.6542894   0.4085788   0.81809647]]\n",
      "w_jk:  [[ 0.3121981 ]\n",
      " [ 0.47372609]\n",
      " [ 0.77010679]\n",
      " [ 0.38592418]]\n"
     ]
    }
   ],
   "source": [
    "print(\"New synaptic weights after the backpropagation process: \")\n",
    "print(\"w_ij: \", w_ij)\n",
    "print(\"w_jk: \", w_jk)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "let's now predict the output with the new updated weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "i = array([[0.1, 0.4, 0.5]])\n",
    "\n",
    "# new hidden layer weights\n",
    "w_ij = array([[ 0.10723859,  0.21085788,  0.42171576,  0.30361929],\n",
    "              [ 0.52895435,  0.44343152,  0.78686304,  0.91447717],\n",
    "              [ 0.23619293,  0.6542894,   0.4085788,   0.81809647]])\n",
    "\n",
    "# new output layer weights\n",
    "w_jk = array([[ 0.3121981 ],\n",
    "              [ 0.47372609],\n",
    "              [ 0.77010679],\n",
    "              [ 0.38592418]])\n",
    "\n",
    "z_ij = dot(i, w_ij) # calculating the dot product of the inputs and weights of the hidden layer\n",
    "a_ij = ReLU(z_ij) # applying the ReLU activation function\n",
    "\n",
    "z_jk = dot(a_ij, w_jk) # the dot product of the inputs in the hidden layer after activation and weights of the output layer\n",
    "a_jk = Sigmoid(z_jk) # applying the Sigmoid activation function "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.74992269]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the new predicted output as you will see is closer to the ground truth than the initial output before backpropagation\n",
    "a_jk "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! you've just done forward and backpropagation with just a few lines of code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Building a one hidden layer Neural Network:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "let's now build a fully functional multilayer artificial neural network, our neural network will tackle the XOR task.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://cdn-images-1.medium.com/max/800/1*WlCd6kgxqb2yQpcutLhYwQ.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the ANN's architecture will look like this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://cdn-images-1.medium.com/max/1600/1*-a-flCLHLCGM0-7TOcNJnQ.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Layer 1 (4 neurons, each with 3 inputs): \n",
      "w_ij:  [[-0.16595599  0.44064899 -0.99977125 -0.39533485]\n",
      " [-0.70648822 -0.81532281 -0.62747958 -0.30887855]\n",
      " [-0.20646505  0.07763347 -0.16161097  0.370439  ]]\n",
      "Layer 2 (1 neuron, with 4 inputs): \n",
      "w_jk:  [[-0.5910955 ]\n",
      " [ 0.75623487]\n",
      " [-0.94522481]\n",
      " [ 0.34093502]]\n",
      "loss:  2.07175185207\n",
      "loss:  0.0252235360038\n",
      "loss:  0.00812448219046\n",
      "loss:  0.00483745527192\n",
      "loss:  0.00344285541416\n",
      "loss:  0.00267203784984\n",
      "loss:  0.00218307936662\n",
      "loss:  0.0018453146627\n",
      "loss:  0.00159802272115\n",
      "loss:  0.0014091540602\n",
      "New synaptic weights after training: \n",
      "w_ij:  [[  0.15412064   5.61526375  -6.92492097 -10.55136518]\n",
      " [  0.13145897 -10.51956567  -6.93646143   5.44436653]\n",
      " [ -0.01611526  -0.75294181   0.19272139  -0.54865495]]\n",
      "w_jk:  [[-13.08177073]\n",
      " [ 15.15669436]\n",
      " [-31.14768507]\n",
      " [ 15.0867679 ]]\n",
      "Considering new situation [1, 1, 0] -> ?: \n",
      "(array([[  5.70913606e-01,   7.36004539e-03,   9.55163456e-07,\n",
      "          6.01779348e-03]]), array([[ 0.00069824]]))\n"
     ]
    }
   ],
   "source": [
    "# Import our dependencies\n",
    "from numpy import exp, array, random, dot, ones_like, where, log10\n",
    "\n",
    "# Create our Artificial Neural Network class\n",
    "class ArtificialNeuralNetwork():\n",
    "    \n",
    "    # initializing the class\n",
    "    def __init__(self):\n",
    "        \n",
    "        # generating the same synaptic weights every time the program runs\n",
    "        random.seed(1)\n",
    "        \n",
    "        # synaptic weights (3 × 4 Matrix) of the hidden layer \n",
    "        self.w_ij = 2 * random.rand(3, 4) - 1\n",
    "        \n",
    "        # synaptic weights (4 × 1 Matrix) of the output layer\n",
    "        self.w_jk = 2 * random.rand(4, 1) - 1\n",
    "    \n",
    "    def Sigmoid(self, x):\n",
    "        \n",
    "        # The Sigmoid activation function will turn every input value into probabilities between 0 and 1\n",
    "        # the probabilistic values help us assert which class x belongs to\n",
    "        \n",
    "        return 1 / (1 + exp(-x))\n",
    "    \n",
    "    def SigmoidDerivative(self, x):\n",
    "        \n",
    "        # The derivative of Sigmoid will be used to calculate the gradient during the backpropagation process\n",
    "        # and help optimize the random starting synaptic weights\n",
    "        \n",
    "        return x * (1 - x)\n",
    "    \n",
    "    def crossentropyerror(self, a, y):\n",
    "        \n",
    "        # The cross entropy loss function\n",
    "        # we use it to evaluate the performance of our model\n",
    "        \n",
    "        return - sum(y * log10(a) + (1 - y) * log10(1 - a))\n",
    "    \n",
    "    def train(self, x, y, learning_rate, iterations):\n",
    "        \n",
    "        # x: training set of data\n",
    "        # y: the actual output of the training data\n",
    "        \n",
    "        for i in range(iterations):\n",
    "            \n",
    "            z_ij = dot(x, self.w_ij) # the dot product of the weights of the hidden layer and the inputs\n",
    "            a_ij = self.Sigmoid(z_ij) # applying the Sigmoid activation function\n",
    "            \n",
    "            z_jk = dot(a_ij, self.w_jk) # the same previous process will be applied to find the predicted output\n",
    "            a_jk = self.Sigmoid(z_jk)  \n",
    "            \n",
    "            dl_jk = -y/a_jk + (1 - y)/(1 - a_jk) # the derivative of the cross entropy loss wrt output\n",
    "            da_jk = self.SigmoidDerivative(a_jk) # the derivative of Sigmoid  wrt the input (before activ.) of the output layer\n",
    "            dz_jk = a_ij # the derivative of the inputs of the hidden layer (before activation) wrt weights of the output layer\n",
    "            \n",
    "            dl_ij = dot(da_jk * dl_jk, self.w_jk.T) # the derivative of cross entropy loss wrt hidden layer input (after activ.)\n",
    "            da_ij = self.SigmoidDerivative(a_ij) # the derivative of Sigmoid wrt the inputs of the hidden layer (before activ.)\n",
    "            dz_ij = x # the derivative of the inputs of the hidden layer (before activation) wrt weights of the hidden layer\n",
    "            \n",
    "            # calculating the gradient using the chain rule\n",
    "            gradient_ij = dot(dz_ij.T , dl_ij * da_ij)\n",
    "            gradient_jk = dot(dz_jk.T , dl_jk * da_jk)\n",
    "            \n",
    "            # calculating the new optimal weights\n",
    "            self.w_ij = self.w_ij - learning_rate * gradient_ij \n",
    "            self.w_jk = self.w_jk - learning_rate * gradient_jk\n",
    "            \n",
    "            # printing the loss of our neural network after each 1000 iteration\n",
    "            if i % 1000 == 0 in range(iterations):\n",
    "                print(\"loss: \", self.crossentropyerror(a_jk, y))\n",
    "                  \n",
    "    def predict(self, inputs):\n",
    "        \n",
    "        # predicting the class of the input data after weights optimization\n",
    "        \n",
    "        output_from_layer1 = self.Sigmoid(dot(inputs, self.w_ij)) # the output of the hidden layer\n",
    "        \n",
    "        output_from_layer2 = self.Sigmoid(dot(output_from_layer1, self.w_jk)) # the output of the output layer\n",
    "        \n",
    "        return output_from_layer1, output_from_layer2\n",
    "    \n",
    "    # the function will print the initial starting weights before training\n",
    "    def SynapticWeights(self):\n",
    "        \n",
    "        print(\"Layer 1 (4 neurons, each with 3 inputs): \")\n",
    "        \n",
    "        print(\"w_ij: \", self.w_ij)\n",
    "        \n",
    "        print(\"Layer 2 (1 neuron, with 4 inputs): \")\n",
    "        \n",
    "        print(\"w_jk: \", self.w_jk)\n",
    "\n",
    "    \n",
    "def main():\n",
    "    \n",
    "    ANN = ArtificialNeuralNetwork()\n",
    "    \n",
    "    ANN.SynapticWeights()\n",
    "    \n",
    "    # the training inputs \n",
    "    # the last column is used to add non linearity to the clasification task\n",
    "    x = array([[0, 0, 1], \n",
    "               [0, 1, 1], \n",
    "               [1, 0, 1], \n",
    "               [0, 1, 0], \n",
    "               [1, 0, 0], \n",
    "               [1, 1, 1], \n",
    "               [0, 0, 0]])\n",
    "    \n",
    "    # the training outputs\n",
    "    y = array([[0, 1, 1, 1, 1, 0, 0]]).T\n",
    "\n",
    "    ANN.train(x, y, 1, 10000)\n",
    "    \n",
    "    # Printing the new synaptic weights after training\n",
    "    print(\"New synaptic weights after training: \")\n",
    "    print(\"w_ij: \", ANN.w_ij)\n",
    "    print(\"w_jk: \", ANN.w_jk)\n",
    "    \n",
    "    # Our prediction after feeding the ANN with new set of data\n",
    "    print(\"Considering new situation [1, 1, 0] -> ?: \")\n",
    "    print(ANN.predict(array([[1, 1, 0]])))\n",
    "    \n",
    "if __name__==\"__main__\":\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Vizualization of the loss function:***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0xc1c6b00>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAETCAYAAAA/NdFSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X10FOXdPvBrdjebZHdDXthNkA3h0fCj8FQEpKIYICGV\nAiXRvNmEtAGBVrFYhQMco0VqKwJR8dhqAYNaLaKCgDVQalt5KQI2EgTlpeU8UI0mUEhCErK7YV/n\n90fIJgtJCJtMZnfn+hxtdmd2Zr57S3Nxzz0ztyCKoggiIlIcldwFEBGRPBgAREQKxQAgIlIoBgAR\nkUIxAIiIFIoBQESkUAwACmhVVVUYPXq03GX4rby8HBkZGT3ax5w5c3Dx4kUAwM9+9jOcPn26N0oj\ngkbuAoioawcOHPC+Xr9+vYyVUKhhD4CCVlNTExYvXoyMjAxkZmbiueeeg8vlAgD87ne/Q2ZmJnJy\ncjB37lxcuHChy+XtORwOrFixAtnZ2bj33ntRXFwMi8WC/fv3IzMz0/u5S5cu4Y477kBjYyP27NmD\ngoIC5OTkIC0tDS+99NI1+y0uLsbrr7/e4fvOtn/iiScAALNmzcK5c+eQnp6OY8eOAQA2bdqEjIwM\n3HvvvZgzZw6++uor736XL1+OoqIiTJ48GQ899BCsVmuP25tCDwOAgtby5csRExOD7du3Y+vWrTh1\n6hTeeOMNnDt3Dm+99Ra2bt2Kbdu2ISUlBV9++WWny69WWloKtVqNbdu2oaysDPHx8XjhhReQkpIC\nq9Xq/QW8Y8cOpKamol+/fnjjjTewatUqbNu2DZs2bUJpaan3tM31iKLY6fYrV64EALz11lu46aab\nvNt8+umneO211/DHP/4RZWVlyMjIwPz589F6Y//x48fx+uuvY+fOnbhw4QI++uijnjY3hSCeAqKg\ntW/fPrz77rsQBAFarRYFBQV466238NOf/hTDhg1DdnY2Jk6ciIkTJ2LcuHHweDwdLr/a3r170dTU\nhIMHDwIAnE4n+vfvD0EQkJeXhw8++AAjRozAtm3bsGTJEgiCgHXr1mHv3r3YsWMHzpw5A1EU0dzc\n3K3v4c/2n3zyCX74wx8iLi4OAJCTk4Nnn30WVVVVAIAJEyZAq9UCAIYOHYrGxsYbaltSBgYABS2P\nx3PNe5fLBZVKhbfffhvHjh3Dp59+ihUrVuDOO+/E0qVLO11+9X6efPJJpKamAgCsVivsdjsAIDc3\nF1lZWbj//vvR1NSEO++8EzabDdnZ2bjnnnvwve99D7m5ufj4449x9WO2BEHwWeZ0OgGg29u319E6\nURS9p8AiIiI6PS5RK54CoqA1fvx4bNy4EaIowuFwYPPmzbj77rvx73//GxkZGUhOTsZDDz2EBx54\nAKdOnep0eWf7dTgc8Hg8eOqpp/Diiy8CABISEjBy5EgsW7YMeXl5AIDKykpYLBYsWLAA6enp+Oyz\nz7zbthcbG4vjx48DAC5evIiKiopuba9Wq72/2NvXuHPnTu9ppq1btyImJgaDBw/uxRamUMceAAU8\nm812zaWg7733HpYuXYrly5cjMzMTTqcTEyZMwLx586DVajFt2jTk5uZCp9MhIiICS5cuxbBhwzpc\nfrWf//znKCkpQXZ2NtxuN4YPH47i4mLv+vvvvx+PPfYY1q5dCwD4zne+g7S0NEybNg39+vVDUlIS\nhgwZgsrKSu9pGAAoKirC4sWLMWXKFCQmJmLs2LHX3T4pKQmTJ09GYWEh1qxZ491XSkoKHnjgAcya\nNQsejwdxcXF49dVXoVLx73TUfQIfB01EpEz86wIRkUIxAIiIFIoBQESkUAwAIiKFCpqrgGpqmvze\nNjZWh/p6Wy9WE9zYHr7YHm3YFr5CoT1MpqhO1ymiB6DRqOUuIaCwPXyxPdqwLXyFensoIgCIiOha\nDAAiIoViABARKRQDgIhIoRgAREQKxQAgIlIoBgARkUKFfABcaGjG2q1foNnuuv6HiYgUJOQD4OTX\nF7Hz4Nc4+n+1cpdCRBRQQj4ABsTqAABVtRaZKyEiCiwhHwBmkx4AUF1jlbkSIqLAEvIBEKXTIjYq\nHNU17AEQEbUnSQA4nU4sWbIEhYWFyMvLw65du3zW7969G7m5ucjPz8fmzZulKMHH4AH9UHfJzoFg\nIqJ2JHkcdFlZGWJiYvD888+joaEBWVlZ+P73vw+gJRxWrlyJLVu2IDIyEjNmzEB6ejqMRqMUpQAA\nBt/UD0f/rwbVtVYMMUdLdhwiomAiSQ9g6tSpeOyxxwAAoihCrW57pOqZM2eQlJSE6OhoaLVajBkz\nBocOHZKiDK/BA1qeh13F00BERF6S9AD0+paBV4vFgkcffRQLFizwrrNYLIiKivL5rMVy/V/MsbE6\nv5/NPbi55dTPRYujy8kRlITt4Ivt0YZt4SuU20OyGcHOnTuH+fPno7CwEJmZmd7lBoMBVmvbFTlW\nq9UnEDrTk1l5khJa9n/6m/oezSwWKkymKLZDO2yPNmwLX6HQHn0+I1htbS3mzJmDJUuWIC8vz2dd\ncnIyKisr0dDQAIfDgYqKCowePVqKMrwiwjUwxUSgupaXghIRtZKkB7Bu3TpcunQJa9aswZo1awAA\n999/P5qbm5Gfn4/i4mLMnTsXoigiNzcXCQkJUpThw2w04OjpWjRaHYjWayU/HhFRoBNEURTlLqI7\netINM5misG7LUfz500osLhiF//2fuF6sLPiEQre2N7E92rAtfIVCeyh+UngASDQZAPCOYCKiVooJ\ngNZHQvBSUCKiFooJgAFxOqhVAgeCiYiuUEwAaNQqDOivQ3WtFZ7gGPYgIpKUYgIAAMxGPewON+oa\nL8tdChGR7JQVABwIJiLyUlQAJLbODcDJYYiIlBUArT2AKvYAiIiUFQDG6AiEh6k5OQwRERQWACpB\nwECjHufqbHC5PXKXQ0QkK0UFANByQ5jbI+J8fbPcpRARyUpxAZBobJ0knqeBiEjZFBcA5ngOBBMR\nAQoMAPYAiIhaKC4A+um1MESG8WYwIlI8xQWAIAgwG/WoaWiG3eGWuxwiItkoLgCAlrkBRABn69gL\nICLlUmQAcG4AIiKFBwDHAYhIyZQZAMYrTwXl5DBEpGCKDABdhAZx/cJ5CoiIFE2RAQC09AIaLQ5Y\nmp1yl0JEJAvFBoB3bgD2AohIoRQbAN6BYI4DEJFCKTcAjHwmEBEpm2IDYKBRB0HgKSAiUi7FBkCY\nRo2EWB2qa6wQRVHucoiI+pxiAwBoGQew2V2ob7LLXQoRUZ9TdgAYORBMRMql6ABINF25I5gDwUSk\nQIoOAD4UjoiUTNEBEB8bCY1axR4AESmSogNArVJhoFGHs3VWeDy8EoiIlEXRAQC03BDmdHlwoaFZ\n7lKIiPqU4gOAzwQiIqVSfACYeSUQESmU4gOgtQdQxXsBiEhhFB8AsVHhiAzX8BQQESmO4gNAEASY\nTXqcv9gMp8stdzlERH1G8QEAAIlGPTyiiHN1NrlLISLqM5IGwBdffIGioqJrlr/55puYPn06ioqK\nUFRUhP/85z9SlnFdHAgmIiXSSLXj9evXo6ysDJGRkdesO378OEpKSnDrrbdKdfgb0vpQuKpajgMQ\nkXJI1gNISkrCyy+/3OG6EydOoLS0FDNmzMCrr74qVQnd5p0ekj0AIlIQyXoAU6ZMQVVVVYfrpk+f\njsLCQhgMBjzyyCPYs2cPJk2a1OX+YmN10GjUftdjMkV1vg4tVwOdu2jr8nOhRCnfs7vYHm3YFr5C\nuT0kC4DOiKKIWbNmISqqpVFTU1Nx8uTJ6wZAfb3/A7QmUxRqapq6/MxN/XU4+XU9vqmqR2R4nzdL\nn+pOeygJ26MN28JXKLRHVwHW51cBWSwWZGRkwGptmYqxvLw8IMYCvHMD8IYwIlKIPvur7vbt22Gz\n2ZCfn4+FCxdi5syZ0Gq1GDduHFJTU/uqjE55B4JrLBhijpa5GiIi6UkaAImJidi8eTMAIDMz07s8\nKysLWVlZUh76hvFSUCJSGt4IdoV3fmA+EoKIFIIBcEW4Vg1TTATHAIhIMRgA7ZiNBjTZnGi0OuQu\nhYhIcgyAdhLjeRqIiJSDAdCO2ciBYCJSDgZAO62PhKhiD4CIFIAB0M6AOB3UKoEDwUSkCAyAdjRq\nFQb016G61gqPKMpdDhGRpBgAVzEb9bA73KhrvCx3KUREkmIAXIV3BBORUjAArpLYOjcAJ4chohDH\nALhKaw+gij0AIgpxDICrGKMjEB6m5s1gRBTyGABXUQkCBhr1OFdng8vtkbscIiLJMAA6YDbp4faI\nOF/fLHcpRESSYQB0IJGPhiYiBWAAdMAcz4FgIgp9DIAOsAdARErAAOhAP70Whsgw3gxGRCGNAdAB\nQRCQaNKjpqEZdodb7nKIiCTBAOiE2WiACOBsHXsBRBSaGACd4NwARBTqGACdaA0AjgMQUahiAHTC\nOz0kJ4chohDFAOiELkKDuH7hPAVERCGLAdAFs9GARosDlman3KUQEfU6BkAXvHMDsBdARCGIAdAF\n70AwxwGIKAQxALrQOhDMZwIRUShiAHRhoFEHQeApICIKTd0OgAsXLgAAKioqsHHjRthsNsmKChRh\nGjUSYnWorrFCFEW5yyEi6lXdCoBf/epXWLt2LU6fPo1FixbhxIkTePzxx6WuLSCYTXrY7C7UN9nl\nLoWIqFd1KwCOHTuGZcuW4S9/+Qvy8vKwYsUKnD17VuraAkKiiTeEEVFo6lYAuN1ueDwe7Nq1CxMn\nTkRzczOam5UxXaLZyEdCEFFo6lYAZGVlYfz48TCbzRg5ciRycnKQn58vdW0BgQ+FI6JQpenOh2bP\nno2ZM2dCrVYDAN555x3ExsZKWligiI+NhEatYg+AiEJOt3oAe/bswYsvvgir1Ypp06Zh6tSp2Lhx\no9S1BQS1SoWBRh3O1lnh8fBKICIKHd0KgFdeeQU5OTnYuXMnbrvtNuzevRtbt26VuraAYTYa4HR5\ncKFBGeMeRKQM3b4PIDk5GXv37kV6ejr0ej2cTuU8II3PBCKiUNStADAajXjmmWdw7NgxTJgwAatW\nrcLAgQOlri1gmFsvBeU4ABGFkG4FwOrVqzFixAi8/fbb0Ol0GDRoEFavXn3d7b744gsUFRVds3z3\n7t3Izc1Ffn4+Nm/efONV97HWHkAV7wUgohDSrauA9Ho9rFYrXnjhBbhcLtx5553Q6XRdbrN+/XqU\nlZUhMjLSZ7nT6cTKlSuxZcsWREZGYsaMGUhPT4fRaPT/W0gsNiockeEangIiopDSrR7Ac889hwMH\nDuC+++5DTk4OysvLsXLlyi63SUpKwssvv3zN8jNnziApKQnR0dHQarUYM2YMDh065F/1fUQQBJhN\nepy/2Aynyy13OUREvaJbPYADBw7gT3/6E1SqlrxIS0tDZmZml9tMmTIFVVVV1yy3WCyIioryvtfr\n9bBYrv8369hYHTQadXfK7ZDJFHX9D3VhyKBYnK5qhF0UMLCH+woEPW2PUMP2aMO28BXK7dGtAHC7\n3XC5XNBqtd73rTeF3SiDwQCrte1cutVq9QmEztTX+//0UZMpCjU1TX5vDwD9DS3f/dipCzCEBfdT\ntHujPUIJ26MN28JXKLRHVwHWrQDIzMzEzJkzMX36dADAn//8Z2RkZPhVTHJyMiorK9HQ0ACdToeK\nigrMnTvXr331pbaBYI4DEFFo6FYAzJs3D8OHD8c///lPiKKIefPmYe/evTd0oO3bt8NmsyE/Px/F\nxcWYO3cuRFFEbm4uEhIS/Km9Tw3kQ+GIKMQIop8zndx+++34/PPPe7ueTvWkG9Zb3biFL++HRi3g\n+Z+n9HhfcgqFbm1vYnu0YVv4CoX26OoUkN8ns5U4Q1aiSY+6S3Y0211yl0JE1GN+B4AgCL1ZR1Aw\nc3IYIgohXY4BFBUVdfiLXhRF2O3KmyKxdXKYqhoLhpijZa6GiKhnugyAX/ziF31VR1DgM4GIKJR0\nGQBjx47tqzqCQtv0kLwUlIiCX3Df0dTHwrVqmGIiOAZARCGBAXCDzEYDmmxONFodcpdCRNQjDIAb\nlBjP00BEFBoYADfIbORAMBGFBgbADTKb2i4FJSIKZgyAGzQgTge1SuBAMBEFPQbADdKoVRjQX4fq\nWis8CnwcBhGFDgaAH8xGPewON+oaL8tdChGR3xgAfkjkHcFEFAIYAH5oHQiu5uQwRBTEGAB+aH0m\nUBV7AEQUxBgAfjBGRyA8TM2bwYgoqDEA/KASBAw06nGuzgaX2yN3OUREfmEA+Mls0sPtEXG+vlnu\nUoiI/MIA8FMiHw1NREGOAeAnczwHgokouDEA/MQeABEFOwaAn/rptTBEhvFmMCIKWgwAPwmCgEST\nHjUNzbA73HKXQ0R0wxgAPWA2GiACOFvHXgARBR8GQA9wbgAiCmYMgB7gQ+GIKJgxAHpgYOuVQJwc\nhoiCEAOgB3QRGsT1C+cpICIKSgyAHjIbDWi0OGBpdspdChHRDWEA9FCiiTeEEVFwYgD0UNvkMBwH\nIKLgwgDoIbORzwQiouDEAOihgUYdBIGngIgo+DAAeihMo0ZCrA7VNVaIoih3OURE3cYA6AVmkx42\nuwv1TXa5SyEi6jYGQC/w3hHMgWAiCiIMgF5g9s4NwAAgouDBAOgFfCgcEQUjBkAvSIjVQaNWsQdA\nREFFI9WOPR4Pnn76aZw6dQparRbLly/H4MGDvevffPNNvP/++4iLiwMA/PrXv8Ytt9wiVTmSUqkE\nDDTqcLbOCo9HhEolyF0SEdF1SRYAH3/8MRwOBzZt2oSjR49i1apVWLt2rXf98ePHUVJSgltvvVWq\nEvqU2WjAN+ctuNDQjAFxOrnLISK6LskC4PDhw5gwYQIAYNSoUTh+/LjP+hMnTqC0tBQ1NTVIS0vD\nQw891OX+YmN10GjUftdjMkX5vW13DLs5Dp+e+C+a7G6MkPhYvUHq9gg2bI82bAtfodwekgWAxWKB\nwWDwvler1XC5XNBoWg45ffp0FBYWwmAw4JFHHsGePXswadKkTvdXX2/zuxaTKQo1NU1+b98d0ZFh\nAIB/nanF/7spsP/A9EV7BBO2Rxu2ha9QaI+uAkyyQWCDwQCrtW1Q1OPxeH/5i6KIWbNmIS4uDlqt\nFqmpqTh58qRUpfSJ1qeCVvFeACIKEpIFwO233459+/YBAI4ePYqhQ4d611ksFmRkZMBqbXl8Qnl5\nedCPBcRGhSMyXMNnAhFR0JDsFNDkyZNx4MABFBQUQBRFrFixAtu3b4fNZkN+fj4WLlyImTNnQqvV\nYty4cUhNTZWqlD4hCALMJj3+U30JTpcbYT0YryAi6guSBYBKpcJvfvMbn2XJycne11lZWcjKypLq\n8LJINOpxuqoR5+psSEoI7HEAIiLeCNaLzK3PBOINYUQUBBgAvahtIJjjAEQU+BgAvWggHwpHREGE\nAdCLonRaROu1vBKIiIICA6CXJZr0qLtkR7PdJXcpRERdYgD0MjMnhyGiIMEA6GWtk8NwbgAiCnQM\ngF6WGM9LQYkoODAAetnA/q1XArEHQESBjQHQy8K1aphiIjgGQEQBjwEgAbPRgCabE41Wh9ylEBF1\nigEggcR4ngYiosDHAJCA2ciBYCIKfAwACZhNvBSUiAIfA0ACA+J0UKsEDgQTUUBjAEhAo1ZhQH8d\nqmut8Iii3OUQEXWIASARs1EPu8ONusbLcpdCRNQhBoBEEjk5DBEFOAaARFoHgqs5OQwRBSgGgERa\nnwpaxR4AEQUoBoBEjNERCA9T82YwIgpYDACJqAQBA416nKuzweX2yF0OEdE1GAASMpv0cHtEnK9v\nlrsUIqJrMAAklGjkM4GIKHAxACRkjudAMBEFLgaAhNgDIKJAxgCQUD+9FobIMN4MRkQBiQEgIUEQ\nkGjSo6ahGXaHW+5yiIh8MAAkZjYaIAI4W8deABEFFgaAxDg3ABEFKgaAxPhQOCIKVAwAiQ1svRKI\nk8MQUYBhAEhMF6FBXL9wngIiooDDAOgDiSYDGi0OWJqdcpdCROTFAOgDZt4QRkQBiAHQB9omh+E4\nABEFDgZAH0jk5DBEFIAYAH3gpv46CAJPARFRYNHIXYAShGnUSIjVoarGitPVjYgIUyNCq0a4tuWn\nRq2CIAhyl0lECiNZAHg8Hjz99NM4deoUtFotli9fjsGDB3vX7969G7///e+h0WiQm5uLH/3oR1KV\nEhAGxRvw339fwIoNh69Zp1YJCA9rC4SWfzUID2t7H97Fsgit+prlahU7d0TUNckC4OOPP4bD4cCm\nTZtw9OhRrFq1CmvXrgUAOJ1OrFy5Elu2bEFkZCRmzJiB9PR0GI1GqcqRXV5aMgbFG9DscMHucOOy\nw93y0+nG5XbLmmxO1DZehtPVs2kkwzSqjsMiTA2DPhwOhwsqoeWBdSrVlZ+CAEHwfa0SBAgq+L4X\nBO+2ggCoVAIEXFmmumrbdtt4P9t+2ZV6WzpAV47f8hJCy/9A6M76Kztp+yy8vSrhqs+2btv6ucbL\nbtQ3WNFWTfuartqXd0H7177rrred98fV2/kcvP3Ltu/Rkc56jz51dLCwo+NpLXY02Rzd2n9nx7pm\n3x0s6U6Ht6PPXP3fqIsDdrq482Nfu8Lp8vg1pWtX36/T79BxCQBa/v8iBckC4PDhw5gwYQIAYNSo\nUTh+/Lh33ZkzZ5CUlITo6GgAwJgxY3Do0CFMmzZNqnJkZ4qJRMbd/9Ptz7s9nragcLb89IaGw9US\nHPaWde2XtW7Ttp0L9U12XHa44faI0n1BIpKEAOBH6UMwZWxSr+9bsgCwWCwwGAze92q1Gi6XCxqN\nBhaLBVFRUd51er0eFkvXA6SxsTpoNGq/6zGZoq7/oRDndLnRbHfD6XLD4wE8oghRFOERRXg8IkQR\n8HjEK8vbXntEEeKVz3u38YjwiPC+FkV0sp/W/bc/Hq58ruWzIkRc+QeiCABXlrd/jZZj4cpnrtlG\nFL2vfde128Zn+7b38B73yrYtL1pftX2u3Q/vvrw78PnR8b67OF67VT6uOc5Vb9pvJ/qsuPalz/E7\n25/vmk7r6qjG7m5z9fqrj9nd43Z2/Lb9dndhx/vosoQuVnb0fdqO09VOOyYIwP8mmyT5HSZZABgM\nBlitbZc9ejweaDSaDtdZrVafQOhIfb3N71pMpijU1DT5vX2oaW2P1lECNdB2HkXlPaGiGPzz0YZt\n4SuQ2sPfOroKDslGCm+//Xbs27cPAHD06FEMHTrUuy45ORmVlZVoaGiAw+FARUUFRo8eLVUpRETU\nAcl6AJMnT8aBAwdQUFAAURSxYsUKbN++HTabDfn5+SguLsbcuXMhiiJyc3ORkJAgVSlERNQBQezs\nBFqA6Uk3LJC6cYGA7eGL7dGGbeErFNpDllNAREQU2BgAREQKxQAgIlIoBgARkUIxAIiIFCporgIi\nIqLexR4AEZFCMQCIiBSKAUBEpFAMACIihWIAEBEpFAOAiEihGABERAoV0gHg8XiwbNky5Ofno6io\nCJWVlXKXJCun04klS5agsLAQeXl52LVrl9wlya6urg6pqak4c+aM3KXI7tVXX0V+fj5ycnLw/vvv\ny12OrJxOJxYtWoSCggIUFhaG7J+PkA6A9hPTL1q0CKtWrZK7JFmVlZUhJiYG77zzDl577TU888wz\ncpckK6fTiWXLliEiIkLuUmRXXl6OI0eO4N1338WGDRvw3//+V+6SZPWPf/wDLpcL7733HubPn4+X\nXnpJ7pIkEdIB0NXE9Eo0depUPPbYYwBa5kBVq/2fYzkUlJSUoKCgAPHx8XKXIrv9+/dj6NChmD9/\nPubNm4e0tDS5S5LVzTffDLfbDY/HA4vF4p3ONtSE5re6oquJ6ZVIr9cDaGmXRx99FAsWLJC5Ivls\n27YNcXFxmDBhAkpLS+UuR3b19fU4e/Ys1q1bh6qqKjz88MP46KOPIAjKmh+6lU6nQ3V1NaZNm4b6\n+nqsW7dO7pIkEdI9gK4mpleqc+fOYebMmbjvvvuQmZkpdzmy2bp1Kw4ePIiioiL861//wuOPP46a\nmhq5y5JNTEwMxo8fD61Wi1tuuQXh4eG4ePGi3GXJ5s0338T48ePx17/+FR9++CGKi4tht9vlLqvX\nhXQAdDUxvRLV1tZizpw5WLJkCfLy8uQuR1YbN27E22+/jQ0bNmD48OEoKSmByWSSuyzZjBkzBp98\n8glEUcT58+fR3NyMmJgYucuSTb9+/RAV1TKVYnR0NFwuF9xut8xV9b6Q/utwRxPTK9m6detw6dIl\nrFmzBmvWrAEArF+/noOghEmTJuHQoUPIy8uDKIpYtmyZoseIHnjgATz55JMoLCyE0+nEwoULodPp\n5C6r1/Fx0EREChXSp4CIiKhzDAAiIoViABARKRQDgIhIoRgAREQKxQAgRZo9ezb+/ve/e9+XlJRg\n9OjRcDgc3mXjx4/Ht99+2+H2v/zlL3Hs2LFO919VVYX09PQO1z3xxBOorq72s3Ki3sMAIEUaN24c\nPv/8c+/7gwcPYuTIkTh8+DAAoLKyEjqdDoMGDepw+2effRYjRozw69jl5eXg1dcUCBgApEh33XUX\njhw5AgA4f/48tFotpk6div379wMAKioqcPfdd+PLL7/EjBkzkJ2djTlz5nh7BEVFRSgvLwcArF69\nGj/4wQ+Qn5+PRx55BNu2bQMAXL58GQsXLkRGRgYKCwtRX1+P0tJSXLhwAQ8++CDq6+tRUlKCe++9\nF9nZ2XjllVdkaAlSMgYAKdJ3v/tdfPPNN7Db7di/fz9SUlKQkpLiEwBjx47F0qVLsXr1anzwwQeY\nPXs2nnrqKZ/97N69G4cPH8aOHTtQWlqKkydPetddvHgRs2fPxo4dO2A0GrFz5048+OCDiI+PR2lp\nKWw2G/bt24eysjK89957+Prrr0PyeTMUuEL6URBEnVGr1Rg5ciSOHTuG/fv348c//jEGDRqEy5cv\no7GxEUeOHMGsWbPw7bff4uGHH/ZuZ7FYfPZz8OBBTJs2DVqtFlqtFvfcc493XXx8PG677TYAwJAh\nQ1BfX++zbUJCAsLDw1FQUIBJkyZhwYIFCA8Pl/BbE/liAJBitY4DfPnll3j++ee9y3bt2uV9EFpi\nYiI+/PAQlVzVAAABeElEQVRDAIDb7UZtba3PPlQqFTweT4f7b//kWUEQrjnvr9Fo8P777+Ozzz7D\nvn37UFBQgA0bNuDmm2/ute9I1BWeAiLFuuuuu/Dhhx9i6NCh3l/WKSkp+MMf/oCUlBTccsstaGxs\nREVFBYCWR0gvXrzYZx8pKSn429/+BofDAYvFgr179173GfpqtRputxsnT57ET37yE9xxxx14/PHH\nkZycjK+++kqaL0vUAfYASLGGDh2KhoYGFBYWepfdddddWLBgAVJSUqDVavHb3/4Wzz77LOx2OwwG\nA0pKSnz2kZqais8//xzZ2dmIjo5GfHz8dU/jpKWl4cEHH8Rrr72GUaNGISMjA5GRkRg+fDgmTpwo\nyXcl6gifBkrUA0eOHMHXX3+N7OxsOJ1O5OfnY8WKFRg2bJjcpRFdFwOAqAcaGhqwaNEi1NTUQBRF\nZGVlYe7cuXKXRdQtDAAiIoXiIDARkUIxAIiIFIoBQESkUAwAIiKFYgAQESnU/wcOAAWx92VzGgAA\nAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xbf5a898>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualizing the loss function\n",
    "weights = array(range(10)) # the weights after each 1000 iteration\n",
    "loss = array([2.0717518, 0.0252235, 0.0081244, 0.0048374, 0.0034428, 0.0026720, 0.0021830, 0.0018453, 0.0015980, 0.0014091])\n",
    "plt.plot(weights, loss)\n",
    "plt.title(\"Loss evaluation\")\n",
    "plt.xlabel(\"Weights\")\n",
    "plt.ylabel(\"Loss\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the appearance of Big Data, more Computational power and more challenging and mathematically intense tasks,  Neural Networks will be used more than ever."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- https://en.wikipedia.org/wiki/Artificial_neural_network\n",
    "\n",
    "- https://en.wikipedia.org/wiki/Activation_function\n",
    "\n",
    "- https://en.wikipedia.org/wiki/Sigmoid_function\n",
    "\n",
    "- https://theclevermachine.wordpress.com/tag/tanh-function/\n",
    "\n",
    "- http://dataaspirant.com/2017/03/07/difference-between-softmax-function-and-sigmoid-function/\n",
    "\n",
    "- https://github.com/Kulbear/deep-learning-nano-foundation/wiki/ReLU-and-Softmax-Activation-Functions\n",
    "\n",
    "- http://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html\n",
    "\n",
    "- https://en.wikipedia.org/wiki/Backpropagation"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}