Chap10-Introduction_to_ANN/Chap10-Introduction_to_ANN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chap10 - Introduction to Artificial Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SetUp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "# 일관된 출력을 위해 유사난수 초기화\n",
    "def reset_graph(seed=42):\n",
    "    tf.reset_default_graph()\n",
    "    tf.set_random_seed(seed)\n",
    "    np.random.seed(seed)\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sn\n",
    "sn.set()\n",
    "plt.rcParams['axes.labelsize'] = 14\n",
    "plt.rcParams['xtick.labelsize'] = 12\n",
    "plt.rcParams['ytick.labelsize'] = 12\n",
    "\n",
    "# 한글출력\n",
    "# matplotlib.rc('font', family='AppleGothic')  # MacOS\n",
    "matplotlib.rc('font', family='Malgun Gothic')  # Windows\n",
    "plt.rcParams['axes.unicode_minus'] = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10.1 생물학적 뉴런에서 인공 뉴런까지\n",
    "\n",
    "**인공 신경망**(ANN, Aritificial Neural Networks)은 1943년 신경생리학자 Warren McCulloch과 수학자 Walter Pitts가 ['A Logical Calculus of Ideas Immanent In Nervous Activity'](https://pdfs.semanticscholar.org/5272/8a99829792c3272043842455f3a110e841b1.pdf) 처은 소개했으며, 명제 논리(propositional logic)를 사용해 동물 뇌의 생물학적 뉴런이 복잡한 계산을 위해 어떻게 상호작용하는지에 대해 간단한 계산 모델을 제시했다. \n",
    "\n",
    "1960년대까지는 이렇게 등장한 인공 신경망을 통해 사람들은 지능을 가진 기계와 대화를 나눌 수 있을 것이라고 생각했다. 하지만 아래 그림(출처: [beamandrew's blog](https://beamandrew.github.io/deeplearning/2017/02/23/deep_learning_101_part1.html))처럼 사람들의 기대와는 달리 인공 신경망으로 `XOR`문제를 해결할 수 없게 되었고, 1990년 대에는 SVM과 성능이 좋은 다른 머신러닝 알고리즘들이 나오게 되면서 인공 신경망은 암흑기로 접어 들게 되었다.\n",
    "\n",
    " \n",
    "\n",
    "![](./images/history.jpg)\n",
    "\n",
    "\n",
    "\n",
    "2000년 대에 들어서면서 인공 신경망은 2012년 ILSVRC2012 대회에서 인공 신경망을 깊게 쌓은 딥러닝 모델인 AlexNet이 압도적인 성적으로 우승하면서 다시금 주목받게 되었다. 이렇게 인공 신경망(딥러닝)이 다시 주목받게 된 계기는 다음과 같은 것들이 있다.\n",
    "\n",
    "- 빅 데이터 시대인 요즘 신경망을 학습시키기 위한 데이터가 엄청나게 많아 졌다.\n",
    "- 신경망은 다른 머신러닝 알고리즘보다 규모가 크고 복잡한 문제에서 성능이 좋다.\n",
    "- 1990년대 이후 크게 발전된 컴퓨터 하드웨어 성능과 Matrix연산에 고성능인 GPU로 인해 상대적으로 짧은 시간 안에 대규모의 신경망을 학습시킬 수 있게 되었다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.1.1 생물학적 뉴런\n",
    "\n",
    "인공 뉴런에 대해 살펴보기 전에 생물학적 뉴런을 먼저 살펴보도록 하자. 아래의 그림(출처: [wikipedia](https://en.wikipedia.org/wiki/Neuron))은 사람의 뇌에서 볼 수 있는 세포로써 핵(nucleus)을 포함하는 **세포체**(cell body)와 구성 요소로 이루어져 있다.\n",
    "\n",
    "- **Dendrite** : 수상돌기, 다른 뉴런으로부터 신호를 수용하는 부분\n",
    "- **Axon** : 축삭돌기, 신호를 내보내는 부분\n",
    "- **Synaptic terminals** : 시냅스(synapse) 뉴런의 접합부, 다른 뉴런으로 부터 짧은 전기 자극 **신호**(signal)를 받음\n",
    "\n",
    "![](./images/neuron.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.1.2 뉴런을 사용한 논리 연산\n",
    "\n",
    "처음 인공 신경망을 제안한  Warren McCulloch과 Walter Pitts의 인공 뉴런 모델은 아래의 그림(출처: 핸즈온-머신러닝)과 같다. 이 모델은 하나 이상의 이진(`on/off`) 입력과 하나의 출력을 가진다. \n",
    "\n",
    "![](./images/mcp.png)\n",
    "\n",
    "\n",
    "\n",
    "- ①은 항등함수를 의미하며, 뉴런 `A`가 활성화 되면 뉴런 `C` 또한 활성화된다.\n",
    "- ②는 논리곱 연산을 의미하며, 뉴런 `A`와 `B`가 모두 활성화될 때만 뉴런 `C`가 활성화된다. \n",
    "- ③은 논리합 연산을 의미하며, 뉴런 `A`와 `B` 둘 중 하나라도 활성화되면 `C`가 활성화된다.\n",
    "- ④는 어떤 입력이 뉴런의 활성화를 억제할 수 있다고 가정하면, 위의 그림에서 처럼 뉴런 `A`가 활성화 되고 `B`가 비활성화될 때 뉴런 `C`가 활성화된다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.1.3 퍼셉트론\n",
    "\n",
    "**퍼셉트론**(Perceptron)은 Frank Rosenblatt가 1975년에 제안한 인공 신경망 구조 중 하나이며, 이 퍼셉트론이 바로 신경망(딥러닝)의 기원이 되는 알고리즘이라고 할 수 있다. 퍼셉트론에 대한 자세한 내용은 [여기](http://excelsior-cjh.tistory.com/169?category=940400)서 확인할 수 있다.\n",
    "\n",
    "퍼셉트론은 **TLU**(Threshold Logic Unit)이라는 형태의 뉴런을 기반으로 하며, 아래의 그림과 같이 입력과 출력이 어떤 숫자고 각각의 입력에 각각 고유한 **가중치**($\\mathbf{W}$, weight)가 곱해진다.\n",
    "$$\n",
    "z = w_1x_1 + w_2x_2 + \\cdots + w_nx_n = \\mathbf{W}^{T} \\cdot \\mathbf{X}\n",
    "$$\n",
    "그런 다음 계산된 합 $z$에 **계단 함수**(step function)를 적용하여 결과 $h$를 출력한다.\n",
    "$$\n",
    "\\begin{align*}\n",
    "h_w(\\mathbf{X}) &= \\text{step}(z) \\\\ &= \\text{step}(\\mathbf{W}^{T} \\cdot \\mathbf{X})\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "\n",
    "\n",
    "![](./images/perceptron02.png)\n",
    "\n",
    "\n",
    "\n",
    "퍼셉트론에서 가장 널리 사용되는 계단 함수는 **헤비사이드 계단 함수**(Heaviside step function)이며, 단위 계단 함수(unit step function)라고도 한다. 간혹 계단 함수 대신 부호 함수(sign function)을 사용하기도 한다.\n",
    "$$\n",
    "\\text{heaviside}(z)=\\begin{cases} 0 \\text{  (}z < 0)  \\\\ 1 \\text{  (}z \\ge 0) \\end{cases}, \\quad \\text{sgn}(z)= \\begin{cases} -1 \\text{  (}z < 0)\\\\ 0 \\text{ (}z = 0)\\\\ +1 \\text{  (}z > 0)\\end{cases}\n",
    "$$\n",
    "\n",
    "\n",
    "\n",
    "위의 그림에서 처럼, 하나의 TLU는 입력과 가중치의 선형결합인 $z$가 $z \\ge 0$ 이면 양성 클래스, $z < 0$이면 음성 클래스를 출력하게 하는 **선형 이진 분류**(linear binary classification)에 적용할 수 있다.  이러한 TLU를 학습시킨다는 것은 분류를 잘할 수 있는 최적의 매개변수 $w_1, w_2, \\dots, w_n$을 찾는다는 뜻이다.\n",
    "\n",
    "아래의 그림은 위의 TLU를 세 개로 구성한 퍼셉트론에 편향($x_0=1$)을 추가한 것이다. 이 퍼셉트론은 샘플 세 개의 클래스(레이블)로 분류할 수 있는 **Multioutput Classifier**이다.\n",
    "\n",
    "\n",
    "\n",
    "![](./images/multi-tlu.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 퍼셉트론 학습\n",
    "\n",
    "Frank Rosenblatt가 제안한 퍼셉트론의 학습 알고리즘은 **헤브의 규칙**(Hebb's rule)로 부터 영감을 받았다. Donald Hebb는 1949년에 출간한 책 'The Organization of Behavior'에서 뉴런이 다른 뉴런을 활성화시킬 때 이 두 뉴런의 연결이 강해진다고 주장했다. 즉, **두 뉴런이 동일한 출력을 낼 때마다 이 둘 사이의 연결 가중치가 증가**하며 이러한 규칙을 헤브의 규칙 또는 **헤브 학습**(Hebbian Learning)이라고 한다.  퍼셉트론은 네트워크가 만드는 에러를 반영하도록 학습되며 잘못된 출력을 만드는 연결은 올바른 출력을 만들 수 있도록 가중치를 조정한다.\n",
    "\n",
    "예를 들어 하나의 데이터 샘플이 퍼셉트론에 입력되면 각 샘플에 대해 출력(예측)이 만들어지고, 잘못된  예측을 하는 모든 출력 뉴런에 대해 올바른 예측을 만들 수 있도록 아래의 식과 같이 입력에 연결된 가중치를 강화한다.\n",
    "\n",
    "\n",
    "$$\n",
    "w_{i,j}^{\\text{(next step)}} = w_{i,j} + \\eta \\left( y_j - \\hat{y}_j \\right) x_i\n",
    "$$\n",
    "\n",
    "- $w_{i,j}$ : $i$-번째 입력 뉴런과 $j$-번째 출력 뉴런 사이를 연결하는 가중치\n",
    "- $x_i$ : 현재 학습 데이터 샘플의 $i$-번째 뉴런의 입력값\n",
    "- $\\hat{y}_j$ : 현재 학습 데이터 샘플의 $j$-번째 출력 뉴런의 출력값\n",
    "- $y_j$ : 현재 학습 데이터 샘플의 $j$-번째 출력 뉴런의 실제값\n",
    "- $\\eta$ : 학습률, learning rate\n",
    "\n",
    " \n",
    "\n",
    "Frank Rosenblatt는 학습 데이터가 선형적으로 구분될 수 있으면, 퍼셉트론은 정답에 수렴한다는 것을 보였는데 이를 **퍼셉트론 수렴 이론**(Perceptron convergence theorem)이라 한다. 하지만, 출력 뉴런의 decision boundary는 선형(결합)이므로 퍼셉트론은 복잡한 패턴 (e.g. 비선형)은 학습하지 못한다. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Scikit-Learn의 Perceptron 클래스\n",
    "\n",
    "Scikit-Learn은 하나의 TLU 퍼셉트론을 구현한 [`Perceptron`](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Perceptron.html) 클래스를 제공한다.  아래의 예제는 iris dataset(붓꽃 데이터셋)에서 꽃잎의 길이(`petal length`)와 너비(`petal width`)를 가지고 붓꽃 종류 중 `Setosa`인지 아닌지를 분류하는 이진 분류기를 구현한 것이다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Perceptron(alpha=0.0001, class_weight=None, eta0=1.0, fit_intercept=True,\n",
       "      max_iter=100, n_iter=None, n_jobs=1, penalty=None, random_state=42,\n",
       "      shuffle=True, tol=None, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "from sklearn.linear_model import Perceptron\n",
    "\n",
    "iris = load_iris()\n",
    "X = iris.data[:, (2, 3)]  # petal length, width\n",
    "y = (iris.target == 0).astype(np.int)\n",
    "\n",
    "per_clf = Perceptron(max_iter=100, random_state=42)\n",
    "per_clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\n"
     ]
    }
   ],
   "source": [
    "y_pred = per_clf.predict([[2, 0.5]])\n",
    "print(y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnMAAAEUCAYAAABAqbBxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd4VVXWx/HvTYdAgEiNgKEMG1EZRUTGgiKKSBOGJrZRmoAiSLWLioqNdxhAKQldQIoYBlFGRUdhkAFGxEHZiLRBpJkQEkJIff+4ySWGJCQht4Xf53l4Jnedfc5duYFx5Zy993JkZ2cjIiIiIv4pwNsJiIiIiEjpqZgTERER8WMq5kRERET8mIo5ERERET+mYk5ERETEj6mYExEREfFjKuZERERE/JiKORERERE/pmJORERExI8FeeqNjDEvAjcBFYBfgX7W2sQ8x+sBMUAVIAN4zFq7LedYb+BJIBXYBQy21qZ6KncRERERX+XJO3M7rbXtrLU3AD8AT+c7/g4wxVrbGhgOLAIwxtQBxgO35ZybCIzwWNYiIiIiPsxjd+astYvzvNwM9Mx9YYypBlxurV2dM3arMeaUMaYpcCewxFp7Imf4TCAWmFiMtw0FrsN5JzDzwr8LEREREbcJBOrgrJPOFPckjxVzuYwxwcDjwF/zhOsD+/IN3Q9cCkQDWwuIF8d1wNelyVNERETES24G1hd3sEeLOWNMY5zz4hbn3oXLEYpznlxemUBWAcdy48XxK0B6+lc4p9v5nq1b9/Dcc+9z8mSKK9aixf3ceONQHI5AL2YmIiIipbVu7z8KPXZbg/YFjr2kShhvDGsDOfVLcXlyAURXYBwwyFq7I9/hgzjvzuV1Gc67dfmP5caLI+fRaipwuiTpesy119Zh+vT7GTDgXXbtcv7sDh6cyA8//JsePd4hNLSylzMUERGRkjqaUHjdkVzjvGNLNDXMIwsgjDG1gDeBjgUUclhrDwEnjDG35oxvAaRZa/cCK4AHjDHhOcOHAAs8kben1K9fnRUrRnP77Ve5Yrt3ryM2tgvx8Xu9mJmIiIiURkhgaLHjhY0tLkd2dvYFXaA4jDEdgPeA7/OE44HVwBFr7Uc5j2Bn4Xysmopz+5FdOef3BwbjfLy6GRhurS1O1RoN7E1P/we+emcur8zMLN56axXvvnv21mxYWFV69ZpBgwY3eTEzERERcbdKleC++wBoQPGfQnqmmPOiaPyomMv14Yf/ZuzYhaSlOacKOhyBdOjwEi1b/gWHw+Hl7ERERMQdSlvMqQOED+rWrRVLl46kZs0qAGRnZ/Lxx8+wZs2TZGamezk7ERER8SUq5nzU1VdHs2rVOJo3P7v2Y+vWhSxc2JeUlHgvZiYiIiK+RMWcD6tduypLl46ka9eWrtj+/RuJienE0aM7vZiZiIiIlJV//PwJq3fFFbmdSVFUzPm4sLAQJk9+mDFjurpiJ04cYPbsrlhbuh+6iIiI+I60zGI3eyiQijk/4HA4ePTRDsyc+Qjh4c7ly2lpp3j//X6sXz+Fcr6IRURERIqgYs6PtG//R1asGE3dupfkRLJZt24iK1cOIz3df1brioiISNlRMednmja9lLi4sbRq1dgV++9/VzJvXk9OnixR9w8REREpB1TM+aFLLqnMwoWP07fv2Y2EDx3aRkxMJ3755VsvZiYiIiKepmLOT4WEBPHqq3158cXeBAY6f4zJyUeYO7cH33+/0svZiYiISHH5RTsvL4rGDztAlNSGDTsZOjSGxMQUV+zGGx/jttvG4XCoXhcREfEH6gBxEbvxxqZ8+OFYGjWq5Ypt2DCV99/vx5kzyV7MTERERNxNxVw50aBBTVauHEvbtle4Yrt2fcrs2V1JSNjvxcxERETEnfSYtZzJzMzi9dc/ZObMz1yxChWq0avXTKKjb/BiZiIiIv5t9a64Qo91bnJ3qcfm0mNWASAwMICnn/4zb731ICEhQQCcPp3AwoV92bJlvpezExERkbKmYq6c6tmzNYsXj6B69QgAsrIyWLPmKdaseZrMzHQvZyciIiJlRcVcOXbttQ1ZtWosV15ZzxXbsmUe7713Hykp8V7MTERERMqKirlyLioqkmXLRtGpUwtXbN++DcTGduHYsV1ezExERETKgoq5i0CFCiFMndqfkSM7u2IJCfuIje3Crl2fFXGmiIiI+LogT76ZMeZG4E1gqLV2W574VcCUfMNvAFpaa7cbY9YA1YHcXXGftdau90TO5YXD4eDxxzvSpEkdnnhiHqdPp5GWlsySJQ9x++3P8Kc/DcbhcHg7TRERESkhj21NYoyZD1QG/gDcn7eYK2DsHcAQa+2fc15/A/Sy1v6vhG8bzUW2NUlx/PDDQQYOnM4vv5ydN9e8eQ86d36DoKAwL2YmIiJy8fKHrUkGW2u7A8eLGmSMcQCvAWPzhKud7zwpvmbN6hIXN47rrmvkim3fvoJ583qSlHTEi5mJiIhISXmsmLPWppx/FAA9gR3W2t354p8YY742xowzxmiu3wWqXr0y7703nD59zm4k/Msv3xIT05FDh7Z7MTMREREpCV8sih7HOa/OxVprrLW3AJ2B1sBQbyRW3oSEBDFx4n08/3xPAgKc8+WSkg4zd253duwofOdqERER8R0eXQBxPjkLISpYa/9b0HFrbaIxZjbQy7OZlV8Oh4N+/W6jcePaPPpoLElJp8nISGXFiqEcPWq59dbROBy+WPOLiMjF5B8/f0Ja5plz4iGBobRv1MELGZWd3O+tZrUK3Ef7Ep/va/+V7gkszxswxoQaYyrnfB0E9AE2eiG3cq1Nm2bExY2lYcOartjXX09m2bJBpKWd8mJmIiIiFFjIFRX3Jxf6PXi1mDPG9DPGdMoTuoVzC7UI4J/GmK+B9cBuYLqHUryoNGxYi5Urx9KmTTNXbOfOj5kz525OnCjpQmIRERHxBI9tTeIl0WhrkhLLyMjktddWEhu7zhWrWDGSXr1iuOyy672YmYiIXKxW7yp8LnfnJnd7MJOyl/u91axWgdhn24MPb00ifiIoKJDnnuvJG2/cT3BwIAApKfEsWtSL//xnkZezExERkbxUzEmheve+gcWLR1C9emUA0tMzWb16DJ988jxZWRlezk5ERERAxZycR8uWjYiLG8fll9d1xf7971gWLXqA06dPeDEzERG5mIQEhpYo7k8u9HvQnDkplpSUM4waNZ+PP/7WFYuMbMA998ylevXGXsxMRESkfPCHdl7ixypWDGXatP4MH97RFYuP30tsbBd27/7Ci5mJiIhc3FTMSbEFBATwxBOdmTZtAGFhwQCcOXOSxYsfZOPGGZTzu7wiIiI+ScWclFinTi1YvnwUdepUBSA7O4tPP32JVatGkpHh/5s3ioiI+BPNmZNSO3bsJI88MoP//GevK1a3bkt6946hUqUaXsxMRETKG3e18/KlNmGaMyceV6NGBIsXj6Bnz9au2MGDW4iJ6civvxbYXldERKRU3NXOqzy0CVMxJxckNDSYN998gGef7UFAgAOAkycPMXduN374YbWXsxMRESn/VMzJBXM4HAwY0I7Y2KFUrhwGQHr6aZYvf4Qvv3yb7OwsL2coIiJSfqmYkzLTtu0VrFw5hujos/PlvvpqEsuXP0JaWooXMxMRESm/VMxJmWrcuA4ffjiWm25q6or9+OMa5s7tRmLiL17MTEREpHxSMSdlrmrVcObOfZSHHrrVFTt8eAcxMR05cGCz9xITERG/5a52XuWhTZi2JhG3Wrx4Pc89t4SMDOe8uYCAYDp3fp2rr+7j5cxERER8i7YmEZ/Ut+9NvPfecCIjKwGQlZXOqlUjWbt2PFlZGV7OTkRExP+pmBO3u/76PxAXN46mTaNcsU2bZrF48V9ITU30YmYiIiL+T8WceES9epewYsVo7rijuSv2889fEhvbhd9++9mLmYmIiPg3j86ZM8bcCLwJDLXWbst3rDfwFrAnJ7TeWvtsnmNPAqnALmCwtTa1GG8ZjebM+ZSsrCwmTVrN1KmfuGJhYVXo0WM6jRq18WJmIiJSEF9pd7V6V1yhxzo3uft3r0uSs7u+v9LkW7NaBWKfbQ++OmfOGDMfGA1EFDIkEphlrb01509uIVcHGA/cZq29AUgERnggZXGDgIAARo/uyt/+1o/Q0GAAUlMTWbTofjZtiqWcL8gREfE7/tjuqiQ5+8L3d6Hv5cnHrIOttd2B44UcjyzkWG9gibX2RM7rmUA3N+QnHtS1a0uWLRtJ7dpVAcjOzmTt2udZvXoMmZlpXs5ORETEf3ismLPWnq8FQBgw0Biz0Rgz3xhTPycezdlHrwD7gUvdkKJ4WPPml7Fq1Tj++MdoV+zbbxczf34fTp0qrOYXERGRvHxmAYS19nlrbQvgBuBT4IOcQ6FA3j0sMgE1+ywnataswvvvP0H37q1csf/979/ExHTi8OEdXsxMRETEP/hMMZfLWpttrV0AXGaMCQYOAvXzDLmMEkwKFN8XFhbMpEl/4cknu+FwOABITDzInDnd2LnzYy9nJyIi4tt8ppgzxtTI8/VdwB5rbTqwAnjAGBOec3gIsMALKYobORwOBg9uT0zMYCpVCgMgPT2FpUsH8NVXf9XCCBERL/HHdlclydkXvr8LfS+Pt/MyxnwJjLDWbjPG9AOOWGs/MsZMA67DuYdIQs6YfTnn9AcG43y8uhkYbq3NLMbbRaOtSfzOrl2HGDBgOgcOnJ03d8UVXenadRLBwRW8mJmIiIj7lLadl3qzik9KSEhm6NAYNm7c5YrVqdOcPn1iiYiIKuJMERER/6TerFKuVKtWifnzh/HAA2c3Ev711+3MmtWRgwe3ejEzERER36JiTnxWcHAgL798DxMm3ENQkPOv6qlTx5g3ryfffbfMy9mJiIj4hiBvJyByPvff34aGDWsxdGgMJ06cIjMzjbi4ERw9amnX7ikCAgK9naKIiFv4SistdyhJu6uSKsnn5q42YZ6kO3PiF264wbBq1TiaNKnjim3c+C5LljzEmTNJXsxMRMR9fKHVlD9y1+fmqz8PFXPiN+rXr86KFaO5/farXLHdu9cRG9uF+Pi9XsxMRETEe1TMiV+pXLkCM2Y8wpAh7V2x48d/IiamM3v3rvdiZiIiIt6hYk78TmBgAOPGdeOvf32IkBDntM/U1BMsXHgvmzfP1QbDIiJyUVExJ36rW7dWLF06kpo1qwCQnZ3Jxx8/w5o1T5KZme7l7ERERDxDxZz4tauvjmbVqnE0b362fe/WrQtZuLAvKSnxXsxMROTC+UKrKX/krs/NV38e6gAh5UJqahpjxy5k1aotrljVqvW555451KzZ1IuZiYiIFI86QMhFLSwshMmTH2bMmK6u2IkTB5g9uyvW/sOLmYmIiLiXijkpNxwOB48+2oGZMx8hPNx5yzst7RTvv9+P9eunaGGEiIiUSyrmpNxp3/6PrFgxmrp1L8mJZLNu3URWrhxGeroet4uISPmiOXNSbsXHJzNkyCw2bfrJFYuKuprevWOIiKhTxJkicrHz1bZNRSlJWypfGFvSz7gk4/3x5weaMydyjsjISixYMIx7773JFTt0aBsxMZ345ZdvvZiZiPg6X23bVJ6U9DMuSfxi+/mpmJNyLSQkiFde6ctLL/UhMND51z05+Qhz5/bg++9Xejk7ERGRC6diTso9h8PBgw/ewvz5j1GlSkUAMjPPsHLlY3z++WtkZ2d5OUMREZHSUzEnF40bb2xKXNxYGjWq5Ypt2DCV99/vx5kzyV7MTEREpPSCPPlmxpgbgTeBodbabfmO/Rl4DGeBGZQzZnvOsTVAdSAlZ/iz1lp1VZcSi46uycqVYxk+fDZffLEDgF27PmX27K7cc88cqlW7zMsZioiIlIzH7swZY+YDo4GIQoY4gA7W2luBF4BZeY5FAj2stbfm/FEhJ6UWEVGBmJghDBp0uyt27Jhl4cI72LfvX17MTER8ha+2bSpPSvoZlyR+sf38PLY1iTGmorU2xRjzJTAi/525fGMjgP9aa+vnvLbA1dbaku4vEo22JpEiLF/+DU8/vYi0tAwAgoICaN/+FVq2fNDLmYmIyMWmtFuTFPiY1RgzByhplfe8tfZgYQettSmFHSvAaGBpvtgnxpgAYDXwprVWs9blgvXs2ZqGDWsxaNAMjh8/SUZGFmvWPMXRozu5884XCQwM9naKIiIiRSpszlxpHmNe8AxyY0xlYDKQCQzJjVtrTc7xKsBcYCgw9ULfTwSgRYsGrFo1lkGDZvDf//4PgC1b5nH8+G569pxOxYqRXs5QRESkcAUWc9baWE8nYoz5A7AQeNVaW+AW0tbaRGPMbKCXR5OTci8qKpJly0YxevR8PvroPwDs27eB2Ngu3HPPHGrUaOLlDEVERApW6GpWY0zIec7NKONHnYuBwdbaLfnyCAVCrLVJxpggoA+woQzfVwSAChVCmDq1P02bXsrbb/8dgISEfcTGduHPf55Gkya3n+cKIiJlyxdaWPnbdf01jwtR1NYkqTjnzTn4/fy53NczcD7uLDVjTD/gCLAOuBp4yxiTd8iDOFcurDXGnAKCgU+A6RfyviKFcTgcDBt2F3/4Q22eeGIep0+nkZaWzJIlD3H77c/wpz8NxuFweDtNEblI+EILK3+7rr/mcSEKLeasta5tS3IWRCyx1q7Ned0NaF+aN8zZeiT369nFyQVoUZr3EimtDh2uoX79GgwcOJ1ffokHsvnsswkcPfojnTu/QVBQmLdTFBERAUq2z1x2IV+LlEvNmtUlLm4c113XyBXbvn0F8+b1JCnpiBczExEROUvtvESKUL16Zd57bzh9+tzgiv3yy7fExHTk0KHtXsxMRETEqbjFnO7EyUUrJCSIiRPv44UXehEQ4Jwvl5R0mLlzu7NjR4ELr0VERDymqNWsX3C2iLscaGGMeTLndXVKtxediF9yOBw8/HBbGjWqxWOPxXLy5GkyMlJZsWIoR49abr11NA6HbnSLSNkKCQwtdKXlhYx1Vw6+cF1/zeNCFNrOyxhz53nOPWCt/bHsUypT0aidl5SxPXuOMGDAu+zZc9QVa9r0Lrp1m0xISLgXMxMREX9W2nZeHuvN6iXRqJgTN0hMTGHYsNl89dUPrlitWpfTp88cqlat58XMRETEX5W2mNNzIZFSqFKlIrNnD6F//9tcsSNHfiQmpiP792/yYmYiInKxUTEnUkpBQYE891xP3njjfoKDAwFISYlnwYI+/Oc/i7ycnYiIXCwKXACRs0lwSZ+/Pm+tPXjhKYn4l969b6Bhw1oMHjyT48eTyMpKZ/XqMRw9upP27Z8nIKCo/bBFfEd5aGvk60r6GetnIsVR2H9l8q9UHQfMBo4Vca3kMslIxA+1bNmIuLhxDBgwnR9/dP5O8+9/x3L8+E/06PEuFSpU9XKGIudXHtoa+bqSfsb6mUhxFFjMWWtj8742xjwELLXW7sl5XQ84Yq1Nc3uGIn7i0ksjWbFiFKNGzefjj78FYM+er4iN7cw998ylevXGXs5QRETKo9LOmXsNuL4sExEpDypWDGXatP4MH97RFYuP30tsbBd27/7Ci5mJiEh5Vdpi7jgQVZaJiJQXAQEBPPFEZ6ZNG0BYWDAAZ86cZPHiB9m4cQblfDsgERHxsOIWc2uBpDyvTwGVyj4dkfKjU6cWLF8+mqioagBkZ2fx6acvsWrVSDIyNN9FRETKRrGKOWvtBGtt3sUPZ4Bg96QkUn5ceWU94uLG0aJFA1fsu++WMn9+b5KTi1pPJOJ5hbUv8qe2Rr6upJ+xfiZSHKXqAGGMeRY4Ya2dWvYplalo1AFCfMCZM+k888xili//xhWLiIiiT5851KlzpRczExERX+HpDhBZgKOU54pcdEJDg3nzzQd49tkeBAQ4/+mcPHmIhQu78MMPq72cnYiI+LNCdzM1xuyl8I2DqwAvuiUjkXLK4XAwYEA7GjWqzeOPx5KUlMrp02ksX/4IbdqM5JZbnsDhUFMWEREpmaK2pr/9POeWeMKPMeZG4E1gqLV2W75j9YAYnIViBvBY7hhjTG/gSSAV2AUMttamlvT9RXxB27ZXsHLlWAYMeJd9+5z/jL76ahLHju3k7rsnExJS0csZinjX6l1xhR7r3ORun7uur3RpKEkevpKzlI1Cizlr7c9l+UbGmPlAZSCikCHvAFOstauNMdcCi4Bmxpg6wHjgBmvtCWPMZGAEMLEs8xPxpMaNaxMXN45HH41h/fqdAPz44xoSEvbTp88cqlS51MsZikhx+UqXhpLk4Ss5S9nw5DOdwdba7jj3qPsdY0w14HJr7WoAa+1W4JQxpinQG1hirT2RM3wm0M1DOYu4TZUqFZk791EeeuhWV+zw4R3ExHTkwIHN3ktMRET8iseKOWttShGH63Puqo39wKU4V6TuKSAu4veCggIZP743r712L8HBgQCcOnWc+fN7sW3b+17OTkRE/IGvzLYOxTlPLq9MnKtm8x/LjYuUG3373sTChY8TGencizsrK51Vq0aydu14srLy/9MQERE5y1eKuYM4787ldRnOu3X5j+XGRcqV66//A3Fx42ja9GynvE2bZrF48V9ITU30YmYiIuLLfKKYs9YeAk4YY24FMMa0ANKstXuBFcADxpjwnOFDgAVeSVTEzerVu4QVK0Zzxx3NXbGff/6S2Ngu/PZbma5JEpEy4itdGkqSh6/kLGWjVB0gLoQx5ktghLV2mzGmH3DEWvuRMaYxMAvnY9VUnAsmduWc0x8YjPPx6mZguLU2sxhvF406QIgfysrKYtKk1Uyd+okrFhZWhR493qVRo1u8mJmIiLhLaTtAlLadV2tr7Td5Xj9urf1biS/kftGomBM/9ve/b2H06AWcOZMOgMMRQPv2L9CqVX8cDjVhEREpT0pbzBW4z5wxpivwNgV3gBgNTAei8sSeBHyxmBPxa126tCQ6uiYDB07n8OETZGdnsXbtCxw9upOOHV8lMDDE2ymKiIiXFbZp8BdA50KOHQLCjTGP57x2ANqyXsRNrrqqPqtWjWPQoBls27YPgG+/Xczx4z/Tu/cswsOrezdBERHxqtI+Zk0GZuMs5AAestZWLsvEykg0eswq5URqajpPP/0eH3zwb1esSpW69Okzm9q1r/BiZiIiUhbK9DFrLmPMVUAvIA1YaK3NvXCStfbxPON6lixdESmpsLBg3n77LzRpEsXrr8eRnZ1NYuJB5szpRvfuf6Np07u8naKIiHhBoVuT5GwTsgbnhr3hwNfGmCY5h/PfzvPskliRi5TD4WDw4PbExAymUqUwANLTU1i6dABfffVXPL06XUREvK+oO3MvA92ttVsAjDGbgeeB+4EAY0wFnI9Zc/+IiIe0a3cVH3wwmgEDpnPggLPd8ZdfvsmxY5auXScRHFzByxmKiIinFLVpcFRuIZfjE+CqnK9rAEeAwzl/argnPREpTJMmUcTFjeVPf2riiu3YsYq5c//MyZOHvJiZiIh4UlHFXKYxJjjP6+rAiZyvj1prI3L+VAaOui1DESlUtWqVmD9/GA8+eHYj4V9/3c6sWR05eHCrFzMTERFPKaqY+wh4wxjjMMaEAq8DK3OO5f+vhB6zinhJcHAgL73UhwkT7iEoyPlP+tSpY8yb15Pvvlvm5exERMTdiirmngPqAieB40AiMAXAWpt/D7qGbslORIrt/vvbsGDB41St6mxjnJmZRlzcCD79dAJZWcXpficiIv7ovPvMGWMqApnW2jOeSalMRaN95uQic+DAcQYMeJddu351xRo3vo0ePd4hNNQXt4MUEREo/T5zRd2ZA8Bam+KnhZzIRal+/eqsWDGa22+/yhXbvXsdsbFdiI/f68XMRETEHc5bzOVljIk6/ygR8bbKlSswY8YjDBnS3hU7fvwnYmI6s3fvei9mJiIiZa1ExRzwTf6AMUadvkV8UGBgAOPGdeOvf32IkBDnlpKpqSdYuPBeNm+eqw2GRUTKiaI6QPzRGNPMGNPQGHNJTtiRc2xCnqF9jDHj3JmkiJRet26tWLp0JDVrVgEgOzuTjz9+hjVrniQzM93L2YmIyIUq6s7cv4GFwN+B73MKutxf5fvlGXcH8IN70hORsnD11dGsWjWO5s3ru2Jbty5k4cK+pKTEezEzERG5UEUVc0ettS2stVdYa6Ostb/lOZZ7h64B0ApY7c4kReTC1a5dlaVLR9K1a0tXbP/+jcTEdOLo0Z1ezExERC5EUcVcURNqsnN6sy4AnrPWavKNiB8ICwth8uSHGTOmKw6Hc6/vEycOMHt2V6z9h5ezExGR0iiqmMu9+3a3MWa2MWZ2zutBQGVgO7DRWqst5kX8iMPh4NFHOzBz5iOEh4cCkJZ2ivff78f69VO0MEJExM8EFXHsjpz/PQBsyPn6diAr5+tMoNizp40xbXG2BMvA2cu1n7U2PufYVeR0l8jjBqCltXa7MWYNzt6wKTnHnrXWan8FkQtwxx3NWbFiNAMGTOfgwd+AbNatm8jRo5YuXd4kOLiCt1MUEZFiKLQDhDHmFHAC+BH4EngT+MlaW98Ycwhnd4XVwKzz3Z0zxoQB3wF3WGsPGGOeAIy1dnAh4+8Ahlhr/5zz+hugl7X2fyX8/qJRBwiRIsXHJzNkyCw2bfrJFYuKuprevWOIiKjjxcxERC4u7ugAkYDz7thrQIV8XSAc1to04CHgZWNM4Hne505gg7X2QM7rGKBrQQONMY6c9xybJ1wNZ39YESljkZGVWLBgGPfee5MrdujQNmJiOvHLL996MTMRESmOIhdAWGv3W2s/t9Y+U9AAa+0h4Gugy3neJxrYk+e8JCDIGBNcwNiewA5r7e588U+MMV8bY8YZY0q62bGIFCEkJIhXXunLSy/1ITDQ+c8rOfkIc+f24PvvV3o5OxERKUpRRdFnRRybmefrdUDD87xPKM65cnllUPCK2cdxPtJ1sdYaa+0tQGegNTD0PO8nIiXkcDh48MFbmD//MapUqQhAZuYZVq58jM8/f43s7KzzXEFERLyh0GLOWvtwAWFHzrEX8sRWWGsnned9DgKu3UqNMeFAqrX2dwVezkKICtba/xaSUyIwG+fediLiBjfe2JS4uLE0blzbFduwYSrvv9+PM2eSvZiZiIgUpKRro/7LAAAgAElEQVSPK1vmD+TMnTufj4EOxphaOa8HAYsKGNcTWJ43YIwJNcZUzvk6COgDbCxJ0iJSMtHRNfnggzG0bXuFK7Zr16fMnt2VhIT9XsxMRETyK1ExZ609Upo3sdYmAMOAv+esTL0OeM0Y088Y0ynP0Fs4t1CLAP5pjPkaWA/sBqaXJg8RKb6IiArExAzhkUdud8WOHbPExHRi375/eTEzERHJq9CtScqJaLQ1icgFW7HiG556ahFpac6ZEQEBQXTo8DItWz7o5cxERMoPd2xNIiICQI8erVmy5AmqV48AICsrgzVrnmLNmqfJzCz23uEiIuIGKuZEpFhatGjAqlVjufLKeq7Yli3zeO+9+0hJifdiZiIiFzcVcyJSbFFRkSxbNopOnVq4Yvv2bSA2tgvHju3yYmYiIhcvFXMiUiIVKoQwdWp/Ro06u1d4QsI+YmO7sGtXUdtTioiIO6iYE5ESczgcDBt2F9OnD6JixVAA0tKSWbLkIf71r3cp5wurRER8ioo5ESm1Dh2uZvnyUVx6aWROJJvPPptAXNxwMjJSvZqbiMjFQsWciFyQZs3qEhc3juuua+SKbd++gnnzepKUVKqtKUVEpARUzIn4uITDR/nxX5vZ/sV6fvzXZhIOH/V2SueoXr0y7703nD59bnDFfvnlW2JiOnLo0HYvZiYiUv6pmBPxYQmHj3LQ7ib9zBkA0s+c4aDd7ZMFXUhIEBMn3scLL/QiIMABQFLSYRYu7MqOHXFezk5EpPxSMSfiww7v2U92VtbvYtlZWRze45v9UR0OBw8/3Ja5cx8lIqICAKmp6axYMZQvvniD7Oys81xBRERKSsWciA/LvSNX3LivaNOmGR9+OJaGDWu5Yl9/PZllywaRlnbKi5mJiJQ/KuZEfFhwaGiJ4r6kYcNarFw5hjZtmrliO3d+zJw5d3PixP+8mJmISPmiYk7Eh9VueBmOgN//M3UEBFC74WVeyqhkqlSpyOzZQxgwoJ0rduTIj8TEdGT//k1ezExEpPxQMSfiw6rVrkld09h1Jy44NJS6pjHVatf0cmbFFxQUyLPP9uCNNx4gODgQgJSUeBYs6MN//rPIy9mJiPg/RznfqT0a2Jue/g/gtJdTEZEtW35m8OCZHD+e5Iq1atWf9u2fJyAgyIuZiYh4X6VKcN99ADQA9hX3PN2ZExGPadmyEXFx47j88rqu2L//HcuiRQ9w+vQJL2YmIuK/VMyJiEddemkkK1aM4q67rnHF9uz5itjYzhw/vtuLmYmI+CcVcyLicRUrhjJtWn+GD+/oisXH7yU2tgu7d3/hxcxERPyPx+bMGWPaAq8DGcBRoJ+1Nj7P8d7AW8CenNB6a+2zeY49CaQCu4DB1tridPGORnPm5CKScPgoh/fsJ/3MGYJDQ6nd8DKfXyzx0Uf/YdSoeaSmpgPgcARw++3P0rr1IBwOh5ezExHxHJ+eM2eMCQOmAz2ttTcA/wRezTcsEphlrb01509uIVcHGA/clnNuIjDCE3mL+BN/av2VV6dOLVi+fDRRUdUAyM7O4tNPX2LVqpFkZPj25sgiIr7AU49Z7wQ2WGsP5LyOAbrmGxMJHC/g3N7AEmtt7uzomUA3t2Qp4sf8rfVXXldeWY+4uHG0aNHAFfvuu6XMn9+b5ORjXsxMRMT3eaqYi+bs41OstUlAkDEmOM+YMGCgMWajMWa+MaZ+QecC+4FL3ZuuiP/x19ZfuWrUiGDx4hH07NnaFTt4cAsxMR359df/ejEzERHf5qliLhTnXLm8MgDXhD1r7fPW2hbADcCnwAeFnJsJqFu3SD7+3PorV2hoMG+++QDPPtuDgADnfLmTJw8xd243fvhhtZezExHxTZ4q5g4CuXfaMMaEA6nW2vwFHtbabGvtAuCynDt3vzsXuIwSTAoUuVj4e+uvXA6HgwED2hEbO5TKlcMASE8/zfLlj/Dll2+Tna3f5URE8vLUlusfAxOMMbWstUeAQcDv+vgYY2pYa4/lfH0XsMdam26MWQEsN8ZMs9aeAoYACzyUt4jfyF216m+rWQvTtu0VrFw5lgED3mXfPue8ua++msSxYzu5++7JhIRU9HKGcjGJiEikbt2jBAenezsV8XNJSeHs3VuP7Oyyu5/mya1JOgEv4HxEugcYCPQBjlhrPzLGTAOuw7mHSAIwwlq7L+fc/sDgnHM3A8OttZnFeNtotDWJiF9LTEzh0UdjWL9+pytWu/YV9OkzhypVNH1W3C8iIpGGDQ8TEVGdgIBQbZkjpZadnUVKynH27w/nyJFzf9Eu7dYk6s0qIj4vIyOTCRNWMHful65YeHh1evWKoX7967yXmFwUmjX7ierVIwkMDPN2KlIOZGWl89tvR/jvf805x3x6nzkRkQsRFBTI+PG9mTjxPoKDAwE4deo48+f3Ytu2972cnZR3wcHpBAT4z0Ii8W0ORxCBgcV5uFh8KuZExG/cc8+NvPfecCIjKwHO33BXrRrJ2rXjyco6Zz2VSJnRo9WLy/ffb2Hs2H6lOnfr1g1s27ap0OPu+LvkqQUQIn7Hna2xfv52O6dOnHS9Dq8aQaNrml9wHu7K2ZfahLVq1Zi4uHEMHPguO3ceAmDTplkcP/4TPXq8Q1hYFa/kJSL+ZfPmr5kz56/Exx+jTp16DB78JMZcdd7zHnjgDt56ax61akUVePz777cSEhLC1VdfX9YpF0p35kQK4M7WWPkLOYBTJ07y87fbLygPd+Xsi23C6tW7hBUrRtO+/R9dsZ9//pLY2C789tvPXstLxJf83/+9wPvvxxR6PCkpkaefHkRy8slCxxQlMTGBd955lcGD/8xf/nInDzxwO19+uaa06XrUoUMHmDx5PI8++iyLF/+TXr36MWHCEyQnJ3k7tVJRMSdSAHe2xspfyBUVL0ke7srZV9uEhYeHMX36QIYNu8sV++23n4mN7cLPP//Ti5mJ+IfKlavw6qszqVQpolTnT5jwBFFR9Zk2bRnz5q1l6tRlNGjQpFjn7t27i3HjSvcYsyysWrWIzp3v4YorrsHhcHDDDe249tobWbt2hddyuhB6zCpSAF9pjVWSPNyVs698FgUJCAhg1KguNGlSh9GjF3DmTDqpqYksWnQ/7du/QKtW/TXXSXzC+PlfkXQ67Zx45QohjH+wjcfzycrKIiCg9PdzkpIS2blzOxMmTCcw0LkoqUqValSpUq1Y5ycnnyQhoaB27J6xd+8uHnpo+O9iV199PZs3ry/1NZOTkwgODj7/QDdQMSdSgODQ0AKLFU+3xipJHu7K2Vc+i6J06dKS6OiaDBw4ncOHT5CdncXatS9w9OhOOnZ8lcDAEG+nKBe5ggq5ouLu0KVLC0aOfJlly+ZQv34jHn54OIMG3U1c3GbAOXF/4cJ3OXnyBFlZmTz11Fs0aXJFgdcKD6/MJZfUZN68KTz88IgCi5gDB/Ywc+Yb/PrrQUJDQ+nbdxA333wncXHvsXz5XJKSEunfvxPXXnsTQ4c+xZkzqSxZMouNG78gLS2ViIiq3HvvYFq1cha7hw4dYMaM1/n11/9x+vRp+vYdRMeOvTh58gQzZrzBzp3fkZWVxWWXNWbEiBepWjWy0M8iMfEEERFVfxeLiKjKyZMJrtdHjhxk+vSJAAwe/OTvxo4a9QCBgc4SKvcXxgoVKjJkyFNF/gzcRcWcSAFqN7yMg3b37x4vllVrrPCqEQU+Ug2veu6jjpLk4a6c3flZlKWrrqrPqlXjGDRoBtu27QPg228Xc/z4z/TuPYvw8OreTVDEB2zZsoHJkxcTGBjIsWOHXfHU1NO8+upo/u//3qN+/YYkJ58kM7Pw7TMCAgJ4/vnJTJw4lk2bvuSeewbSrl1X192+xMQEnnnmER599Blat76VQ4cOMG5cP+rXb8Tdd99Hw4aGKVNeYubMVa5rTpv2CmfOpDJp0gIqVgzH2u958cXhjB8/hSZNruBvf3uRNm060LFjLzIy0jl+/EhO7incdNPtjBz5Mg6Hg7feepply2YzcODoQvOPjKxOQsJxLr307P+PJST8RmRkDdfrChUq0azZNeecu2DBp0V+xv/5z8Yij7uD5syJFKBa7ZrUNY1dd5+CQ0OpaxqXyQrORtc0P6dwK2w1a0nycFfO7vwsylrNmlVYsuQJ/vznVq7Y//73b2JiOnH48A4vZibiGzp27ElwcPA5j1gdDgfBwcH89NMOsrKyqFQp4ryPTBs2NEybtpwuXfoyb94Uxo59iBMn4gFYt241zZtfR+vWtwIQFVWfG25ox8aNXxR4rZMnT/DPf37CsGHPUbFiOADGXMVdd/Xg008/BCA4OJR9+3aTmnqaoKBgateuC0DNmlH86U+3kZx8kp07txMeXpkDB/YUmXuTJleyadNXv4t9882Xv1vNGhFRlTZt7qRNmzuLvFZ+bdt25Kab2pfonAulO3MihahWu6bbCpbCtiG50DzclbM7P4uyFhYWzNtv/wVjLmXixA/Jzs4mMfEgc+Z0o3v3v9G06V3nv4hIOVWjRp0C46GhYUyYMIMFC6axaNEM7r77Xjp3voeAgADefvsZfvhhm2vspEkLXYVecHAw3brdT/v23XjllVHExr7NqFGvcPjwQbZu3UD//p1c56WlpRVaGB0+/AtVq0aesxijTp267NljARg1agILFkxj4MAutGlzJ/feO5jw8Mr8/PNOpkx5iYoVKxEVVZ/k5JNkZBTdQ7dr1748/vg9NGrUlGuuac0XX3zEnj2WMWNePf+HCLz66ih+/PHcHQhyde/+APXrNyzWtcqCijkRKXccDgePPHIHf/hDbR5/fA7Jyamkp6ewdOkAbr11DDffPFwLI+SiVNSih8aNL+fFF6dy6NABJk4cS1ZWFt263c+oUa+c97oVK1aie/cHmTdvMgCRkTW45ZYOxZ5DVqNGLRITE0hJSaZixUqu+OHDh1x34KpWjWTYsOdITh7OjBmvM2nSczz33F+ZNetNunTpS7t2XQD46KOlrF9f9KPQyMgajB8/lYUL32HJkplER/+BV16ZTkhI8eYCJyWdZOzY17jqqpbnHFuwYBqpqSnFuk5Z0WNWESm3brvtKlauHMNll52dB/Pll2/ywQdDSU9Xv2bxnMoVCl6EU1jc05KTT7Jrl3MqQp069ahXrwGnT58qdHx8/DHWrl3JqVPOfdnS0s6wceM6mjd3TnG45ZYObNjwGTt2fAtAdnY23323ybWnXaVKEZw4kcDp0ylkZmZQrVp1WrVqw5QpL3P6tLMQ2r37Bz7/fBWdOvUGYNu2Ta5HwJdffrVrXHp6OklJiQAcP36ETz4p3vYijRtfzvjxU5g+fSVPPvmGq2j0R7ozJ1KIg3Y38YfOThCOjKpNXdO40PHu6upQEr7UqcFX/OEPdfjwwzEMHRrDxo27ANixYxXx8fvo0yeWiIiCd3EXKUve2H6kJDIyMpg27RXi449RsWI4V155Ld27P1jo+KCgYNav/5R586YQFhZGhQrhXH99G/r2fQSA2rXrMmbMa8TGTiIh4TeCg4No1uwajHH+f2KDBk24/vo2DB7cjZtvvpMBA0YxYsSLLFz4DsOG9QGgevVajBs3kbp1owH45JMPePPNp6hUKYLatS/lsceeBaBfvyeYNu0V4uIWERVVj7ZtO17QFiP+yJGdne3tHNwpGtibnv4PQL+FS/HlL+RyFVbQFdTVAQou6HI7KuRfHXqhiwrcdd3yIj09k5dfXs78+Wc3FA4Pr0GfPrHUrXutFzMTX/fHP/5AZKRvrd4W9/r++y0sWPAOb7wxu8DjTz01kH37fiIo6NwtWVJTU+je/QHuvXdwodePj9/Pd981OydeqRLcdx8ADYB9xc1Xd+ZEClBQIZcbL6iYK6uuDhdSdLnruuVFcHAgL73UhyZN6jB+/FIyMrI4deoY8+b1pHPnN/jjH3t5O0UR8RGNGzdjxIjxhR5/7bVZnkumGDRnTsTDLsZODb7k/vvbsGDB41Sr5tz+IDMzjbi4EXz66QSysgrfV0tELh4VKlQkKqq+t9MoNhVzIh5WWOeEsujU4I7rlkd/+lMT4uLG0aTJ2W0aNm58lyVLHuLMGf9stC0iFy8VcyIFiIyqXaJ4Qd0bCovXbngZjvwbdpZRpwZ3XLe8ql+/OitWjOb2289uErp79zpiY7sQH7/Xi5mJiJSMx+bMGWPaAq8DGcBRoJ+1Nj7P8T8Dj+EsMIOAodba7TnH1gDVgdyNW5611l5cS1XEo3LnxRV3NWuja5oXezVr7vy1sl516q7rlmeVK1dg5sxHeOutv/POO2sBOH78J2JiOtOr1wwaNLjJyxmKiJyfR4o5Y0wYMB24w1p7wBjzBPAqkHephwPoYK1NM8a0A2YB1+cciwR6WGv/54l8RcBZ0BW1FUl+7urqUBL+1KnBVwQEBDB27N00aVKHsWMXkpaWQWrqCRYuvJcOHV6iZcu/aINhEfFpnnrMeiewwVp7IOd1DNA17wBr7QprbVrOy81A3p4j1YDjbs9SRC5a3bq1YunSkdSsWQWA7OxMPv74GdaseZLMzKJbA4mIeJOnirlowNX11lqbBAQZY87doMVpNLA0X+wTY8zXxphxxhjN9RORMnf11dGsWjWO5s3PrmLbunUhCxf2JSUlvogzRUS8x1NFUSjOuXJ5ZQC/27HYGFPZGDMb5125J3Pj1lpjrb0F6Ay0Boa6N10RuVjVrl2VpUtH0rXr2Z6L+/dvJCamE0eP7vRiZiIiBfPUAoiDgGsmsTEmHEi11mbkif0BWAi8aq2NK+gi1trEnGJPu3uWc+5qS1WSFl07v9lC2ulU1+uQCmE0bX1uU+Vc2784d01O87YFT6D//p8byM46+7uMI8DBVbfcWODYHeu/ITP97O9CgcFBXHFT6wLHqk1Y2QgLC2Hy5IcxJoq33vo72dnZnDhxgNmzu9K9+1SMae/tFEWK5f/+7wWiourRp8+Ac44lJSXy2mtjePrpt6hUqeAV+UVJTEzgvffeZfv2LZw+fYqsrEz69x/Jrbd2LIvUpQQ8dWfuY6CDMaZWzutBwKJ8YxYDj+Yv5IwxocaYyjlfBwF9gI1uzle8KLctVe5mt+lnznDQ7ibh8NELum5BLbriDx3moN19ztj8hRxA2ulUdn6zpcBrF1TIFRbPX8gBZGdl8/0/N5wzNn8hB5CZnsGO9d+cM9Zdn5u7ruvrHA4Hjz7agZkzHyE83LlXX1raKd5/vx/r10+hnLdClItA5cpVePXVmaUq5AAmTHiCqKj6TJu2jHnz1jJ16jIaNGhSrHP37t3FuHH9SvW+ci6P3Jmz1iYYY4YBfzfGZOGcPzfQGNMPOAKsA64G3jLG5D31QZxNVdcaY04BwcAnOFfGSjnlrrZUJWnRlb+QO1+8JPIXckXF8xdyRcXVJsw97rijOR98MIb+/d/l4MHfgGzWrZvI0aOWLl3eJDi4grdTFD+RlrydtITPyc5MxBFYhZBq7QipVPxV8GUpKyuLgIDS389JSkpk587tTJgwncDAQACqVKlGlSrVinV+cvJJEhK0rrGseGyfOWvtR8BH+cJ5O9gWlUuLss9IfJXaUpWO2oS5jzFRrFo1jiFDZrFp008A/Pe/K4mP30vv3jFERNQ5zxXkYpeWvJ0zv/0dsp0ro7MzE52vwWMFXZcuLRg58mWWLZtD/fqNePjh4QwadDdxcZsB2Lp1AwsXvsvJkyfIysrkqafeokmTKwq8Vnh4ZS65pCbz5k3h4YdHEBx87nrGAwf2MHPmG/z660FCQ0Pp23cQN998J3Fx77F8+VySkhLp378T1157E0OHPsWZM6ksWTKLjRu/IC0tlYiIqtx772BatWoDwKFDB5gx43V+/fV/nD59mr59B9GxYy9OnjzBjBlvsHPnd2RlZXHZZY0ZMeJFqlaNdN+H6WM8VsyJFFdwaGiBhYLaUhXNXZ+bfh5OkZGVWLBgGOPHL2XRIufj80OHthET04k+fWK59NJrvJyh+LK0hM9dhZxLdjppCZ979O7cli0bmDx5MYGBgRw7dvZpRWrqaV59dTT/93/vUb9+Q5KTT5KZWXiv4oCAAJ5/fjITJ45l06YvueeegbRr19V1ty8xMYFnnnmERx99htatb+XQoQOMG9eP+vUbcffd99GwoWHKlJeYOXOV65rTpr3CmTOpTJq0gIoVw7H2e158cTjjx0+hSZMr+NvfXqRNmw507NiLjIx0jh8/kpN7CjfddDsjR76Mw+HgrbeeZtmy2QwcONpNn6Lv0RYf4nPc1ZaqJC26QiqEFTi2sHhJOAIK3oC2oHhgcMG/bxUUV5sw9wsJCeKVV/ry0kt9CAx0fibJyUdYsKAb33+/0svZiS/LzkwsUdxdOnbsSXBw8DmPWB0OB8HBwfz00w6ysrKoVCnivI9MGzY0TJu2nC5d+jJv3hTGjn2IEyecW/isW7ea5s2vo3XrWwGIiqrPDTe0Y+PGLwq81smTJ/jnPz9h2LDnqFgxHABjruKuu3rw6acfAhAcHMq+fbtJTT1NUFAwtWvXBaBmzSj+9KfbSE4+yc6d2wkPr8yBA3sKfJ/ySsWc+JxqtWtS1zR23fkJDg2lrml8wfOz6prG5xRuha1mbdq65TmFW1GrWQtbtVpQ/KpbbjyncCtsNesVN7U+p3ArbDWruz43d13XXzkcDh588Bbmz3+MKlUqApCWlsHKlY/x+eevkZ2ddZ4ryMXIEVilRHF3qVGj4CkBoaFhTJgwg6++WsvAgV1ZtWoRWTlzZd9++xn69+/k+pOYmOA6Lzg4mG7d7mfmzDhCQysQG/s2AIcPH2Tr1g2/O+9f/1pHUlLBxevhw79QtWrkOYsx6tSp67oDN2rUBDIzMxg4sAuzZr3JqVNJAPz8805GjLiX118fx7p1q0lKSiQj4+La6FuPWcUnuastVUladBW1DUlBCivoClLYNiQFKWwbkoKoTZjn3HhjU+LixjJgwHR273Y+rtqwYSrHjlm6d59KaGglL2coviSkWrvfzZkDwBFMSLV2Hs2jqEUPjRtfzosvTuXQoQNMnDiWrKwsunW7n1GjXjnvdStWrET37g8yb95kACIja3DLLR0YMuSpYuVVo0YtEhMTSElJpmLFs/92Dh8+5LoDV7VqJMOGPUdy8nBmzHidSZOe47nn/sqsWW/SpUtf2rXrAsBHHy1l/fpPi/W+5YXuzImIlFJ0dE0++GAMbduenSS+a9enzJ7dlYSE/V7MTHxNSKXmhF7SxXUnzhFYhdBLunhtNWt+yckn2bVrBwB16tSjXr0GnD59qtDx8fHHWLt2pevuWFraGTZuXEfz5q0AuOWWDmzY8Bk7dnwLQHZ2Nt99t4nk5JMAVKoUwYkTCZw+nUJmZgbVqlWnVas2TJnyMqdPpwCwe/cPfP75Kjp16g3Atm2bXI+AL7/8ate49PR01x2/48eP8MknK8r64/F5ujMnInIBIiIqEBMzhDfe+JAZMz4D4NgxS0xMJ3r1mkl09A1ezlB8RUil5j5TvOWXkZHBtGmvEB9/jIoVw7nyymvp3v3BQscHBQWzfv2nzJs3hbCwMCpUCOf669vQt+8jANSuXZcxY14jNnYSCQm/ERwcRLNm12CM8/tv0KAJ11/fhsGDu3HzzXcyYMAoRox4kYUL32HYsD4AVK9ei3HjJlK3bjQAn3zyAW+++RSVKkVQu/alPPbYswD06/cE06a9QlzcIqKi6tG2bUc2by5478/yylHON76MBvamp/8D53Z1IiLus2LFNzz11CLS0pz7AAYEBNGhw8u0bFn4fxTF9/3xjz8QGXnxLfgR94mP38933zU7J16pEtx3HwANgH3FvZ7uzInf85VWUyXJo6StwsQ/9OjRmgYNajFo0AyOHz9JVlYGa9Y8xdGjO7nzzhcJDDx3Ly4RkQulOXPi13yl1VRJ8ihpqzDxLy1aNODvfx/HlVfWc8W2bJnHe+/dR0pKvBczE5HySsWc+LWiWk35ah7ubBUmvqFOnWosWzaKTp3ONq/Zt28DsbFdOHZslxczE5HySMWc+DVfaTXlK3mI76hQIYSpU/szalQXVywhYR+xsV3YteszL2YmIuWNijnxa4W1lPJ0qylfyUN8i8PhYNiwu5g+fRAVKzr/LqSlJbNkyUP861/vUs4XoJUr+llJWXHH3yUVc+LXfKXVVEnycGerMPFNHTpczfLlo7j00tzG39l89tkE4uKGk5Ghx+u+Lj09mKws3WWXspGdnUFmZmCZXlPFnPg1X2k1VZI8StoqTMqHZs3qEhc3jlatznYg2b59BfPm9SQp6YgXM5PzOXiwJidPHiczM1V36OSCZGdncfp0AkePFt33tqS0z5yIiAelpWXw/PPvs2TJBlescuXa9Okzh6go39xQViAiIpG6dY8SHHxx9fyUspeUFM7evfXIzj73flpp95lTMSci4mHZ2dnMnfslL7+8nKws5/8HBwWFcffdk7jiiru9nJ2IeEtpizk9ZhUR8TCHw8HDD7dl3rzHiIioAEBGRiorVgzliy/eIDs76zxXEBE5S8WciIiX3Hzz5Xz44VgaNqzlin399WSWLRtEWlrhTc5FRPLy2GNWY0xb4HUgAzgK9LPWxuc5Xg+IAarkjHnMWrst51hv4EkgFdgFDLbWFmcJWDR6zCoiPi4xMYVhw2bz1Vc/uGK1al1Onz5zqFq1XhFnikh54tOPWY0xYcB0oKe19gbgn8Cr+Ya9A0yx1rYGhgOLcs6tA4wHbss5NxEY4Ym8RUQ8oUqVisyePYQBA9q5YkeO/EhMTEf279/kxcxExB8Eeeh97gQ2WGsP5LyOASwwGMAYUw243Fq7GsBau9UYc8oY0zTn3CXW2hM5584EYoGJxXjfnI1ctH+XiPi2oCB49tn7ueaaK3jrrVWkp2cA8MUXo2jbdowWRohcBMLDXV+WaCM6TxVz0cCe3BfW2iRjTJAxJthamw7U59zbifuBS3PO3T6EZwUAAAeUSURBVFpAvDjqAAQHtylV0iIintatW3u6dRvn7TRExLvqAD8Xd7CnirlQnPPg8soAsos4nglkFXAsN14cm4GbgV9zzhMRERHxVYE4C7nNJTnJU8XcQeCm3BfGmHAg1Vqbked4/XznXIbzbl3+Y7nx4jgDrC95uiIiIiJeUew7crk8tTXJx0AHY0zu+vtB5CxwALDWHgJOGGNuBTDGtADSrLV7gRXAAzkFIMAQYIGH8hYRERHxaZ7cmqQT8ALOR6R7/r+9e4+xqyrDOPyb1ArFUrVIqw0KCcZXmipaYmkg1UobaVKDiBdMlGjQNsYGJeESRGyiEi5RCQHUKkRBrqkKIdg0INFqRaIVuZgUXkIQDVbuRi3D1NqOf6x9kuPJdOacTnF373mfZDLZa6919pdMzjnf7L3W+oBVwKnA07Y3SHozcDXlseoIZfuRR6uxn6YslthNufX4Bdt5bBoRERFTXtvLeUVERES0WipARERERDRYkrmIiIiIBksyFxEREdFgSeYiIiIiGizJXERERESD/b82Da6FpPcCl1IqSDwDnG77hXqjin5JOh74OvA52w/UHU/0T9JXKBuFz6BUYDnd9j/qjSr6JelK4ChgFqWO9mrbL9UbVQxC0nuATcBru2qbx35M0rcpn5udPOUq2z/uZ2xr78xJOhBYB3zY9nHAL4GL6o0q+iXph8DZlC+TaJ5HbC+r3ntbgfPrDigGcoHt5bYXUcoufrTugKJ/kmYAa4Hn644lBjIbWGN7afXTVyIHLU7mgBOBe2z/pTq+BjipxnhiMJ+1/UHguboDicHZvrnrcAul1mA0ROcuqqRZlL/dg/VGFAO6EvgGsL3uQGIgs9nL77w2J3NHUCpNAGD7X8ArJE2vLaLom+3humOIyaveb58H1tcdS/RP0gmSfgv8Gbgz0xyaQ9KXgG22N9YdSwxsGnCNpHskXVzdYe1Lm5O5Ayhz5br9h/LIICJeZlWJvp8BP7L907rjif7Z/rntY4HDgYWSzqw7ppiYpE8CCyilM6NhqqkpxwPLKFOMvtbv2DYnc08Cb+ocSHoVMGK7N8GLiH1M0knAdZT5H9+tO57YO7b/CXyH8uUS+7/zgLcB90t6AJgHbJY0v96wYhC2Ryhz/hf1O6bNq1k3AhdKmmv7aWA1cFPNMUW0nqS5lFXIi7KCtXkkHQJMt/2UpCHgZGBzzWFFH2wf1X0s6QlgSVazNoOkQ20/W73vPg7c2+/Y1iZztv8u6QzgDkm7KfPnVtUcVsRU8E7gdcDtkjptL9g+pb6QYgAzgfWSdgG7KDsBfLPekCKmhA2SdgJDwH3AOf0OHBodzRSyiIiIiKZq85y5iIiIiNZLMhcRERHRYEnmIiIiIhosyVxEREREgyWZi4iIiGiwJHMREeOQNEvSB/bRa71lb6opSFohadO+iCEi2qe1+8xFxNQi6VpgOWWPptnAMcBdXV3mAMfavr9n3MGUsjkrKZ+J24ALbP+i6jKPss/a7X3EcCtwXFfTUHXdg2y/RKlK8zHg8p5xhwLfAt4H7ACuBtba3j3RNSMiksxFRCvY/hSApCOAe21vBQ7rnK/KG+0cY+gPgL8CC2zvkPQu4DZJK20/OGAM/7MxsqRlwIVVIjee64FHKInjLOAnwFmUShoREePKY9aIaJsjgYclLZR0clf7K4GR7o5VzeYVwFm2dwDY3kK5M/aRyQQhaSZwGfDlCfodDaiKYdj2U8BngHMk5R/uiJhQkrmIaJvlwCZgIaWuaMdsoLdW7C7Ko9DpPe0zgH/vbQBVfdo7gett3z1B96OBu23v6jTYfhgYBt64tzFExNSR//oiolVsf1HSgcAnek4tAJ7v6Tsi6UbgWknnAs8BJwKnAUsGvbakV1NqQK+hzLu7sY9hc6rr9nq2OvenQeOIiKklyVxENJqk1wO/H+f8kz3HAIttd9rXAGcANwCvAf4ILLf9+IBxLAFuBW6mLLR4ZoxujwJX9LT9DZg/Rt83UBZjRESMK8lcRDRaNcfssN52SauAabbXTTB+J2Vu22V76LKNshhhojg2SzoceCvwUJU0juU+4Kau498Bl0qaXsWCpGOAUcrCjIiIcQ2Njo7WHUNExKRJ+iqwuqtpZvV7e1fb922fP8bYDZStTOZQHm92PhgPpsyrGwZOsf2bSca4FLjE9uKe9huqa54LHALcAlxh+3vV+RXAebaXTub6EdFOuTMXEa1gey2wFkDSAcBDlM+4t9t+cYKxK6tx24EjbW+vjq8CHrN9+Xjj94HVwEXAr4EXgXWdRC4iYiJJ5iKiVSTNA64DbqOsUr1L0mmDzoGbxPUXA78CxpozB/CH3gbbw8CZ1U9ExECSzEVE41V34j4EnAC8m1I94Zbq3KnARklbKBUh1tse2eOL7Rtbbb/jZb5GRASQOXMR0RKSzgYeB+7oLCToOjcNeD8w3/bFPecOqsbB+HPmLunncWt1Z24zPdug9Fhs+4mJXqvrNTNnLiL2KMlcRMR+rqofO9f2Y3XHEhH7nyRzEREREQ2Wcl4RERERDZZkLiIiIqLBksxFRERENFiSuYiIiIgGSzIXERER0WBJ5iIiIiIa7L8CoenTkrmKrAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "a = -per_clf.coef_[0][0] / per_clf.coef_[0][1]\n",
    "b = -per_clf.intercept_ / per_clf.coef_[0][1]\n",
    "\n",
    "axes = [0, 5, 0, 2]\n",
    "\n",
    "x0, x1 = np.meshgrid(\n",
    "        np.linspace(axes[0], axes[1], 500).reshape(-1, 1),\n",
    "        np.linspace(axes[2], axes[3], 200).reshape(-1, 1),\n",
    "    )\n",
    "X_new = np.c_[x0.ravel(), x1.ravel()]\n",
    "y_predict = per_clf.predict(X_new)\n",
    "zz = y_predict.reshape(x0.shape)\n",
    "\n",
    "plt.figure(figsize=(10, 4))\n",
    "plt.plot(X[y==0, 0], X[y==0, 1], \"bs\", label=\"Iris-Setosa 아님\")\n",
    "plt.plot(X[y==1, 0], X[y==1, 1], \"yo\", label=\"Iris-Setosa\")\n",
    "\n",
    "plt.plot([axes[0], axes[1]], [a * axes[0] + b, a * axes[1] + b], \"k-\", linewidth=3)\n",
    "from matplotlib.colors import ListedColormap\n",
    "custom_cmap = ListedColormap(['#9898ff', '#fafab0'])\n",
    "\n",
    "plt.contourf(x0, x1, zz, cmap=custom_cmap)\n",
    "plt.xlabel(\"꽃잎 길이\", fontsize=14)\n",
    "plt.ylabel(\"꽃잎 너비\", fontsize=14)\n",
    "plt.legend(loc=\"lower right\", fontsize=14)\n",
    "plt.axis(axes)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 퍼셉트론의 약점: XOR 문제\n",
    "\n",
    "1969년 Marvin Minsky와 Seymour Papert는 '퍼셉트론'이란 논문에서 퍼셉트론의 심각한 약점이 있다는 것을 보였는데, 그 중에서도 가장 결정적인 것은 선형결합인 퍼셉트론이 배타적 논리합인 XOR 분류 문제를 해결할 수 없다는 것이었다. \n",
    "\n",
    "![or-vs-xor](./images/or-vs-xor.png)\n",
    "\n",
    "\n",
    "\n",
    "이를 계기로 인공 신경망은 암흑기를 맞이하게 되었다. 하지만, 단일 퍼셉트론을 여러개 쌓아 **다층 퍼셉트론**(**MLP**, Multi-Layer Perceptron)을 통해 XOR 분류 문제를 해결할 수 있었다. \n",
    "\n",
    "![](./images/xor_gate05.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.1.4 다층 퍼셉트론과 역전파\n",
    "\n",
    "위에서 살펴 봤듯이 다층 퍼셉트론(MLP)은 아래의 그림과 같이 입력층, **은닉층**(hidden layer)이라 부르는 하나 이상의 TLU 층과 마지막으로 **출력층**(output layer)으로 구성된다. 인공 신경망의 은닉층이 2개 이상일 때, **심층 신경망**(**DNN**, Deep Neural Network)라 하고 이를 학습하여 모델을 만드는 것을 우리가 익히 들어온 **딥러닝**(Deep-Learning) 이라고 한다.\n",
    "\n",
    "![](./images/mlp.png)\n",
    "\n",
    "\n",
    "\n",
    "이렇게 여러층을 쌓은 MLP를 통해 XOR 문제를 해결 했지만, 층이 깊어질 수록 증가하는 가중치 매개변수의 수로 인해 다층 퍼셉트론을 학습시키기에는 오랜 시간이 걸리는 문제가 발생했다. 하지만, 1986년 **역전파(backpropogation)** 알고리즘이 등장하면서 계산량을 획기적으로 줄일 수 있게 되었다. 역전파법에 대한 자세한 내용은 [여기](http://excelsior-cjh.tistory.com/171)를 참고하면 된다.\n",
    "\n",
    "역전파법을 간단히 설명 한다면 다음과 같다.\n",
    "\n",
    "1. 먼저, 각 학습 데이터 샘플을 네트워크에 입력으로 넣어주고 출력층까지 각 층의 뉴런 마다 출력을 계산한다. 이를 **순전파**(forward propagation)이라고 한다. \n",
    "2. 그 다음 네트워크의 마지막 출력층에 대한 결과(예측값)와 실제값과의 차이, 즉 오차(error)를 계산하는데, 손실함수(loss function)를 이용하여 계산한다.\n",
    "3. 그리고 이 오차를 역방향으로 흘러 보내면서, 각 출력 뉴런의 오차에 마지막 입력 뉴런이 얼마나 기여했는지 측정한다. 이말을 쉽게 설명하면, **각 뉴런의 입력값에 대한 손실함수의 편미분, 그래디언트(gradient)을 계산**하는 것을 말한다.\n",
    "4. 3번과 같은 방법을 입력층에 도달할 때까지 계속 반복해서 역방향으로 흘러 보낸다.\n",
    "5. 마지막으로, 계산한 그래디언트를 네트워크의 모든 가중치 매개변수에 반영해주는 **경사 하강법 단계**를 수행한다.\n",
    "\n",
    "\n",
    "\n",
    "따라서, 먼저 순전파를 통해 네트워크가 예측을 출력(출력층의 출력 결과)하고, 이를 손실함수를 이용해 오차를 계산한 뒤, 역방향으로 거슬러 올라가면서 각 뉴런의 입력값에 대한 손실함수의 편미분을 계산하고(backpropagation), 이를 경사 하강법을 이용해 가중치를 조정하는 방법이 역전파 알고리즘이다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 활성화 함수 (activation function)\n",
    "\n",
    "역전파 알고리즘이 잘 동작하기 위해서 다층 퍼셉트론(MLP)의 구조에 변화를 주었는 데, 그것이 바로 활성화 함수 부분에서 계단 함수를 시그모이드 함수(로지스틱 함수)로 바꿔준 것이다. 이렇게 활성화 함수를 시그모이드 함수로 바꿔준 이유는 가중치 매개변수를 조정 해주기 위해 그래디언트, 편미분을 계산하게 되는데, 계단 함수는 0을 기준으로 기울기가 없는 직선이므로 그래디언트를 계산하는 것이 의미가 없기 때문이다(0을 기준으로 불연속이기 때문에 미분이 불가능한 이유도 있다).  활성화 함수로는 아래의 그림처럼 로지스틱 함수 외에 다양한 활성화 함수를 사용할 수 있다.\n",
    "\n",
    "![](./images/activation02.png)\n",
    "\n",
    "\n",
    "\n",
    "위의 그림에서 **ReLU** 또한 0에서 연속이지만 첩점(뾰족한 점)이므로 미분이 불가능하다. 하지만 0보다 큰 경우에는 미분을 적용하고 0 이하인 값에는 0을 줌으로써 해결할 수 있다. 또한, ReLU가 성능이 좋을 뿐만아니라 'cs231n' 강의에서는 실제 생물학적으로 시그모이드 보다 그럴듯한 작용을 한다고 한다.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def logit(z):\n",
    "    return 1 / (1 + np.exp(-z))\n",
    "\n",
    "def relu(z):\n",
    "    return np.maximum(0, z)\n",
    "\n",
    "def derivative(f, z, eps=0.000001):\n",
    "    return (f(z + eps) - f(z - eps))/(2 * eps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAApgAAAENCAYAAACrchWkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xl8nFW9+PHPbJnse5qm2du0p3tLF0qhBQqySxXBq4go4nKVKxe8/lRUBEUuXq4/rwqICwio/BAVLgIFxIUdhLbQlbYnaZu9afZ9n5nn98czM03ayTLJzCSTfN+vV1+deZ7nPOc8mfT0O2e1GIaBEEIIIYQQoWKd6gIIIYQQQoiZRQJMIYQQQggRUhJgCiGEEEKIkJIAUwghhBBChJQEmEIIIYQQIqQkwBSTopT6gFLqlQDH71ZKfXQC90tVSn1qlPMfVEr9PsDxO5VS3w0yr1eUUh8IcHyvUio/mHsJIUQ0UEo9opT6XIDj25RSZ01FmcTMZJ/qAojoo5T6b2DrSccODXl7PpANpIxyj18Au7XWvzjp1FzgDuC3Q65dCiR73y4F5imlzhiSZtc4yx0PnDfkUAawQSkV631fq7XeBaQDtnHc73HglQDPIIQQYeOts3qB5lEum6+17lBKWYDLhhzPA1YqpT7ofd+qtX4TSAUc48j7vwC01rdMqPBi1pAAUwRNa/11pdTtwL8Aa4F4oBZ4Umu9F0ApNdZtEoDEcWZ5LbBiyPtO4NYh768f530SgA8Pef8OUOz9A7Cd8QerqcC5QBIgAaYQItLcWuvMcVxnYXi9V4FZZ/uOHQHeHE+GSqkY4FLAo5T6jtZ6cPzFFbONdJGLiforsBB4HPgZ8C7wqFLqgnGmXwecA6CU2qCUalJKNQH/PPlCrfU3gcuBXwEaqAH+AVyrtf6g1rphPBlqrRu11p8D7gUMb/nnAWXATVrrX43nPt4W1b8BPwD6lFK/UUqNp6IXQoiI0lp7vPXeXUAPMB8oAI4D39Ba/2A891FKFQDbgBeBl4FnvMeECEhaMMVELcDsyv6n1tpQSh3FbMUswQy+AG5TSn0ZuE1r/YwvoVLqk8BhIEspdYXW+ikg03tuMfCXAPk9iFkp/hzoAs4G9iilNmqta8dbaG9w+CzwFeB7mC2QNwJ/AD445NKnlFL9wOVa60alVAmwGTPQzQFu1lq/qZS6B/gi8JZS6lXMloC/aq2PjbdMQggRTkqpOZhB4feAH2F2hV8DvKyUWqW19m3pd79SqgP4gtZ6r1IqF7On5kJgNfAdX13uHWP/vFJqL/AK8JLW+nAEH0tMcxJgiom6APgWcK+3u7gMeBr45ZBrfgU8AdT5DiilLvSm24L5+/d3b+X3gNbaEygjpVQyZnd8htZ6wHv4r0qpucCVwD3eY4neY11a664Ryn028LzW+skh9/8K0KaUsg4pw82YAXOL9/1coAi4W2v9ji+tt2L+uVLqQcyuo7OBF0bIWwghpsIa4KDW+qEhx76nlPoCkIvZKwTwn5hDh3zvk4CVmGPirxsSiKK1/pNS6gngA94/b4T3EUS0kQBTBMU7YDwXcxzkdzBb9W7EbBF0AhcopXzfYo9rrQ8NSXsH5rifS7TW9d5jm4FHADdmK2UgPcAAsBjwjfF0AsswWyN9PgVc7L3PT0a41z+AbymlrgN2Yo4D/VfgmZMC3EqtdYXvjdb6DUapQL1jkZ72/hFCiHCzeYcVjeQjWuvXvK93AIuUUjdhtmTagY8B1VrrmiFpaoe2Qnrr72+MlIE34PwbJ3qthPCTAFNMxCOYYxhdmIHfUeAmzFmN3ZhBVifQd1K6XwJ3aa39x7XWTQzvmu7mpEBOa+1SSl2N2W1dgdlFvgJ4XGs9NKC7X2v93dEKrrUuU0qdD3wBcyZ8N/C695l82gF/sKmUOh14hlNZMX8OxskntNZzRyuHEEJMlLcOtfjeK6XuBOwjzezWWjcrpTZhDue5DRjEDDqHjpnvxKzTfffMIfCkR9/cjUA9Tuu11tVBPIqYwSyGccr/jUKMi1LqWqBwlEu2aa13j5B2DWY39BmYX3QMzMDxSeAerXVbgDQWzMDwKuCyId3lvgrWNVaAGUpKqUcxlykaqeVVCCHCbqwAMwx5obW+daxrxewmLZhiMjTQOMK5mzFnKZ4SYHqDy79htnp+Xmvd7z2eg7n80EtKqXUnj8n0TiZqBrq11gNKKSvmchtB/x4rpQzMMZaBzAVKhnaRCyFEtPP2ADkxhySdLAu4SGv9SgSLJGYwCTDFZHwBc7mhQPIxJ/gEcgHwstb60aEHtdZ1SqmvYbZk5gC1SqkNmK2aFsyuGTtgVUrVYnbn9GKOq2wNtvBa67xAx72VsBBCTCujdFsnes9fF+Dcyd3WGwN9eQ60I5sQkyEBppiMEszlel4JMt2LwNe9yxU94RuTqZTKA24H3gOOAXhnbAcMBIfyddsIIcRMpbWuw+xhEWLakwBTTNZvlVI9I5zbrrU+ZV9xrfVu74LsX8FcKsOBOQazHbO18v8MXQ4jXJRSx0c4lRXuvIUQYorsUEoF6iJPj3hJxIwmk3zEjOBd/xKt9UhBYzjyXAS0+5ZcEkKImc63e4/WumqqyyKmNwkwhRAigpRSZwE/BG44eZUFpdRHgC9zYrzxDVrrvZEvpRBCTI50kQshRIQopX6LuTtK8giXWICLvasknA88AGyIVPmEECJUpmOA6QTWY24vGGiciBBCTIQNc3WCHUD/FJXhi1rrnpFm7A7dwhSznDlB3FvqTiFEOEyo7pyOAeZ6zJ1VhBAiHDYzRfsma61HmhAXyP8B/hjE9VJ3CiHCKai6czoGmHUAra3deDyRGR+akZFIc3NXRPKaCvJ84dPV0c+ONyooKsmgeFFmWPKQzy80rFYLaWkJ4K1jpiulVBLwU8xWyC8FkVTqzhCT54tuM/n5IvlsE607p2OA6QbweIyIVZK+/GYyeb7wOHywgQO76+jvc1FYkhG2fOTzC6lp232slFoIPArcpbV+OsjkUneGgTxfdJvJzzcFzxZU3TkdA0whokbJ0jnYY2ykpMZNdVHEzPB7zHGaO6e6IEIIMRkSYAoxCfEJMSxdFcw8DCGGU0pdD9QDLwGrgf+rlBp6yadkzUEhRLSRAFMIISJMa33ukNcPDTkldbIQYkaQykyICXrjb4cxDIPVG/JJSomd6uIIIYQQ04Z1qgsgRDRyuzwc2nec/e8dQzbDEkIIIYaTAFOICaitamNwwE1GVgLJqdJ6KYQQQgwlAaYQE1Be1gRA0cLwLU0khBBCRCsJMIUIkmEYVJY1A4RtcXUhhBAimkmAKUSQGo930t01QEKSk8zsxKkujhBCCDHtSIApRJAqvK2XRQszsFgsU1waIYQQYvqRAFOIIJX7Aswwbg0phBBCRDMJMIUIQkdbLy2N3ThibOQWpE51cYQQQohpacILrSulzgJ+CNygtd590rl84EEgBXABXz75GiGika/1smB+Oja7fD8TQgghAplQgKmU+i2QBCSPcMn9wL1a621KqbXAY8DSiRVRiNEZHg+GyxWRvCpKGwEoWpAWsTw9LlfE8poKoXo+Y8iK98PWvve9sckXAiGEiJSJtmB+UWvdo5R65eQTSqk0YInWehuA1vpdpVS3Umqx1vrQJMoqxCkGjtex/eb/xNXVFfa8Bq0x1BVfjQUYvPd2yjwDYc8ToGyS6T1Y6Lc66LM56bXG0GdzMmix4bLYGbTaGbTYGbTavH/bcVnsuCxWPBYrHqy4fa8tVtxYhrz2HbfgwYphsWAABubEJwMLhmXIa/BeM/Q8wIl0Jx/z8aUBYIITq+akxfHrWy+cUFohhBDBmVCAqbXuGeV0AVBx0rFKIBcYd4CZkRHZ5V+yspIiml+kzdTna3h/lz+4tNhsYc3LY4thXmcZbquDGIsbwpzfmOXBQps9gQ57Ap32eDpt8XTY4+myxdFpj6fPGuP/M9GgLKoZBszCxxZCiOlgwmMwR+HEHHc5lBvwBHOT5uYuPJ7IbPKclZVEY2NnRPKaCjP5+To7zO86SWdsJOdz/xr2/Fb4X30m7Hn5ZGYmcvBwI+V1HVTWd3K8uYfjLT00tPbiHue/kXinnfhYOwlxDuKddmJjbMQ4bMTYrTgd3tcO72u7FbvNis1mwWa1YrNaTry2WbBbzdceXPS4e+hxddHt6qbL1UW/u5/zC87GYgGrxcLvDvyBmq46etw9GHjw91dbYGPOOq5ceDlZWYnsqjjET977hTcgPHGN7/U31v07+cm5ADx28AneqtuBw2rHYXUQY3OYr20O8pNy+fTSjwPgMTw8uP9R7BYbNquN9NiUkH0mQgghRheOALMGsxVzqEJObdUUYtIMb4Blscyc8XUew6CqvpP9R1vQ1W1U1XfS2TMY8Nr0ZCeZKXGkJzlJS3KSmuQkPclJaqKTxDiHP6C0WoNryvMYHjoGOmnqbaGpt5mcpFxyE3MA2HF8F38ofYpeV1/AtFcuPx+H1axaXI5ueqwtYIVYWwyJjkTiHfHE2ePITckgPtZOfKyDuUkZbF14IbE2J7F2p/fvWJw2J05bDJlxGVi9rbBXL/4In1hyJdYxPnOrxcoXVnzqxPsgfwZCCCEmLuQBptb6mFKqTSl1rtb6FaXUGmBAa10e6ryEGN7aFT6NxztpbuymqCSD2DhHyO/fN+Bid1kTuw83caCila7e4QFlUryD4pxkCrOTyM1KYG56PNlp8ThjQtNNbxgGTx15jrruepp7W2jua8XlOdERcfn8i/0BZozNQa+rD7vFRoozhRRnMinOZFJjkkmMSRw22ea6pVdjt9qId8T7g85AkmOSuLjovHGV1Wad2qEJQgghxhayAFMpdT1Qr7V+DvgU8IBS6i6gD/hcqPIRYhhfMBPmFswDu+s4sLuOtWcVcvrmopDc02MYvF/ewut769h7uIkB14lRJBnJsSyfn87SonTWLc+BQdekdg3yGB7qexqp6qihpusYdd31dA928431NwFgsVjY0/g+Tb3N/jRJjkQy49LJiEsnJ2GO//iSdMXdm28nwR4/Zpky4tImXGYhhBDRa1IBptb63CGvHxry+jCwZTL3FmJcfGMQwzyJJScvhY62PooXTn73nt5+F6/vOcZLu2ppaO31Hy/JS2H94jmsmJ9BdlqcP3jLSouf8BjaQy1lPF/+d6q7ahlwnzrrvWewh3hHPAAfWnAJVouVrLgMMmLTibU7A94zxmaOexRCCCFGEo4xmEJEjK871hLm8XWLlmezaHn2pO7RN+DiH+/W8OL2an8XeEayk3NW57Jx2VwyUmIndF/DMGjoaeRwWzmH28tZlLqAjfPW+88faTdHp6THplGQlEd+Ui7zErLJSZhLrP1EnmvmrJzE0wkhhBAnSIApopvh7VaexsvweAyDN/fW8cSrR/yTdUryUrjk9AJWlWROaPJJz2Avh1rLONCsOdCsaR/o8J/rc/X7A8yi5HxuWPVZCpJySYqJ7NJfQgghZi8JMEV0i8AYzF1vV5Gdm0xOXkrQ4yCr6jv53V81R2rNAHDBvGQ+vHk+S4vSgr6XYRj+NL858Dj7mw/6zyU5EilJLaYkdT6L0hb4j8faY1mWoYLKRwghhJgsCTBFVPPPWA5TC2ZHWy9vv1JOjNPGdf9+Jjbb+PLxGAZ/eaeKp147ittjkJIQw8fOK2HD0uygAsu67npeqX+VNyp28nH1EUpSiwFYnrmEPncfS9MVSzMWk5eYM6lJQEIIIUQoSYApopt/HczwBFflpeas6oL56djGuZd1W1c/D247wIGKVgDOW5PLR85eQHzs+P65tfS18k7du7zbsIe67nr/8T2N+/0B5ubcM9ice0YwjyKmCaXUWcAPgRu01rtPOpcPPAikYG5Y8eWTrxFCiGggAaaIbr4xmGGa5FNxuAmAooWZ47v+eAc//dNe2rsHSIxz8NnLlrCqZHxpAR5+/zHerd+D4V3fM94ex+n5q1mavASVVhL8A4hpRSn1WyAJSB7hkvuBe7XW25RSa4HHgKWRKp+Yvfp6BzGC2DzPYmHYmsC+9M7YExs7DPS7cLuD25EvUPqYGBs2u/kFf3DQjWtwfBsD9nQN0NszGDC93W7F4V1H2O32MNDvDqqcgdJbrRac3oYEwzDo6z15U8PRBUo/0s/Z92xjCcXnNN7GlZNJgCmiWxjHYPb1DlJX3Y7VaqFgfvqY1+8ua+IXz+xnYNDDovxUvvihZaQmBl7qx+d4dz0pzhTivLO5k2ISsVmsrMpazoactSxOW8jc7NQZu9XnLPRFrXWPUuqVk08opdKAJVrrbQBa63eVUt1KqcVa60ORLqiYmQ5UtPDEK0eobujCGWPj7JXz+OCZhfzvI+/R2R54d65AnLF2rr/5LP/7J7zpr/ni6SSnxgHw8vOao7opqPIFSn/hh5eyYHEWAPt21vLOq8Ht2xIo/Wln5HPGufMBOFbVxrY/7AvqnoHS5xWlcvnHVwHQ3+fikXveCuqegdKP9HMer1B8Tilpcdx06/lBPQtIgCminH+ZojB0kVceacEwILcwxf+tciSv7K7ldy9qDAM2LpvLZy5djH2Eb30ew8PBllJern6Dgy2lfFx9xN/dfWHhFi4p+gAJ3rUpxcyite4Z5XQBp26pWwnkAuMOMDMyIrtaQFZWUkTzi7SZ9HzPvHaEB57eD0AqkNTn5s3tVewrb2Z1vAPX4Phb8WLjHMN+NolJTlyDbjIyEklNN+uv5OQ44uKDWzM3UPq0tHh/Xqmpwd8zUPrklDj/se72/qDvGSh9YlKs/1hvz0DQ9wyUfqSf83iF4nOKjZtYqGgxgmkTj4wioLy5uQuPJzJly8pKmtEtRDP5+VpeeJ6mJ/9I2kWXkPXRj4X03i8+9T5HdRObLihhxdrcEa97bc8xHnnB/P//Q5uK2XpWUcCAd9Dj4p26nbxU/Tr1PY0AOKwOLiu+gAsKzx3x/jP584PIPZ/VavEFX8WcGshFlLcF8+ah4yuVUqcDd2itLx5y7A/AL7TWL4/jtkVI3RlSM+n5th+s5xdPvw/AhzcVk9w9yMFddbQn2Cnt7qdobhLfuGYNTsfM2Yp1Jn1+J4vks0207pQWTBHdwrQOpsvloepoCwBFJSPv3vPmvjp+4w0uP35eCReeXhDwuh3Hd/HU4ef861WmOlM4J+9Mzpq3QVorhU8NZivmUIVMcTAsol979wC/+YsG4KNbFnDJhkJqK9vIz0/DGmvj5y8eouJ4J9vequDKcxaMcTchxie8GzgLEWYndvIJ7a9ybWUrrkEPmXMSSRphh52dhxp46LmDGMBHz10wYnAJ4DbctA90kJuYw2eWfYI7Nt7ChYVbJLgUflrrY0CbUupcAKXUGmBAax3cgDMhTvKHl8ro7XexYn4GF3vrqdzCVM44Zz7F89P5wtZlAPzlnSqONXVPZVHFDCItmCK6hWkdzIoyc3miohH2Hq843sGD2w5gAB/eXMwlZxT6z7k8Lt48tp1+dz8XFm4BYH32aSQ6EliWsVjWqxTDKKWuB+q11s8BnwIeUErdBfQBn5vSwomoV17Xwdvv1+OwW7nmwkUB65+S3BTOWT2PV3cf448vH+bmj66agpKKmUYCTBHdwhBgGoZBxeGRA8y2rn7ufXIfAy4Pm1bmcPmZRYA5eWdn/W62Hf0rzX0tOKx2NsxdR4ozCZvVxvLMJSEro4huWutzh7x+aMjrw8CWqSiTmJme/2clAOevzWOOd9YwwPHaduqq2olPiiElLY4rzp7PP/cfZ++RZqobusifI1vLismRLnIR1QxP6MdgNtR10tM1QGKyk8zs4ZXsoMvNvU/uo7Wzn4V5KVx7ocJisXC0vYIf7ryP3xx4nOa+FubGz+G6ZZ8gWfb/FkJMkbrmbt4rbcRus3Lh+vxh5w7uOc6fH9tNbVUbAMnxMZy9ah4AL7xdGfGyiplHWjBFdAvDGMyujn5i4xwUlWSc0p30p5ePUF7XQUZyLP92xQqwuHn4/cfZWW9OBk6JSeby+RexIWct1jDujy6EEGP5245qDGDTirmnrMnrG79uHVLHXXR6AS/vqmX7wQY+uqWEtKTR1/EVYjQSYIroFoYu8gWLsyhelMngwPC1xvYeaebv79Zgs1q44YrlJCfEYBgG3YM9OKx2zi84hwsKziXWLpWyEGJq9Q+4efuAudXs+evyTznvX6FwSNWZkRLLaQsz2akbeWNfnX/4jxATIQGmiG5hmuQzdMsugI7uAR56/iAA55yeRnxKvzdbCx9XV2DBQkbc2Lv9CCFEJOzUDfQNuFmQm0xuZsKpF4ywScXZq+aZAebeY1y2sXBYC6cQwZA+PBHVfGMwQzUzu62lh/6+4fvHGobBw88fpKN7gLRMF/90/57fH3rS38WUGZchwaUQYlp5bc8xADavnBfwvGeE7+ZLi9JJT3bS2NbHocrWcBZRzHASYIroFuK9yF99oZRH7nmLmooTFes7B+rZc6QZi81FT+4bWK0WilMK8fgWeRdCiGmkqa2Xspp2YhxW1i+eE/iiEVowrVYLZy3PAfB3sQsxERJgiujmCzCtk2/B9HgMLFYLFquFrLnm3q1NnR08/Nd9ANgLDrIgaw63rL+JDy24BJt15mypJoSYOXZqcyvaVQsyiXMGHgk32uii05dmA7CrtBGXW75Ii4mRMZgiqhkjfAufCKvVwtarVzHQ7yLGaWfQ4+KOp55jsD8bW2IbHz9zLefknSmzw4UQ09qOQ2bL44itl4xed+ZmJpCblUBtYzcHKlpZuWDk7XKFGIn8TymiWxj2Io/xfuOvqe+h61g2WAxu3LqeLfmbJLgUQkxrTW29lNd14nTYWDFKYGh4fAFm4PO+4NQXrAoRLPnfUkQ3T2jGYLpcHuqPdXC0rYJ9TQfwGAaP/lUDcNH6AlYWnLrMhxBCTDfvlTUBsKokA6dj5GE8J7rIA0eYvgBzd1kTbo90k4vgSRe5iGpGiMZg1lS08MIT79OV0kjDsn1cmnod5XWdpCTG8KFNxSEoqRBChN++I74AM3PU6050kQc+n5ORQHZ6PPUtPRyuaUcVpIW0nGLmkxZMEd1CMAazY6CT595+C4DuxFZOz17Hi2/VAfChTcXExsj3MCHE9Nc/4EZXt2EBlhePvnTaWC2YAKtLzC72PYebQ1VEMYtIgCmi2yTHYB5qKeMH7/wEd725+87Fp28gs/s0Gtr6yE6PZ/PKnFCVVAghwupgZSsut0HxvGSS4mNGvXbp6hwu3LqUtMz4Ea9Z7W0F3X24KaTlFLODBJgiuk1iHcyXql/nvt0PMthqwzEYS3ySg9VFS3nmzQoArjx7PrYQ7nEuhBDhtPeo2dK4cv7Ys76LF2VyxjnzSU6NG/GakrwUEmLtHG/pob6lJ2TlFLOD/O8potqJmZDBt2AWJxdgs9pYY5wBwPyFWfz93RrauwcozklmrcoKaVmFECJcDMNg3xEzwBxt9ngwbFYry7xd7fvLW0JyTzF7SIApoluQk3w6Bjr9r4tTCrlj4y1YG81F1XOL03hxexUAV50zP2TbTwohRLgda+6huaOPpHgHhd6NIkZTXd7C3ndr6OkeGPW6ZUVmgHmgQgJMERwJMEV0C2IM5vbj73H7W//F7sb9Jw72OGht6iHGaaOsuZvuPhcLcpNZXCgzJoUQ0cPfejk/A+s46sN336riz4/tpq159K7vpd4A81BVqyxXJIIiAaaIaieW2hj5V9ntcfNE2TP85sDjDHgGKWs94j9X4V0zLr84nb++WwPAZWcUSeulECKq7Dt6IsAcj/yiNJafNo+4hNEnA2WkxJKdFkdvv5uKus5RrxViKFl/RUQ3zygb6gKdA138ev+jlLUdxWax8dFFW9k07wz/+fIys1Lud9po7ewnNyuBlSWyLZoQInr09rsorW7DYsE/ZnIsa88qJCsricbGsYPGpcXp1LfWcqCihQW5KZMtrpglpAVTRLdRxmBWddRw9457KGs7SnJMEjed9q9szt3ob53s7RnkeE07VquFf1aagealZxSOq3tJCCGmi9LqNtweg/k5ySTGOUJ+/6WFZtD6fkVryO8tZi5pwRTRbYQxmB7Dw8PvP0ZrfxvFyQV8bsW1pDqHf/OuPNKMYUBSZjzHGzvITInl9CVzIlVyIYQIiUNVZuAXzNjxjrZe8IDL48FuH72taXFhKhYLHKltp2/AJZtPiHGRFkwR1UYag2m1WLlu2dWcnbuRm9Z88ZTgEqC63KyUjw8OAnDxhgJZ91IIEXUOVbUBwQWYLz51gJ//96u0NnWPeW1CrIOiucm4PQal1e0TLqeYXeRriIhuJ/Y7o2Ogk31NBzhr3gYACpPzKUzOHzHpeZcp0vOS+eVfNXFOO2cunxuJEotZTCm1BbgbcAENwPVa65Yh5/OA+4F4IAG4U2v93FSUVUSHnr5Bquo7sVktlAQzPtIILp9lxWmU13VwoKKFlSFaZ1PMbBMOMMdRUf4L8H+Bo95Db2itb51EWYU4lXeST7W7lf+34x7a+tuJt8dz2pwVYya12azsOd6BC9iyIke6fURYKaVigV8AF2itq5RSXwHuAr445LIfAY9rrR9TSs0B9iqlcrXW7ikosogCpdXtGAbMz03G6bCNO53HCG6TiqWF6Wx7q1LWwxTjNqH/UcdZUaYDD2itvz/5YgoRmGF4OFAcy8vdL+HCw/yUQuanFI6ZzuMx6Ol38c6BegDOW5Mb7qIKcRHwpta6yvv+QUAzvN6sBXxN6RlAEyCLD4oR+cdfFgS3dq9/eNE4N6lYkJtCjN1KTWM37d0DpIyxvJEQE22yGU9FmY5ZOQoRFi6PixezW3l3aTLgYXPuRq5aeDl26+i/1oZh8Ngvt+OyW/C4PCyfn052enxkCi1msyJO9Oigte5UStmVUg6t9aD38PeB7UqpzwEFwAe11kF1ZmZkJIaqvOOSlTX2rjHRbLo/3+HaDgA2rJwXVFltNnO8eUZ6wrjTLSlOZ09ZE/Ud/ZQURUc3+XT//CZjuj/bRAPMIsauKGOBzyulPgWUAbcOCUjHJJVkaM2052vv6+DeN3/NoYwebG6DT2Rv5vJN14wrbX1dB53tfbis4AauPG/RtP/5TPfyTdZMfz4vJ+aQoqGFbGEFAAAgAElEQVRcDB8N9wTwFa31NqVUPvCsUmprMHVnc3MXHk+QA+wmaLzrKEar6f58Xb2DlB9rx26zkJngCKqsrkFz1EVrWw+Ms2e9eG4Se8qa2Lm/jkU50//f7HT//CYjks9mtVomFJNNNMAcs6LUWt8G3KaUsgCfBP4XWDfeDKSSDJ2Z+Hxdgz00drWQOGjlspeaWH5N4bif0Wq3sPayRTz63EGyUmPJz4ib1j+fmfj5DRWp55toJRlCNcAm3xulVALQp7V2ed9nAoVa620AWutqpdQ/gAsxe4mEGKa0ug0DWDAvhZggxl/CsPmR46byUwHQ1W1B5SVmp4muyVKD2X0DnFpRDqW1NrTWvwMKlVKhXwFWzCoe77qXiY4EvrTqej5TmsbcZte4xxH5vFPaSBdw7upcWVhdRMoLwMVKqWzv+y8Ajw053wyglFrh/TsROB/YG8lCiuhxqNIcf6kKUoNOawQ5yQegOCcZu81CTUMX3X2DYycQs9pEWzBfAO5USmVrres5taJEKZWltW70vr4EODqk+1yIoLg8Lv5U9gwOq52rFm4FICchG/eglR6AUfYiH8rt9tDZO8jeI81YLZaglybq7e2mq6sNt/uU71Jh09BgxeOZufM8Qvd8FmJiYklLy5qWe8lrrVuVUjdidnt7MIcZfV4pdT1Qr7V+Til1FfBjpZQdc6mi+7TW26ew2GIa861/uSSI9S99JtKCGeOwUZyTTFlNO2U17awuyQw6XzF7TCjAHE9FCXxXKbUe6AVagY+FqtBidmnv7+DB/Y9ytL0Cu9XOuXlnkRnnHWAeZC25b2ctO/5ZRaoH8ksySEl0jrscvb3ddHa2kpqahcMRE7Egxm634nLN3AAzVM9nGB7a2pro6monKSn4Fp1I8NaNJ69r+dCQ87uBD0S0UCIqdfUOUtPYhd1mZf685KDTT6QFE2BRfiplNe2UVrdJgClGNeGF/8ZRUf7bRO8thM+Rtgoe3P87OgY6SXWm8PkV154ILgm+kqwoa8LV58IANq/MCaosXV1tpKZmERMz/qBURI7FYiUpKY2WlvppG2AKESqHa80ddebnJOGwBzf+EobWncGlUwWpPPfPSkplHKYYg6wsLaYlwzB4teYtnjz8LB7Dw8LU+Vy//BqSY06auegJvBd5IL09g9TVduDBwBNvZ0WQu1G43S4cDln7bTqz2ex4PLImuZj5ymrMAG9h/sS+TJ3o/AkuwlwwLwWrxULl8U7Zl1yMSn4zxLT0Ss2bPFH2DADn5W/mwwsuxWYN8C3dV0uOYw/xyiPNYEAncMaKHOy24Oe4TcexfeIE+XzEbFHm3RN8Yd7EAsz5izKxYMEe5OzzOKedwrmJlNd1cuRYB8uK0ieUv5j5JjqLXIiw2jB3LbmJOVy/7BNcufDywMElwXWRH9Xmuv+tGGxaEVz3uBjZFVdcSl3dsakuhhCzxsCgm/K6DixASW7w4y8BNl+4kCuuOQ1nbPDtTIu8raalVdJNLkYmLZhi2ihrPUJxSiF2q514Rxy3rL8J61izw8c5ycc16Ka63NxDNyU7gXmZCaEoclS7//57ALjhhn8f8ZqrrrqcwcEBrEMC/Lvv/h+6urp46KFfcd99vwp7OYUQw1Uc78TtMcjLSiQ+NvKr/y3KT+XF7dWyHqYYlQSYYsq5PW6eL/8bL1a+zObcM/iYugJg7OASwBjfGMyayjY8boNuDDasnDfZIs8IlZXl42r5/dnPHiQvL3/Ysffe2xmuYgkhxuCbYLMoP2XC92ht7sHiseDBwBrkOsK+bvmjxzoYdHlw2KUzVJxKAkwxpdr623lo/2McaS/HgoUUZzKGYYx/LJ1vs6cxri872ABAOwbrF8+ZRIlnhr17d3PkyGEsFgv79u1hxYpVI167Z88uamqqAVi16jR27HiHo0cPR6qoQoiTlNVMbvwlwJO/eY/BATef/cpZxDiDCwUS4xzkZiVQ29hN5fFOSvImHuiKmUsCTDFl3m/W/PbA43QNdpMck8Rnln2CRWkLgrrHeMZgGoZBeVkzAOm5ySQnzO6Z4Hv27ObOO2/j29/+LlarjTvuuI1vfvM7rFkTeCfXV199ibi4OACUWszrr79Cc3NzJIsshPDyeAwO13pnkE8isEtNj8PjDuLL/EkW5qVS29hNaU2bBJgiIAkwRcR5DA/PHn2Rv1a+DMDitIVct+xqkmImsE+0f5mikbto6o914B5w04/BhtWh7R6v/en/0L0vsjv5JaxYSe5N/xF0uqqqCp599mlee+1lbr/9P1m+fAUAd955N3fccSvr1p3O1q0fYcGCkmHp/v3fvzqsi/zb3/4u7723k4cekvGXQkRaTWMXvf1uMlNiSU+OnfB9rrpuLVlZSTQ2dk4o/aK8FF7ZVUtpdRuXnlE44XKImUsCTBFxFiw09DRiwcIH51/EhYXnjm+8ZSD+ZYpG/hZ+cH89AB0WWKOyJpZPlNu5czt3330nl122lUce+b2/RRLMVsmHH36MJ5/8I9/+9tf56le/zvr1Z/jP33PPj4iNjcPtdtHf38+yZStYteo0//mMjExstuAXehZCBO9E9/jUthr6uucP17TjMQysskSYOIkEmCIiDMOg19VHvCMOi8XCNYs/ypb8zZSkFk/6vjB6F/lh7/JEWfkpIV8UeCIticEKxVaK69adzp/+9MyI52NiYrj66k9y9dWfHHb8m9+8ja6uTqqqqli2bDlOZyxZWVn09vayYcOZAHz9698iNTX4vZCFEMGb7ALroZKREktGspPmjn6ONXaTN2cCPVBiRpOpXyLsuga6eWD/7/jprl/i8rgAiHfETTq4BMZcpsgwDOrtUI+HDevyJp/fLLN27XoWLFjI008/yZo161i2bDlz5mRTWFjEtddeB8C3vvU1mpubpragQswChmH4Z5BPZoIPwAM/ep0ffPMFPB5j7ItH4CuDL+gVYihpwRRh9X7zIR49+Cc6BjqJtTk51nWcguQQBnrG6GMw65p7ONrRR0KsnZVBbg0pTmhsbOCKKy4NeK6lRSb8CBEJze19tHUNkBBrJycjfsL3MQwD16BZd06mZ3thfipvH6intKadLWvkC7wYTgJMERY9g738+chzvHlsOwALUor59NKPkREX4m3FPKO3YO48ZC5PdNrCrAltDTkTNDU1cf311wSV5oEHfkN29lz/+6ysOTzxxLMBr73qqssnVT4hxPiU1pxovQzJmEfL5LZX9Y0DlRZMEYgEmCLk9jcd5LFDT9I+0IHNYuOD8y/kAwXnTHwizyj8YzADTPLp7Rng8PYa0rGwbhavfZmZmckzz7w41cUQQkySf4LPJBZYh6EjiyYXpM7LTCAh1k5LRz9N7b1kpsSNnUjMGhJgipBrH+igfaCD4uQCPrH4KuYlzh070USNMgZz9+464gY8zLFYWVokk1Amo6Ghnq1bLwp4rq2tNcKlEWJ2CtX4S8Pjmxw5ufJYLRZKclPYc6SZsup2CTDFMBJgikkzDIO67np/IHlmzunE2mI5bc6KsLRaDs985DGYx3sHqMTDgoLUWds9Hgp5efm89tr2qS6GELNaV+8gdc09OOxWiuYmTepevp6fUHSzL8pPNQPMmjY2Lg9jY4KIOvK/rpiU6s5a/ue9+/nvnffS2GNO9rBYLKzNXhX+4JLRlyl6r6KFBgw2nJ5/yjkhhIgmZd7Wy/k5yZP+wuzv+AlyD/JAfK2ppd7ueyF8pAVTTEjXYDfPHn2RN2vfwcAgKSaR5r4WsuIjPFPbE3ih9brmbmobu4l32llaFOKJRUIIEWGlIVz/8sQX80nfisK5STjsVo41ddPVO0hinGPyNxUzggSYIiiDHhdv1L7NC+V/p9vVg9ViZUveJi4t/gBx9ikYfzPCGMx/vFhGHhYKCtOke1wIEfVKq80WwkWTnOADoZvkA+CwWynOSaa0uo2ymjZOWzg7d0sTp5IAUwTlj/rPvFVnjsdTaSV8dNGHyEnInrLyGAHGYHo8Bm3V7eRgZWmBTO4RQkS3/gE3VfWdWCywYF4oAsyxd0ALxqL8FDPArG6XAFP4SYApRjV0i0eALfmbqOys5vL5F7E8Y0nIKqhJFBAYXlHq0kZsBgwA61blTFHBhBAiNI4ca8ftMSicm0Scc/L/bY+xAVrQzHGYlbIephhGAkwRkGEYHGgp5fnyv2G32vjKmi8BMC9xLt9cf/PUB5Y+ARZa3/XeMQBi0mKJcdimolRCBKSU2gLcDbiABuB6rXXLSdd8HrgB6ALKtdafinhBxbTiW55o0SSXJ/IZbf3giSjJTcFigYrjnfQPunFKvSuQWeTiJB7Dw76mA/zw3fu4f8+vqeio4nh3A239J2YITpvgEk58FR9SUbYc6wBg4RLpqgm3HTve4ctf/kLE8mttbeHhhx+IWH6hpJSKBX4BXKW1PhN4FbjrpGsuBz4KnKW13gx8OuIFFdOOf4H1vMl3j4P5fTwtI56MzISQ3C/OaSc/KxG3x6DcW/8KIS2YAjAn72yve5d/VL9GfU8jAImOBC4oPJfNuRtx2mKmuISB+cZgWrxjMGtq2rG5DFwYbFwne+MOdeDAfm655asjnm9vb+NnP3uQ5ctXAPDMM0/x4IO/AKC3txcwiIsz9z/++Mc/ySc+ce24877//nsAuOGGfx/1uttv/yZvv/0WTmfssONnnHEm3/rW7XR0dPDcc8/wmc98ftx5TyMXAW9qrau87x8ENPDFIdd8G7hWa90DoLU2IltEMd243B6OHPPt4BOaFsy4+Bg+/vn1ZGUl0djYGZJ7LsxPpaqhi9KaNhYXyth3IQGm8HJ5Bvnfw9voc/eT5kxlS/4mNuWeMW0DS7+TBhNt31Fjvk2MISF+mpc9wpYuXT7qlpGf/ezwgHHr1ivYuvUKAB555EF6e3v50pdunFDelZXl4275vumm/8Oll87I/c2LgKO+N1rrTqWUXSnl0FoPKqXswDzgg0qprYAB/FBr/UIwmWRkJIayzGPKyprcot/T3VQ/n65sYWDQQ25WAiVFoV8GLlTPt27ZXP7xbg2V9V1T/jMbajqVJdSm+7NJgDkLeQwPh1rKeOf4u1yz+CpibDHE2ePYuuASEuxxnDZnJTZrlIyhOSnArKtsxQoUlkR4Pc4Zrquri56e7gml3bt3N0eOHMZisbBv3x5WrFg16vX33vtjHnjg58OOPfXU8xPKe5pxYo69HMqFGUgCZAJzgCNa6y1KqXzgdaXUmVrrY+PNpLm5C48nMg2foWwBm46mw/Nt31cHmAush7osoXy+7GQnAAcqWjhe347NOvUj8KbD5xcukXw2q9UyoS+uEmDOIvXdDeyo38X247to7jPnFSxOX8TGnHUAnJN35lQWb0L8e+paLbS09mDpc+EBNm4omNqCTUPvvbeTb3zjP0hMDFxRDA4OYrcHrhJqa6vp7g4+wNyzZzd33nkb3/72d7Fabdxxx21885vfYc2adSOmufHGr8zUFswaYJPvjVIqAejTWvuCziagS2v9DIDWuloptQNYAow7wBQzi3+CT4i6xwE62np5/MGdpGfEc9Vn1obknqmJTuakxtHQ1kt1QxdFc5NDcl8RvSTAnOEMw+D50pd46fA/qeqs8R9Pj03jrHkbWJquprB0ITBkHcy336nAggWX00ZmWuQWff+3l74+4rmr1UfYlHsGAG/Uvs3v9f+OeO3Pzvtv/+v/2vFTqjtrx7wuGG63m+Li+fzqV48Ela6np5vdu3dhGAZNTU1kZmaOmaaqqoJnn32a1157mdtv/0//uM4777ybO+64lXXrTmfr1o+wYEHJuMrg8Xim1+SyiXkBuFMpla21rge+ADzmO6m1diml3lBKXaq1fl4plQGsAPZNUXnFFPMYhn/pn1CNvwSz48ft8uByeUJ2TzAnITW09VJW3S4BppAAcyZq6WslzZmKxWLBYrHwRuUOqjpriLU5WT1nBeuzT2NR2oKI7BUedkO6yCsPm3uh5xSFriIWsG3b06xfv4Hk5BT++MfHxpyos3Pndu6++04uu2wrjzzye+LiTgT7Si3m4Ycf48kn/8i3v/11vvrVr7Nx4/CW83vu+RG//OXP/L+/FosFm83Gj3/8s7A8X6RorVuVUjcCzyqlPJjjMT+vlLoeqNdaPwd8CfiVUurrmPXzzVrrhqkrtZhKdU3ddPe5SE2MISslduwE45ScGsvnv7qJzMwk2tp7QnbfhfmpvLn/OKU1bVywPj9k9xXRSQLMGcBjeKjpPMbepgPsbXqf2q46bll/M/lJ8wC4atmlNLS0syJzKTG2GbZPrDfA7O1z4ekaxAqcHuGKbbwtiptyz/C3Zo7llvU3+V/b7daQtTSUlh5i69aLRjx/wQUXc+ONX/G/r62t4be/fYhf/vIREhIS+fSnP85ZZ21m1arTRrzHunWn86c/PTPi+ZiYGK6++pNcffUnTzn3ve/9YNTyt7W1ceWVHxv1munMG0Q+d9Lhh4acrwNm5PgAEbzSGt/2kKkhbcG3WCzYHTYcMaEda+9bRqmsph3DMGZCr4OYBAkwo5TL4+KfdTvRrYcpaz1C1+CJ8XFOWwwNPY3+AHPNvBU0OmbmQGffgsEHK9poxkOSw05hiNaKm2nWr9/AK6+87X//61//koGBgRFnhtfXH+cb3/gPbrjhJnJzzSWfvv/9/+I737mFW2/9HuvXbwhLOe+663u8+eZrI57Pzp4bMDgVYqYp846/XBiiBdbDbW56PEnxDjq6B2ho7SU7PX6qiySmkASYUcDtcXOsu576ngbWZa8GwGqx8uzRv9A9aHZvpDlTWZ65hBWZS1mUtgCHdZZ8tB6zZW9/TTsVGGw9PXeKCzRz3HffT7j44kuHTbhZuXI1t932fV5//ZWwBZjf+tbtI56rrKzgq1+d2FJJQkSb0prQT/ABaG/t5aXnDpGdk8yZ5y8I2X0tFgsL81J5r7SR0po2CTBnuVkShUQPj+GhqbeFmq5jVHfWUt5eSWVnDQPuASxYWJ6xhFi7E6vFysWF5+GwxaDSSsiKy5id3RGGgQcLu4+Y4y/XLJLde07W1NTE9ddfc8rx3l7zy8kLL2w75dwDD/yG7373P7HZTu1CW7t2PWvXrg8qr9E8/PDvyMiYE1QaIWa6pvZeWjr6iXfayc0KzY47PgP9Lo7XdPjnSIbSorwU3ittpKy6nc0r54U+AxE1JMCcIoZh0D7QQUNPE3H2WPKTzJa3fU0H+dW+35xyfWZcBsXJBfS5+4i1m+uNnVdwdkTLPN34usfLEwuxdw+Slewkf05kF5mOBpmZmaMusD7VeYVyjKkQM0VZtTn+siQvBWuYGg/CcVvfbHff8kpi9pIAM0Leb9ZUdlRR39NIQ08jDT1N9Ln7Adgwdy2fWmpOXMhLzCElJpm8pHnkJc6jKDmf4pRCkmIkcDqFN8CsSF3GAqzExDpnZyvuLGK325k7N2eqiyFE2B2sagVAFYR+/KXvy7nVGvr6siA7kdgYGw1tvbR09JGeHLrZ7yK6SIA5CX2uPo5119Mx0ElHfycdAx209rXT0tdKS18rX1t3I4kxZtfGazVvsr/50LD0CfZ45sRnMSf+RLduRlw6d226NaLPEbW84y/r7QnEY3D2iuwpLtDss2zZCm655TsRyy83N4/77vtVxPITYqocqjQDzMUFod/X27e6WziaMG1WK4vyU9l7pJlDVa2cuVy+EM5Wsz7AdHvcdA8MXwdsf9NBWvvb6B7spcfVQ89gLz2uXroGulmZtZQPFJwDQHVnLT/Z9csR793S1+oPMFdmLSMnYS7Z8VlkJ5hBZaIjtONqZhvDMGh2JFNljyUh1sr6tTLBJ9Li4+OJj5eB/EKEUlN7L03tfcQ57RRmh36/af8OaGHq8FlckOYNMNskwJzFJhxgKqW2AHdj7qXbAFyvtW4Zcj4feBBI8V7zZa317vHev9fVR79rAJfHxaDHhcv7x2qx+ZffAXi3fjcDQ867PC4G3AP0uwdYmbWMktRiAPY1HeAvFS/R7+6n3z3g/9vlcWHBwj1bfuBfePy58r8N2/VmqIy4E98mU52pFCblk+xMJDkmiaSYJNKcKaTHppEem0ZmXLr/2rPmhWfG7axmGJQmmFtCrirJnBZ73wohxGTpKnP8ospPDUs3tq8FM1xjOxcXmt36vlZYMTtNKMBUSsUCvwAu0FpXKaW+AtwFfHHIZfcD92qttyml1mJuibZ0vHn8/Pk/0dzZfsrxZEcSFxef73//bNnbuA03HWnHccWYYxrjO9OI7UnGuSSJktVmgNna3ENHmQWIBWJxAk7AggW7zc6uHVX+RciLWpczx7WAxCwbqXNiibfHYelx0nsc5lpPBI3J1mQusX4EBjH/eJeidAONDNJI/SnlT0mLo2C+eY/+vkFK32/A6bSzaPmJ7t33dx3D4zFOSTuSkdIvXZ2DzWYGXUd1E91d/eO+JxAwffGiTBKTzElGddXtNDV0BXXPQOnn5iaTNdf8lt7W0kN1+fgqpd7OXprTVhADnLZw7O0LhRAiGpzoHg/P+pe+MZiWMASvAAVzkoh32mlq76OpvZfMlMht3Sumj4m2YF4EvKm1rvK+fxDQeANMpVQasERrvQ1Aa/2uUqpbKbVYa30o4B1PMqd2Ec7W3oDn3jh82P86myUALMzLJzbbwG610bM3js5KO/G5J4LB1L45zKtcPmJ+249WDnlnAxI5bct8Vheau8KUvl/PP948hGepk2WLzat6ewZ542+HT7nXaEqWzvEHmL70KWlxwwLEf758lMEB97jvOVJ6tTzbHyDu21nDsepTA/bRBEqfnpngDxDLS5vYsyNwS+9IAqXfuGW+P8BsqOsM6mcaZ4sly3CxvDgjqHIIIcR0ZBgGh7wTfBYXhn78pZmH+Xe4usitVguL8lPZfbiJQ5VtbFopAeZsNNEAswhzH10AtNadSim7UsqhtR4ECoCKk9JUArnAuALMVety6e0ZHHeBTt9cTEaWOdP6YEIdFVlNqOU5ZGWZgcviRbkMdAS3FMqCRXP86Qfmu1l/ViE5ean+Y/GxMaw/qzCoewZKH5/g9B8DWLuxELdr/AHmSOmzs5OxO8x1DFesySU3yG/DgdLnF6b5f86LlmYTE+RWY4HSB/o5j0dFbRu7KlpJ6K8jLzf8O100NFix26emG36q8o2UUD6f1Wod9u9BiGjS2N5Hc0c/CbF28sK07Jq/BTOMq24sLkwzA8yqVjatlHGYs9FEA0wn5rjKoVyAMcp5NzDuCG/F+ryguok9GDQ2mtshZuYkkplj/sP0HYuJs7Fuc1HAtFlZSf7rThYo/dBrR7rnaAKlH3rstI3B76UdKH1rmzl5KSsriflLsphPcIuQ+9ID/vQj/ZzHazKf08le+/271GFwWV/diJ9fKHk8nilZr3GmrxMZ6ufzeDwBfx+sVgsZGbLcl5jefN3jqiAtbGMkIxJgehs0DlW1yr7ks9REmw1qMFspAVBKJQB9WmtXoPNehZzaqinEhPT2uzhU3QGGwcLB41NdHCGECAl/93iYxl/C0C7y8AV9eXMSSYxz0NLRT2Nb4OFuYmabaID5AnCxUso38O8LmJN4ANBaHwPalFLnAiil1gADWuvySZRVCL/3y1tweQxy+xpJNMY/lEIIIaYrwzBOTPAJ0/hLXz4AljCOvLFaLKh8Xyum7OozG02oi1xr3aqUuhF4VinlwRyP+Xml1PVAvdb6OeBTwANKqbuAPuBzoSq0EO+VNQKwqLtaul5Gceut3+DQoQMANDY2kJqaisMRA8D99z/InDkTW5x+x453+PGP/5vHHnsyZGUVYrarb+2lrWuApHgHuZnhWyc5vzidz391E1lZScOGQoXa4sI03i1t5GBlK2evkn3JZ5sJr4PpDSKfO+nwQ0POHwa2TPT+QozE5faw93AzAAu7qyAxuIlGs8mdd97tf33VVZfzne98n1WrVk9hiYQQI3m/3FxKeklhWli/OFutFqxWm38SZ7gs8bbCHqhowWMYYRtTKqanmT01VcxIpdVt9PS7yEmNJX2wE0s4+3mEECJC9h81vzgvK04f48rokJMRT3qyk86eQarrg1szWUS/Wb9VpIg+u8qaAFhVlAw7Cd9ibuPwkz/tYe+R5ojmuXJBBjd/dNWk71NbW8NPf/ojjhwpw+PxsHr1Gm655Ts4nU5/F/jVV1/LH//4GK2tLZx11tl87Wvfwm4/UW1s2/Y0f/jD/6O1tYUNGzZyyy234XA4Jl02IWYbl9vjH6sY7nV9ayvb2P56OSWL57BiXfi22LVYLCwvTue1PXXsL2+mcK4sHzabSNOPiCqGYbDLO/5ydWEyEL7dKGa6zs5Orr76kzzxxLM8/vhT1NUdY9u2p/3njx8/TlNTI7/97R949NEneO+9d/n7318cdr6xscF/ft++vbz44vNT8ShCRL3DNe30D7rJzUwgzbsZRbj09Q5yvKaD5obusOYDsMwbLPu6/8XsIS2YIqpU1XfR0tFPSmIMhRmxVEJ4p0KOIRQtiWMJ1zqYixebu2A1NjZQU1NNWloa5eX+/ROIiYnh05/+LBaLhdTUVM4++1wOHTrIxRdf5j3vOOn8Fg4fLgt5OYWYDfZ7A7BIdI/PK0jlw9esJmdeStjzWlqUhsUCZTXt9A24iI2RsGO2kE9aRBVf6+VpJZn42i2lBXNi3n77Le677yfMmZNNXl4ePT29pKScWPIpPT0dq/VE8J6UlMTx43VDzmcMO5+YmDjsvBBi/PaXm0Ntls8Pf4AZF+8gLj5l1E1GQiUh1sH8nGSOHOvgUGUbqxdmhjU/MX1IgCmiim/85eqFWeBb/1JmJk7ID35wB9///n+xcqU5q/zee/+H7u7wd5kJIYbr6B6gqr4Lh93Korzwb3sbacuK0zlyrIP3y1skwJxFZAymiBpNbb1UN3ThjLGZy1/4t6OQX+OJGBwcpKOjA4AjRw7zj3/8bYpLNADZVewAACAASURBVPMppbYopbYrpd5SSv1ZKRWwuUopla6UalJKfTjSZRSR936F2T2+KD+VmDAvHQRQf6yDN/52mH3v1YY9LzgxacnXSitmB/mfWUQNX+vlivkZOOxWDI9vNwppwZyIr33tm/z0pz/iyis/yMMP/4oLL7x4qos0oymlYoFfAFdprc8EXgXuGuHyHwINkSqbmFq+CTDLiiKzPFFrUw/73q3laGljRPIrnpdEnNNOfWuvbBs5i0gXuYga/vGXvi6WExvqTlGJossTTzw77P2WLR9gy5YPBLx2/foNp+zSc911nxv3eRHQRcCbWusq7/sHAQ18cehFSqlrgUZge2SLJ6aCxzD8AWYkxl/CkK0iI1R32qxWlnp39dlf3sKW08K3NJKYPiTAFFGhs2eA0up2bFYLKxd414gzzJnVFqs0xIuoUIS5rS4AWutOpZRdKeXQWg+C2YUOfAK4HDMADVpGRmIIijp+WVkze23DcD9faVUr7d0DZKbEsnrJ3IgEfVUJZkBrsUTu89u4KtfcNrKqjX+5cHFE8oSZ/fs53Z9NAkwRFXaVNeExDJYXpZMQ613IW1owRXRxAq6TjrkAA0AptQL4AXCJ1tqllJpQJs3NXXi8w0fCLRKzkKdSJJ7v5R1mg/aKBRk0NUVmt5vOzj7A3DIyUp/fgmxzb/XdpY3U1LbhjAn/WNOZ/PsZyWezWi0T+uIqTT8iKuw8ZA5HW7d4jv9YpLt5hJikGqDA90YplQD0aa19QefNQDrwslJqN7AV+LFS6lMRL6mImN3eseWnlURudvVU1J0piU7mz0vG5fb4JzWJmU1aMMW019U7yMHKVqwWy4nxlyAtmCLavADcqZTK1lrXA18AHvOd1Fp/dujFSqlHgD9rrf8c0VKKiGlq66WmsYvYGBuqIC1i+U5V1bm6JJOjxzr4/+3dd3xc1Znw8d+dopmRRr0Xy7JcjnsFmxhwwRSHhECAEBYWNrAJgWyybJI3u2Szm5C8uyS8m92QTTZhgU3YVFIJPRQDBgMGG9zLcZNlFVu9S6Np9/1jpPHYlq02miI/389H1tw7t5wrS2eee865z9l+sJmls/Jje3IRc9KCKRLe9oPNBIImqjyL9NSUk29ImiKRRLTWbcAXgGeUUpuBC4FvK6XuVEp9JL6lE/Gw/VCo9XL+QGaMWAln4IhxhDnYQLDjcHPMhnGI+JEWTJHwtuozu8cBSVMkko7W+jngudNW/+Qs235qwgsk4mowwIxl9zhE3pvHtu4syUsjP8tJU7uHI/WdzCib+KkqRfxI049IaL0eP3uqWjHgzC6VgafIpYtcCJFsej0+9LF2LIbBgsHMGDESr/HrhmGweEaoHh9MOycmLwkwRULbcSjUPT5rShaZaSmnvhmuJOXXWAiRXHYdaSUQNJlZlonbZY/puQc7p+Nxbz44VeRg662YvOSTWSS0s3WPAyf7eaSLXAiRZAYDrHjMzR2vMZgAM8sySXXYON7Sy4nW3pifX8SOBJgiYfX1+9l1JJTOYqgnDiVN0cj867/ez/r1a7nxxmv4+Mev5q67PsWOHdtHtO/zzz/Dvfd+bsj3Pv/5u3jxxefPWP/znz/Ov/7r/eMpshCTmtcXODn+Mh4Bphm/8es2q4VFM0JDAgbTz4nJSQJMkbB2Hm7BHwgyoyyT7HTHmRsEB8Zgykw+w7r55lv5/e+f4cknn+e22z7FV7/6Zdra2uJdLCHOSzsPt9DvDVBRlE5BdmrMz6/mF3LdrYtZuqJ8+I0nwIVzCgF4b19DXM4vYkM+mUXCCnePqyG6xyFiDKa0YI7GpZeuobCwkL17d8e7KEKclwYDqxVzC+NyfneGk+IpmWTnxj64BZg/LYc0p43aph7qmmIze5GIPUlTJBJSvzfArsMtACw7S0JeM0ESrf/4OxtHtX1eoZtP3LHsjP3vuW91eN3vfvo+zQ1DV7yR241Vd3cPLpcLgKqqIzz00Hepr6/F4XBwxx2fYd26K8d9DiHEmfr6/ewYqNsuHGps+XnAZrWwTOXzxo7jvLuvkevzRz8NoUh80oIpEtK2Q014/UGml2SQm+kceqPwOCL5NR4pv9/Pz372EzIzM1m0aAnt7e3ce+893HjjTfzud0/z7W//Ow899F2OHDkU76IKMSltP9iMzx9kVlkmORlnqdsm2NFDLWx65RCHdfxSBS2P6CYPNxaISUVaMEVCenfPCLqQEiQP5nhbFIfaP7KF02az4PcHx3WOJ574JU899Ufa2lpZt+5K/vM/f4zVauWFF55l6dILuPTSNQBMmVLOmjXreOON16msnDHm88mwBSGG9u5A9/jyOHWPA5yo62DX1jry8t1k5BTFpQyzy7PJSEuhsa2P6oYuKooy4lIOMXEkwBQJp6vXy+6qViyGER4MPhSZyWfkbr75Vj71qU+zdet7fOMbX6WmpgalZlNfX8fmzW9z443XhLf1er1cfvnwXeRut5uenp4z1ns8fTid8WmZESKRdff52DNQt511bHkMTJ2eiys1hcpZsX+CfZDFYnChKmDDB7W8t7dRAsxJSAJMkXC27m8kEDSZPy3nzOTqkWQu8lG74ILlfPrT9/Av//J1fvKTX5KXl8cVV6zny1/+h1Efq7R0CgcO7D9j/eHDB1mxYmU0iivEpPK+DtVt86blkHGuum2CFZdlUlyWSX5+Ok1NXXErx4q5hWz4oJZ39zVw49rpWKTnY1KRT2aRcDbvDXUhXTRvmC6kBHnIJ9lcd90NuN1uHn/8MS6//Cpee+2VcF5M0zTZuvU9Ojs7hz3Otddez4YNL/Hmm69jmiamafLSS39m7949rFt3xURfhhBJ563dJwBYPuf8fLjndJWlGeRmOGnr6md/taRNm2ykBVMklOaOPg7WdpBis7Bk5tBPj4cNjMGULvLRMQyDr3zla9x111+xZs1lfPObD/DDH/4HLS0t2O12Fi1awrx5C8Lb79q145Qu9OnTZ/Dgg9+jvHwqDz74PR599Ef8+78/iGmazJw5i4ce+hEZGZnxuDQhElZdcw+HajtwpFjj/vR44/FO2lr6MOYacW1mshgGlyws5qlNVbyxo565FTnxK4yIOgkwRUJ5d6D1cvHMPFyOc/96Do7BlC7yc/va1+4/Y11l5XReeWVTePnRR3825L5XX30NV199zZDvASxdegE//vFPxl1GISa7N3fUA3DR3EKcKfH96D2wu5Fd79dht1qonDPMjfwEu3RhMU+/VcUHB5ro6vWSnhq/oQMiuuSTWSQM0zTZPPD0+EVzR/BkoyRaF0IkAZ8/yNsD3eOrFpXEuTTxnSrydDkZThZU5uIPmOGfkZgcJMAUCaO6oYu65h7cLjvzK0fQVTI4BjMBKkkhhDibDw400d3no7zATUVReryLEzF8PTHqzsGg+40d9ZITcxKRLnKRMN7ccRyAD80rwmYd/t7HHByDKV3kQogE9sZA9/iqxSUxDepM06TD20ljbzPt/R10e7vp8vUQ6MsGQgFmp7eLt+vfI9XmItuZRY4zm1xnDk6bI2blXDg9l8y0FI639HKwtoNZU7Jidm4xcSTAFAnB6wuEnx6/dGHxyHaSFkwhRIJrbOtlX3UbKTbLyIb+RMnDOx/nYNthPIH+M95b2rcOcGIY0NLXxjNHXjzlfQOD/NRcytwlfGTalRSlTexDSTarhUsWFvPcO9W8saNeAsxJQgJMkRA+ONBEX7+fiqJ0ygpGOC/tYKL1GLZgmqaZMN1K4kzSvSYSzYb364DQvOOpzuh+5JqmSU1XHdubdnOw/TB/u+Sz2C2hcwTMAJ5AP2m2VApS88lxZpGe4sZtd+NpT+cEfVgsBmkpbq6cupYeXw+tnnZaPW009bXQ2NtMY28z18/4aPh8W05sw2JYmJercNqiO5nCpYtKeO6dat7b18gn1s44dw5kkRQkwBQJ4c2doe7xEbdeQszzYFqtNnw+Lykpses6EqMTCPixWKzxLoYQAPR4fOHu8SsunBK1457oaWRrw3beb9hOY19zeP3h9ipm58wE4KaZ1+Gc4yA95cwb9lf37ucEfRiGQa4rm2unf/iU931BPyd6GmnobSTbGWpNNE2TZ4+8SLOnFZvFxpycmSwrWMzi/PnYrfZxX1NBlovFM/LYfqiZDe/Xcv2qynEfU8SXBJgi7prb+9hX3YbdZjn33OOnMWOcB9PtzqK9vYmsrHzs9hRpyUwwphmkq6sNl2uELeBCTLDXt9XR7wswtyKb8sLxP9zT5e3m4Z2Pc7TzWHhdut3NkoKFLMibQ2VmRXh9fmruWY8z3L253WJjSnoJU9JPPvEeNIOsLlvJtqbdVHVUs6t5H7ua95Fqc7G8aClryi455zlHYv2KcrYfaua1D2r5yEVTcaTIzWIyG1OAqZS6CbgP8AAHgLu11p7Ttvl74NNA/cCq32utfziOsopJatOuUOvlMpVPqnMUd8IxnirS5UoDoKOjmUDAH5NzAlgsFoLBYMzOF2vRuz6DlBQnbrckeRfx5/MHeWVrLRAKnMaqy9sdboV029Pw+D24bE4W5y9gWeEiZmVNxzrKVntzDCnerBYrl5Wv4rLyVXT0d7K9aTfvHN9CTVcdr9e+xdzc2eMOMGeWZTK9JIPD9Z28ubOeyy+IXquviL1RB5hKqWLgfmCl1rpdKfV94O+A75y2aQ7wf7XWPx93KcWk5Q8ET3aPLxhF9ziEx2DGcqpIlystHGjGSrznC55ok/36Iiml1gIPAn6gEbhTa90a8f71wOcJpZCzAZ/TWu+MR1nF+Lyz5wQdPV7K8t3MG8MMNXXdx3mp+jW2Ne7iGxd9hVxXDoZh8JkFt5HjzCbFOvYxiifvzcdWd2Y6MlhdtpLVZSs51lXLBw07mTPQNQ/wp0PPk+PM5qLiZaMqp2EYrF9Rzn89uZuXttSwdmkpVotkCUlWY2nBvAl4QmvdPrD8CPA/DB1gNiPEOWw/2ExbVz9FOanMnpo9qn0TKVmwEMNRSjmBh4ErtNbHlFJfBB4A7o7YzADWa629Sql1wKPAitiXVoxH0DR58b1QN/b6FVNG1VJ4qL2Kl6pfY0/LfgAshoWqzmPkukJBalHayIcRnVW4BXP8hypPL6M8vSy83OppY0PNGwTNIM9VvcS6KatYVfahET8UtGRmPgXZLhrb+nhfN7F8ThSuV8TFWALMCuD9iOVqoHSI7VKA+5VS/wxsA76utW4Z6Ulyc2M7jio/P/7JbydSol7fG7/bAcC1q6dTUJAxqn39aSk0AhiWhL2+aJHrmxSuAt7SWg8OoHsM0EQEmFrrP0RsvwUYZbO+SAQf6CaOt/SSne4YcYB0rLOWp4/8mX2tBwBIsdi5uGQFl5VfSo5zdDffwwkGJ24WtMyUDO6YdwuvHNtIdWcNTx15gVdqNnJ5+WpWla4cNr+mxWJw1YVT+PlLB3j27aNcMLsAi4x3T0rDBphKqSLgiYhVRwh17wwKAGcMoNJaf2pgfyvwD8B/AzeOtGAtLd3hP4KJNtm76BL1+moau9l9uAVHipWFFdmjLmN3Vx8QasFMxOuLlkT9/4uWWF2fxWLE/Mb1NBWE6k8AtNZdSimbUsqutfYNsf3/AX472pPIzXl0jfb6AoEgT71VBcDNVyqKi0Y2JvhxvZF9rQdw2Z18ZNZlrJ+5lgzHxPxfLr9kGjNmF1BUmkFufvTPUVR4MVfOXcmuhv38dvezHGg5wlOHX+D12k18/yPfJNXuOuf+1102ixfeq6G2qYe9NR2sXTb2sZiT+fcz0a9t2ABTa30CWDO4rJT6RyByxPJU4Og59g8opX4A7BlzKcWk9OoHoQHwl8wvxuUYfWO6GeM0RUKMk4NTb84ZWD7lTloplQ58n9DN+z2jPYncnEfPWK5v4/Y66pp6KMhysaQy56z7t/S14g36KB7o8r6ydB2Z1iyumLoGtz2N/k6TJibmZ5udn0p2fiq5+e4J/f8rtpbxtws/y/62gzx35GVynFn0tPvpoQvTNPGbgXDeztNds3IqP31+Pz97bi+zSzNGNLvb6Sbz72csr22sN+djGT37B+A2pdTgkw73AGc8yKOUyo9Y/EvgnTGcS0xSPR4f7+w5AcBly4YaYTECcUi0LsQ41BJxcz5Qh3q01v6IdTOBV4CntNafiXxPJD6vL8BTm0Ktlx9fVTlkUNTR38VvD/yJb27+N369/4/hG+USdxEfn/ER3PbYPkQ40QzDYE7OLL687HPcOucT4fW67RDfePs7vFH7Dv7gmb/mK+cXUZybSnOHh43b6894XyS+UX8ya6018BDwulLqXcAK/BRAKXWfUurCgU0fVUq9o5R6E1gJfC5KZRaTwKadx/H6gsytyKY4d4wV6kAeTJkqUiSJF4D1SqnBQXl3Ab86bZtfA3+jtX4qpiUTUbHh/Vrau72UF7q5cM6p0yv2+np56vAL3P/Od9hY+zZBM0iOMxvfEMHVRKs92obe3UBnR1/MzmkYBo6IJ8q3Nmynw9vJbw48ybc2/xvvHN9KIBgIv2+1WLhh9XQAnnmrCo9X7rWSzZjyYGqt/4fQk+Onr/9OxOvrxlEuMYn5A0Fe2lIDwOXjGFvDGHK5CREvWus2pdQXgGeUUkFC4zE/o5S6E2gAXgUWA99VSkXuenvEg0EiQXX2eHnunWoAblw9PfxgSn/Ay+s1m3j52Eb6/KGAblHePD5aeRUl7tjNTR5px3u1HDvSSmFROll5qXEpwy2zb2Be7myePfIiJ3ob+cW+3/JS9at8ZNqVLC1YiMWwsGRmXjgv5vObq7l+1fS4lFWMjczkI2Lund0naOvqpzQvjYUzxp6YNzwGU/KkiSShtX4OeO601T+JeC11cpJ6YsNBevv9zJ+Ww7xpJ/NeegNeXqx+lf6AF5U9g2sq1zMtc+yJ16OhdGoWTpeN9MxzP2wzkSyGhSUFC1iUP4+tDdt57shLNPY289M9v6LT28VlUy7FMAw+edlMHvjF+7yw+Rgr5hZRmje5hhBMZlKZiZgKBk2efzfUGHP1RVPHl35iAlNtCCHESO2uamHz3gZSbBZuuWIGWxu2s7RgIVaLlfQUNzfOvJYcZ1Z4nvB4W7wi1HOUCA/BWAwLy4uWsqxgEZuPb2Vj3dtcVHRB+P38fIPVi0rYuKOen/15P/9w61JJW5QkJMAUMfXBgSYaWnvJy3SyfG7B8Ducy+AYTKlshBBx0u8L8PMXNQBLFzl45MCPaehtwhPwcGnphwBYWXLhuQ4hCE1FeXHpClaWLA83GvgCPv7f1h+QnZNHmms2B2s7eHNHPasXj/HBUBFT0rcoYsY0zfAYpfUrysc9BdjJmXzk11gIER9Pbaqiqd1DSlof240/0dDbRJ4zh/SUxM1R2NneR2tzDz5vYPiNYyyyR6q+5wTegJeqnsN4S0Lzu/zmtYO0d/fHq3hiFOSTWcTM7qpWqhu6yEi1c8lo5x0fiuTBFELE0Uu7dvPnd6sBE2PqdnJcmdyibuDrF32Fxfnz4128s3r1Wc1vHttKfU378BvH0dSMKXxr5X1cXXE5afltWDIb8fQHuf/XG6jplNRFiU66yEVMBE2TP2w8DMBVy8tJsVvHf1AJMIUQcdLR3c9TG5oAg7Qpx7hh6RouLllx1sThicQkecavu2wuPlJ5JaunXMyzeRt5+UUvnS1Ovvf8q3z3k7dgkTzICUv+Z0RMvLevgWMN3WS5U7hsWVlUjmkGQ2MwpYtcCDHRgmaQHU27ea1mE8GgySPP7KXPY1JUaOHfbvoL1pRdnBTBJYAZfkAyzgUZBbc9jZvnX81nr5kPmLQfLUEf6wCgy9tNS19rfAsozpAcfw0iqfkDQZ58IzQF83WXVuKIRuslSAumEGLCBYIB3m/cwUvVr3G8pwG7xc6Jg7nsq24jI9XO339iOU67I97FHJVw1ZmEk1Qsn1VG7Uovz759lEee3sM/3X4BGxpeYVPdu6wsWc7l5avJc+UMfyAx4STAFBNu4/Z6mto9FOemcvGCKCYWlod8hBATpM/v4fkDW3hm3wZaPKHWsSxHJlP7L+Xl945jGPCZa+aR5U6u4BIi782TL8AEuPaSCg7VtrP/WDvf++12Ziz3EzSDvFn3DpvqNrM4fz7ryleRn8DjYM8HEmCKCdXX7+eZt48CcP2qynE/OX4KacEUQkyALm8397/z//AEPADku3K5cupaHN1T+dGTewC4/Sp1SkL1ZBLOwJGkVafVYuFvrl/At3/xAfXNPbh3V/L3H72Y1+veYGvDdrY17WJb0y7U0Uquqbg67ontz1fS9CMm1FObqujs8TK9JIOls/KjeuzwGMxkrSWFEAkhaAap6qgOL6enuJmaUcbc/JncteB2vn7RV8j1z+TRp/dhmvCxiyuSOhejOQmm2U1z2vnSTYvIcqdwoKadpzc0c4v6BN9aeR9XTl2Ly+ZCtxw5pVEjOJg7WcSEtGCKCVPT2M0rW2sxDPjLK1X0KzOZKlIIMQ7t/R1sPv4+79S/R7Onlf+z7PPh1q67F36K0qJcmpq62Hm4hf96chc+f5BLFhZz7SXT4lzy8TlZdSZvgAmQk+Hkizct5ju//IAPDjTxwz/u4p7r5nPt9A9z1dTLqPMfo9x58qHS/9z2CGn2NC4pXYHKniFPoE8wCTDFhAiaJj97cT9B02TdsjKmFk1A0uFJcBcuhIitQDDA3lbNW/Xvsadlf7hVK9uRRYe3M7xdijUFCGXAePSZvQSCJqsWlXD7VRNwsxxjyd5FHmlKgZuv/MVi/uM3O9h5uIXv/XYH9964EJfDwUXFS8NTYbZ52jnccZSgGWR70y7yXLmsKFrKhYVLyU/NjfNVTE4SYIoJsWnncQ7XdZKZlsLHL62ckHOY4dvwSVBLCiEmnGmafHvLQxzvaQBC82Avzp/PypLlzMmZdUqLVtA0+c3Lml/+eT8msH55OZ9YOz3pg0tI/od8TldRlMF9ty7l33+znQM17Tzw8/f5m+sXkJ9/smEj25nF/135Vd6p38pb9e/S3NfCc1Uv81zVy0zLKOe2uZ+kMDW6w7jOdxJgiqhr7fTwu9cOAXDzupmkOifo18wcHIMp3RxCiFMFzSBHO2vY1riT9RXrSLOnYhgGs7KnY5omFxVfwIriZWQMMaVjr8fHo8/sZcfhFgzghtWVXH3R1EkTkJ2cZtcIJ11PdiV5aXz11qV873c7qGvu4VuPb+FLtyxlRkTvWZYjkw9PW8dVFWvZ33qQLQ3b2N60m7ru42SmZIS32996kOK0IjIdiTvdZzKQAFNEVdA0eezZvfR4/CycnsvyOQUTebLQ90lS6Qshxsfj72d/6wF2texjT/N+unzdABSnFbKyZDkAH5/xUWyG9azB4t6jrTz+wn6aOzy4XXY+/dG5LJw+ybpQI6rOyRFehuRlufin2y/gp8/vY6tu4oHHt7B2SSk3rpmOy3Ey3LEYFubmKubmKm4OeKnrrsdpC6Wb8gf9PLrr5/QH+qnIKGdR/jwW5s+T1s0xkABTRNWL7x1j/7F20lPt3HH1nAm944+8CxdCnL9M0+S/d/0v+1o0fjMQXp/rzGZh/jwqMk6mqTnbbDs9Hh+/efUQm3YeB6C80M0///VFWIOT78njipm59PV4cTjt9Hm88S5OVLkcNu65bj4vvlfDHzYe5rVtdew43MztVykWTs87Y3uHNYXKzIrwco+vlxlZ09jfdpCqzmqqOqv50+HnyXPlMjtnJuumrKIg9czjiDNJgCmipvpEF3/cGJqx586r55CZljKxJzw5HcXEnkcIkRBM06Sxt4kD7Yc50lHNbXNuwmJYMAyDPn8fATNIZeZU5ufOYUHeXIrTCoe9yfX5A7z6QR3Pvn2UHo8fm9XgYxdPY/2Kcopy08IPiUwmKy+bDoA73THpAkwIjS1dv6KcS5aW8R+/fJ+jJ7p46Hc7mV+Zw42rp1NeePau70xHBvcsugOPv599rQfY2byH3c37aO5rYVNdC2vKLg5vu6/lAEFMKjPLcdlcsbi0pCIBpoiK7j4fP/7TbgJBk8uWlrJoRgzu8AZzmkkXuRCTkjfg40jHUao6jlHVWc3RjmP0+HvD768tu4TyjFAamptmXUdGSjrpKe4RHbvfG+Ct3cd5fnM1rZ39AMwuz+K2qxTFuWnRvxgRc9NKMvna7ct4eUstT79Vxe4jrew50sryuYWsX15+zuwmTpuDJQULWFKwgEAwwLGuOg53VFGUenLY19NHXuBYVx0GBsVphVRmTqUys4JpmeXkuXLP+zRIEmCKcfMHgvzoyV00tvdRXuDmE2tnxObEkqZIiEnBNE3a+zuo6z6OYRjMy50NQHNfCz/Y/ugp26anuJmZVcms7BlkO7PC60vdxSM6V0NrL2/srOeN7fX0ePwAlOW7uXHNdBZU5pwX9UlbSy+maZKTM/kDaavFwvoV5Vy8oIhn367mtW21vLu3gXf3NqCmZLFuWRmLZuRht509GLRarEzLLD9lRiDTNFHZM7EaNmq6aqnvOUF9zwk21b8LwBXla7huxtUAdHt76PR2UZCah+0sQzQmo/PnSsWE+fUrB9l/rJ2MtBT+9saFOOzWmJxX0hQJkZyOddZyuOMojb3NnOhpoK77eLhlclpGeTjALEorYFbWdErcRUzLnMq0jHJynNmjDgJbOz1sO9jM5j0nOFx/Mtfl9NIMrrywnGUqH8t5EFgOevpXO+jt8fLFb1we76LETHpqCn9x+UyuuLCMV7bW8saOenRNO7qmnTSnjQtnF3Dh7AJmTsnCZh2+5dEwjHAA6Qv4qO6qpaqjmsMdRznWWUuJuyi87c7mPfxy/++xGBbyXXkUpRVQlFoQ/j4lvXRS3thIgCnG5aX3jvHatjpsVgtfuH4BORnO2J1cWjCFSCimadLl66bN006rp502Txut/e0097VwTeX6cCvjloZtvFrz5in7ptlSKXUXMy1zanidxbBw79LPjrocvR4fh+o6OVjbzq7DmXTT5AAAE+FJREFULRxr7A6/50ixcsGsfNYsKWV6aeYYrzT5tL7wPM5p00idPYfMHBcOp+2UaRQBevfvw1NVRc6Hr45TKSdeXqaLm9fN5NpLprFp53He2nWcY43dvL69nte31+Ny2Jg/LYc5U7OZWZZJcV7asDcfdqudGVnTmJE1jSsG1kVOS2maJnnOHFo8bTT0NtLQ28iOgfecVgffXfWt8LZ/OPgMdoudXFc2uc4c8lw5ZDoyz/pwWiJLvhKLhPHqB7U88Woo3+UdV8+OfWUdlId8hIgFX8BHl6+bLm83nd4uOr1ddHm7KezIYUnmEgC6vN3809sP4A/6hzzG0oJF4QBzVvZ0vAEvBan5FKbmU+ouJsuROaabxV6Pj9qmHuqauqlp6uFQbQd1Td2npN9x2K3MrcjmgtkFLJ2ZjyMlNr0sicQ5bRrHH/4RxXd/jutuXQxAqjuFnr7Q+NPe/fvC758PXA4bV1w4hSsunEJtUzeb9zSw/VAz9c09bNnfyJb9jQCkOW3MKM1kalE6ZfluygrcFGS5hp1mM3L85cWlK7i4dAXegJeG3mYaeho40dvIiZ5GrJaTKbOCZpA36t4Z8m8ozZ7KtZUf5uLSFQDUdNSzuWYnmY4M0lPcuO2ppNnTSLW5sFoS4/dbAkwxJhu31/GLlw4AcOsVs/jQvKJh9pgAkqZIJBml1FrgQcAPNAJ3aq1bI96fAjwGZA5s83mt9faxns80TXxBH/0BL4Zh4LaHxtz1+fvQbYfxBrz0+vvo8/XR6++j19dHj7+XG2deQ54rlPvxV/t/z1v17w15/MrscpYsCQWYafZUME3S7KnkOLLIdmaT7cwix5lFjjObyoiWyQV5c1mQN3fY8gdNk16Pn+4+H22dHpo7Ql8tA6+b2vto6+o/Yz+rxWBacQYzyzKZPTWb2eVZ2G2J8aEbL6mz51B89+c4/vCPKLrrbkyfD0/LCfy5RRh2OyceeZjiuz9H6uw58S5qzIXG4IbG4Ta29bLrSCsHa9s5WNtBW1c/Ow63sONwS3j7FJuFguxU8jKd5GY6yRv4ys10kpnmID3VPmQ3e4o1hSnpJUxJLxmyHKZpcrO6npa+Vlo8rbT0tdHiaaXT20WPr/eUwHFv40F+d/CpIY+TanPx4KXfCAe5zx55kV6/B5fNidPqwGlz4LA6cNmcFLjyKEwLPbgUCAbwBf04rClR6RmUAFOM2itba/j1KweB0Ew965aVxaUcpjxFLpKIUsoJPAxcobU+ppT6IvAAcHfEZj8CfqC1flYptQz4FTB8JBbhoQ8e5nh3I96Al/6ANzxTy4qiZdw+95MAtHraeXTXz856jMvLV4cDTIfVgcWwkG5PG3hKOz38tPb0wjKCQRN/IIg/YPKNC/4ZAyuBQJBAxPqA36SxMUiNr4V+b4B+XwBP5PeB1z0eH929Prr6fHT3eunu8xM0z50K3G6zUJKXRlleGqX5bipLMqgoSiclRmPBk0nq7DkU3XU3dd//D7BYwO8Hmw2CQUrv/dJ5GVyeriA7lXXLUlm3rAzTNGnp9HCotoOaxm5qm3qobeqmrauf2qZuapu6z3ocl8NGeqo99OVKwZlixemw4Uqxhl87U6w4U2zYrRZsNgO71UKhbRalaQb2TAt2qwW7zYLFCv1BDw6rHX8giMViUJpRxCWlF9HZH+pN6PH30OPrpdfXB5zagrrlxDaaPa1DlvOyKZdyw8xrADjScZSHtv03BgYOawp2q50USwpF7ny+se7vRv2zNMxh/njjoAKoamnpJhiMTdny89MnZa6zQdG6vmDQ5IlXD/LK1loAblo7g/UryofZa+Icf/Rhut7dzMwv3osxb0ncyjHR5PczOiwWg9xcN8A04OiEn/A0SqlrgWu11ncOLKcDWmtdMrCcDWzRWs+I2GcLcJvWev8ITlEBVD3wwIu0tQ2m8glN1WIAhmliGbgnMw2D4MB92RF6w1O6lJgOXFg5EezHYwYxDYMcw0amcbItwsQ49fUQ93fHCDJYggIMcjBoxKR1INjNAEoY2dAWixnEYgbx+LtweJrJ9HfjMux4U0vJ7j3O7JYtWDDx2NLYXbhqRMcclNN7nMq2UAPx4P5Ofy/zGzaGt9lWciUBY+TB6tn2X1L/EtaBJPA6bwVdjpxRlXWo/VXTu6R7Q4FDTeYcGtwVIzpWh6sIS9DPpVVPYDN9oyqHAI8lhXa7m3abmw67m05bGh12Nx22NHqtTnqtTswYDN2ymMHQ3zVBDMyB1yYGJhYz9GUQxG8D0xj4glC1MPCV4gvi9AYxAK/NoMttwQz/TYf+Xguy0njsyzfAKOtOacEUI9Lr8fPYs3vZfqgZq8Xgzqvn8KH5cegWjyRd5CK5VABHBhe01l1KKZtSyq619gHlnFl5VwOlwEgCTADSgjb8QfvQbw7xmecjLTycOcWwkIaBz+oIB4gWDNJGGAwOSvf34Qj6sJpBsq0u0m2pmL4uMr3t2E0/LouLlNSRpRXCsIJhZUHnIaa27wHghLuSPdkLCHrbsAx8CAYNKx2u0dVJTn9P+PXg/l5vxynbdDgLCFjO8vMcwtn2jwzMux3Zoy7rUPv7rScns/DY3KM6pjXow2IOPV5WnJsz6KWov5Wi/qFbBU1CQehgsNlndeC12PEadvotdryWk9+9FhsBw0rAsOA3rAQMK/6Br8h1AcMSujHECAevQcMCBgQY4Q2Qedp3wGtAtyNim74hdnOMLYm8BJhiWIfqOnjk6T00d3hIc9r4/PULUOXZ8S4WpjzkI5KLg9C4ykh+Tlb3Q70fAEY1V+HiNRX0ewKhcGSggdEY+GcwRLEYRnjdDYVuLFYDi2HQ3tyLzxfghrxUHE47BtDd6aGvx4cxeKyBY1itBtnZbjo7e88oQ26+G4cz9PHS2d5Hd1c/GZku3BmhT7LeHi/trWfudy4ZmReF9y/r8TKjtRdX6oVk594JgM8XoODE6FrCh9rfZrNSUPwxINS6nrGtdthu+kiR+wNk1HYQNE2KSleFHwzJbeymv390wd1Q++fmXxz+ORcN/JyH03foEM1P/ZG03pZwcI7dTsndf4N70eJRlSnRTeben9xcNw2NnQSDJkHTJBgMjVkOmiZm0CQwuN4MfVaanEztZ5oDlY45MIBmYDny/dBqE9NkzMNNJMAUZ+UPBHl+czVPbzpK0DSZWpjO3dfOozAnNd5FCxkYg2lYDBJuoIcQZ6oFLhlcUEqlAR6ttT/i/dPHnExllN358+cUjnl40VBpxjLOMeVr6AP83FPCZmS5yMg6tQUkNS2F1HFMJTvU/na7lZIpWWfZY3hn27+obHzZMYbaP7dgZLMNnc1Q+w/1cx5KT5eLQF8DWC2hPtKBMZiGfeSttCL+LBYj9CBRDIYaD/fE/NlIgCmGtO9oK794+QDHW0KtDFctn8INq6ePKAFtzITnIpcucpEUXgD+RSlVqLVuAO4i9BAPAFrreqVUu1Jqjdb6daXUUsCrta6KV4HF5NK7fx8nHnmY0nu/hOnzYWttwJ9TeN4/RS4mhgSY4hS1jd08tamK9w80AVCYk8ptV85ibsXoBqTHghlOtG6RFkyR8LTWbUqpLwDPKKWChMZjfkYpdSfQoLV+DrgdeFQp9QDgAT4dvxKLySQyz+VgEBnZhTyYwkiCTBEtEmAKAKpPdPH85upwclm7zcI1Kyu4ann5OedojSuZKlIkmYEg8rnTVv8k4v1DwNqYFkqcFzxVVecMHgfzZHqqqiTAFFEhAeZ5rN8X4APdxKsf1Ibn57VZDVYvKuXqD00lO90xzBHiLDgwBlO6yIUQ4pxGMv1j6uw5ElyKqJEA8zzj8frZV93Gln2NbDvYTL8vlFfN5bBxyYJirlo+JbbziY9HuAUzQVtYhRBCiPOUBJiTnD8Q5FBNO5u21bCnqpWDtR0EIp4wnVacwapFxVw0tyjp5uc9OQZTWjCFEEKIRCIB5iTS1++noa2XuqYejp7o4ujxTo41duPzn0yjZxhQWZLB4hl5LJ9bSMEI0lokLHmKXAghhEhIYwowlVIG8GHgu8BKrXX7ENssJDSvrgXoBf5aa109jrKe14KmSa/HT3tXP23d/bR19dPa6aGtq5/mDg/HW3po7/YOuW9xbhozyzKYPy2X2VOzcbsmSb6z4OBMPtJFLoQQQiSSsbZgbiA0hdm5ctf8L3CH1nq7UurjwH8D68d4voQUyp5vEgiczJofCARDrwcy6Ue+9vmD9PsDeH0BvL5g6Ls/9L1/4HW/L0Cvx0+Px0dP3+B3H739foabTMJmNSjMTqU4N5WpRelUFGdQUZROxZScSTmbgTmQaF1aMIUQQojEMtYA86Na616l1NGh3lRKLQLatNbbB1b9CfiRUipVaz2i+cG+/V8baGkLzRN7cvpMI2LZOO290LrIGMw0hlh32nYjPlbEOhODAEbMAxuH6Sfd7Cfd7CfD7CfD9JBu9pNp9pNn9pBperB0AzUn92kC2mwW/P5RzTaXFHxNoZRKEmAKIYQQiWVMAeYIgsQKQkmEB7c3lVJ1QDFweCTnaLO4aE6Cnk+LGcBimlgIhr6bQSwMfDeDWAfWG5jYg35sZgCb6cceDGA3/eF1ka+dgX5cQS/OYD/OgBdXoB9H0It1mHTiZ5vZduiO88nBsFpxFhWSlZ8e76JMqHy5PiGEEElk2ABTKVUEPBGx6nat9bFhdnNwZrwTAEbcjPaFy0rx9XsZbF0caIzEOO01A+8aEcuRDVqh9caZ6yOOkZWdSkdb3ynHDB/PCJ8hvN5igNUYOGcStJ5l56TS1jqihuOkY83MxFlQMCmHAAyKnG1jMorV9VksBrm545sDWgghxMgMG2BqrU8Aa0Z53FqgfHBh4KGgYqBupAeYoqYSDMZmAsD8/HRsk/gDPC0/nd60yXt9QgghhEgsE9UJ/S4wUymlBpavA97QWk/m3lohhBBCCEGU82Aqpe4DNmittyilbgV+OtB62QL8dTTPJYQQQgghEtO4AkytdcVpy9+JeP0esHI8xxdCCCGEEMknCZ7TFkIIIYQQyUQCTCGEEEIIEVUSYAohhBBCiKiSAFMIIYQQQkSVBJhCCCGEECKqJMAUQgghhBBRJQGmEEIIIYSIKgkwhRBCCCFEVEmAKYQQQgghoiqqU0UKIYQ4O6XUTcB9gAc4ANyttfacts3dwE2E6ud+4E6tdU2syyqEEOMhLZhCCBEDSqli4H7gMq31SqAD+LshNm0D1mmtVwFPAN+NWSGFECJKErEF0wpgsRgxPWmszxdrcn3JTa4vquewTvjJhnYT8ITWun1g+RHgf4DvRG6ktf5NxOIW4K9GeHypOyeAXF9ym8zXF6trG2vdmYgBZjFAdnZaTE+am+uO6fliTa4vucn1RVUxcDiWJxxQAbwfsVwNlA6zz5eB347w+FJ3TgC5vuQ2ma8vDtc2qrozEQPMLcClwHEgEOeyCCEmDyuhCnJLLE6mlCoi1MU96Ajgj1gOAMGz7FsIPAps11r/cISnlLpTCDERxlR3JmKA2Q9sinchhBCTUsxaLrXWJ4A1g8tKqX8EyiM2mQocPX0/pdRFwH8CX9Jaj6YulLpTCDFRRl13ykM+QggRG38AblNKDfZh3wP8PHIDpZQd+CVw4yiDSyGESCiGaZrxLoMQQpwXlFJ/DdxNqGt8C3Cv1jqglLoP2AB0ERqneXpX1JVaa29MCyuEEOMgAaYQQgghhIgq6SIXQgghhBBRJQGmEEIIIYSIKgkwhRBCCCFEVEmAKYQQQgghokoCTCGEEEIIEVWJmGg9bpRSfwV8X2udFe+yRItSygF8F1BANrADuEdr7YtrwaJAKbUWeJDQ7CiNwJ1a69b4lip6lFLfBC4BXIRmZ7lTa90R31JFl1JqNfA6kB0xR7dIMlJ3JhepO5NfMtSd0oI5QClVANwe73JMgDTgGa31lVrrC4EU4NNxLtO4KaWcwMOEElKvBDYCD8S3VFG3X2u9buD69gL/GO8CRZNSygV8HWiJd1nE2EndmVyk7kx+yVJ3SoAJKKWswGPAl+JdlmjTWrdqrV+KWLWV0Jyiye4q4C2t9bGB5ceAj8WxPFGntf51xOIWJsf/W6QfEGoh6o53QcTYSN2ZlKTuTH5JUXdKgBnyY+BPWusd8S7IRFJKZQB3AE/GuyxRUAEcGVzQWncBtoGp9iaVgWv6W+C38S5LtCilvgbUa61fiHdZxLhI3Zl8KpC6M2klU9153o3BVEoVAU9ErHoNaNNa/yRORYqqIa7vdq31MaXUBcAPgfu11tviU7qochAaPxTJD0yqqamUUjMItTD8Wmv9bLzLEw0D4/XmA7fEuyxi5KTulLozmUjdGX/n9VSRSik3cJDQIOfBH8QCYBewJlEHzo6WUuqzwPXAp7XWNfEuTzQopW4BLtFaf25gOQ3YpbWujG/Jokcp9THgH4C7tNZ74l2eaFFK7QMCnPyQmwto4JNa671xK5gYMak7k5fUnckr2erO8zrAHIpSqn2SPQm5EPgfYOVkePpxkFIqG3gf+JDWukEp9UUgV2v9T3EuWlQopQqBN4Dlk+3px9MppY4CiydLUHK+krozOUjdOXkket153nWRn4dWAiXAy0qpwXV7B+9ek5XWuk0p9QXgGaVUkNCYos/EuVjRtATIA56K+H9r1VpfH78iCXFekbozOUndmSCkBVMIIYQQQkSVPEUuhBBCCCGiSgJMIYQQQggRVRJgCiGEEEKIqJIAUwghhBBCRJUEmEIIIYQQIqokwBRCCCGEEFElAaYQQgghhIgqCTCFEEIIIURUSYAphBBCCCGi6v8DdX/NpTbc3i4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 792x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "z = np.linspace(-5, 5, 200)\n",
    "\n",
    "plt.figure(figsize=(11,4))\n",
    "\n",
    "plt.subplot(121)\n",
    "plt.plot(z, np.sign(z), \"r-\", linewidth=2, label=\"스텝\")\n",
    "plt.plot(z, logit(z), \"g--\", linewidth=2, label=\"로지스틱\")\n",
    "plt.plot(z, np.tanh(z), \"b-\", linewidth=2, label=\"Tanh\")\n",
    "plt.plot(z, relu(z), \"m-.\", linewidth=2, label=\"ReLU\")\n",
    "plt.grid(True)\n",
    "plt.legend(loc=\"center right\", fontsize=14)\n",
    "plt.title(\"활성화 함수\", fontsize=14)\n",
    "plt.axis([-5, 5, -1.2, 1.2])\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.plot(z, derivative(np.sign, z), \"r-\", linewidth=2, label=\"Step\")\n",
    "plt.plot(0, 0, \"ro\", markersize=5)\n",
    "plt.plot(0, 0, \"rx\", markersize=10)\n",
    "plt.plot(z, derivative(logit, z), \"g--\", linewidth=2, label=\"Logit\")\n",
    "plt.plot(z, derivative(np.tanh, z), \"b-\", linewidth=2, label=\"Tanh\")\n",
    "plt.plot(z, derivative(relu, z), \"m-.\", linewidth=2, label=\"ReLU\")\n",
    "plt.grid(True)\n",
    "plt.title(\"도함수\", fontsize=14)\n",
    "plt.axis([-5, 5, -0.2, 1.2])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 소프트맥스(softmax) 함수\n",
    "\n",
    "**소프트맥스 함수**(softmax function)는 출력층에서 주로 사용하는 활성화 함수이며, 식은 다음과 같다.\n",
    "$$\n",
    "\\hat{y}_k = \\frac{ \\text{exp} \\left( \\mathbf{W}^{T} \\cdot \\mathbf{x} \\right)}{\\sum_{j=1}^{K}{\\text{exp} \\left( \\mathbf{W}^{T} \\cdot \\mathbf{x} \\right)}} \\quad (K=\\text{# of class})\n",
    "$$\n",
    "\n",
    "\n",
    "소프트맥스 함수의 특징은 출력값의 총합이 1이 된다는 것이다. 따라서, 각 출력 뉴런에 대한 소프트맥스의 출력값은 각 클래스에 대응하는 추정 확률값으로 볼 수 있다.\n",
    "\n",
    "![](./images/softmax.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10.2 텐서플로 고수준 API로 다층 퍼셉트론 학습하기\n",
    "\n",
    "텐서플로는 사용자가 쉽게 신경망 모델링을 프로그래밍할 수 있도록 high-level의 API를 제공한다. 대표적인 high-level API로는 `tf.estimator`와 `keras`가 있는데, 이에 대한 자세한 내용은 [여기](http://excelsior-cjh.tistory.com/157?category=940399)에서 참고하면 된다.\n",
    "\n",
    "아래의 예제는 `tf.estimator`의 `DNNClassifier` 클래스를 이용해 MNIST 숫자 데이터를 분류하는 DNN을 구현한 것이다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# MNIST Dataset Load!\n",
    "(X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
    "\n",
    "# reshape : 28 x 28 -> 784\n",
    "X_train = X_train.astype(np.float32).reshape(-1, 28*28) / 255.0\n",
    "X_test = X_test.astype(np.float32).reshape(-1, 28*28) / 255.0\n",
    "y_train = y_train.astype(np.int32)\n",
    "y_test = y_test.astype(np.int32)\n",
    "\n",
    "# split validation set\n",
    "X_valid, X_train = X_train[:5000], X_train[5000:]\n",
    "y_valid, y_train = y_train[:5000], y_train[5000:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "WARNING:tensorflow:Using temporary folder as model directory: C:\\Users\\jongm\\AppData\\Local\\Temp\\tmp16z4p7tg\n",
      "INFO:tensorflow:Using config: {'_model_dir': 'C:\\\\Users\\\\jongm\\\\AppData\\\\Local\\\\Temp\\\\tmp16z4p7tg', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x0000023442A067F0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into C:\\Users\\jongm\\AppData\\Local\\Temp\\tmp16z4p7tg\\model.ckpt.\n",
      "INFO:tensorflow:loss = 118.47757, step = 1\n",
      "INFO:tensorflow:global_step/sec: 300.577\n",
      "INFO:tensorflow:loss = 18.2126, step = 101 (0.333 sec)\n",
      "INFO:tensorflow:global_step/sec: 320.597\n",
      "INFO:tensorflow:loss = 9.526357, step = 201 (0.312 sec)\n",
      "INFO:tensorflow:global_step/sec: 349.977\n",
      "INFO:tensorflow:loss = 8.640748, step = 301 (0.301 sec)\n",
      "INFO:tensorflow:global_step/sec: 292.157\n",
      "INFO:tensorflow:loss = 4.5524364, step = 401 (0.330 sec)\n",
      "INFO:tensorflow:global_step/sec: 302.333\n",
      "INFO:tensorflow:loss = 8.764444, step = 501 (0.331 sec)\n",
      "INFO:tensorflow:global_step/sec: 337.307\n",
      "INFO:tensorflow:loss = 9.664594, step = 601 (0.293 sec)\n",
      "INFO:tensorflow:global_step/sec: 316.077\n",
      "INFO:tensorflow:loss = 5.326588, step = 701 (0.316 sec)\n",
      "INFO:tensorflow:global_step/sec: 270.586\n",
      "INFO:tensorflow:loss = 3.5616407, step = 801 (0.370 sec)\n",
      "INFO:tensorflow:global_step/sec: 305.852\n",
      "INFO:tensorflow:loss = 3.5472608, step = 901 (0.330 sec)\n",
      "INFO:tensorflow:global_step/sec: 289.176\n",
      "INFO:tensorflow:loss = 5.308009, step = 1001 (0.343 sec)\n",
      "INFO:tensorflow:global_step/sec: 343.144\n",
      "INFO:tensorflow:loss = 3.2471476, step = 1101 (0.291 sec)\n",
      "INFO:tensorflow:global_step/sec: 332.338\n",
      "INFO:tensorflow:loss = 1.7393025, step = 1201 (0.301 sec)\n",
      "INFO:tensorflow:global_step/sec: 340.83\n",
      "INFO:tensorflow:loss = 7.621153, step = 1301 (0.293 sec)\n",
      "INFO:tensorflow:global_step/sec: 324.674\n",
      "INFO:tensorflow:loss = 4.4440703, step = 1401 (0.311 sec)\n",
      "INFO:tensorflow:global_step/sec: 278.066\n",
      "INFO:tensorflow:loss = 8.833588, step = 1501 (0.357 sec)\n",
      "INFO:tensorflow:global_step/sec: 337.503\n",
      "INFO:tensorflow:loss = 6.4846, step = 1601 (0.296 sec)\n",
      "INFO:tensorflow:global_step/sec: 332.965\n",
      "INFO:tensorflow:loss = 1.3688798, step = 1701 (0.300 sec)\n",
      "INFO:tensorflow:global_step/sec: 301.374\n",
      "INFO:tensorflow:loss = 8.051659, step = 1801 (0.332 sec)\n",
      "INFO:tensorflow:global_step/sec: 270.771\n",
      "INFO:tensorflow:loss = 6.440572, step = 1901 (0.371 sec)\n",
      "INFO:tensorflow:global_step/sec: 310.156\n",
      "INFO:tensorflow:loss = 2.7557888, step = 2001 (0.322 sec)\n",
      "INFO:tensorflow:global_step/sec: 338.02\n",
      "INFO:tensorflow:loss = 2.7008336, step = 2101 (0.294 sec)\n",
      "INFO:tensorflow:global_step/sec: 273.125\n",
      "INFO:tensorflow:loss = 8.0978985, step = 2201 (0.369 sec)\n",
      "INFO:tensorflow:global_step/sec: 320.689\n",
      "INFO:tensorflow:loss = 1.1877201, step = 2301 (0.311 sec)\n",
      "INFO:tensorflow:global_step/sec: 346.394\n",
      "INFO:tensorflow:loss = 2.3070464, step = 2401 (0.289 sec)\n",
      "INFO:tensorflow:global_step/sec: 330.746\n",
      "INFO:tensorflow:loss = 1.786859, step = 2501 (0.300 sec)\n",
      "INFO:tensorflow:global_step/sec: 329.034\n",
      "INFO:tensorflow:loss = 2.1727102, step = 2601 (0.306 sec)\n",
      "INFO:tensorflow:global_step/sec: 285.387\n",
      "INFO:tensorflow:loss = 5.1974664, step = 2701 (0.350 sec)\n",
      "INFO:tensorflow:global_step/sec: 346.934\n",
      "INFO:tensorflow:loss = 5.202467, step = 2801 (0.286 sec)\n",
      "INFO:tensorflow:global_step/sec: 302.34\n",
      "INFO:tensorflow:loss = 12.756352, step = 2901 (0.332 sec)\n",
      "INFO:tensorflow:global_step/sec: 267.006\n",
      "INFO:tensorflow:loss = 4.6114826, step = 3001 (0.373 sec)\n",
      "INFO:tensorflow:global_step/sec: 301.802\n",
      "INFO:tensorflow:loss = 5.032915, step = 3101 (0.333 sec)\n",
      "INFO:tensorflow:global_step/sec: 264.726\n",
      "INFO:tensorflow:loss = 1.5043229, step = 3201 (0.381 sec)\n",
      "INFO:tensorflow:global_step/sec: 348.243\n",
      "INFO:tensorflow:loss = 1.9262958, step = 3301 (0.298 sec)\n",
      "INFO:tensorflow:global_step/sec: 345.366\n",
      "INFO:tensorflow:loss = 4.6662917, step = 3401 (0.290 sec)\n",
      "INFO:tensorflow:global_step/sec: 314.226\n",
      "INFO:tensorflow:loss = 1.5116744, step = 3501 (0.303 sec)\n",
      "INFO:tensorflow:global_step/sec: 350.863\n",
      "INFO:tensorflow:loss = 0.72464496, step = 3601 (0.285 sec)\n",
      "INFO:tensorflow:global_step/sec: 289.109\n",
      "INFO:tensorflow:loss = 3.3865955, step = 3701 (0.349 sec)\n",
      "INFO:tensorflow:global_step/sec: 317.171\n",
      "INFO:tensorflow:loss = 9.498576, step = 3801 (0.314 sec)\n",
      "INFO:tensorflow:global_step/sec: 300.629\n",
      "INFO:tensorflow:loss = 1.7633529, step = 3901 (0.331 sec)\n",
      "INFO:tensorflow:global_step/sec: 310.592\n",
      "INFO:tensorflow:loss = 2.8007033, step = 4001 (0.322 sec)\n",
      "INFO:tensorflow:global_step/sec: 269.423\n",
      "INFO:tensorflow:loss = 1.505311, step = 4101 (0.374 sec)\n",
      "INFO:tensorflow:global_step/sec: 299.935\n",
      "INFO:tensorflow:loss = 0.51030797, step = 4201 (0.332 sec)\n",
      "INFO:tensorflow:global_step/sec: 323.889\n",
      "INFO:tensorflow:loss = 6.0217066, step = 4301 (0.309 sec)\n",
      "INFO:tensorflow:global_step/sec: 325.583\n",
      "INFO:tensorflow:loss = 0.73372215, step = 4401 (0.306 sec)\n",
      "INFO:tensorflow:global_step/sec: 318.743\n",
      "INFO:tensorflow:loss = 0.39176416, step = 4501 (0.316 sec)\n",
      "INFO:tensorflow:global_step/sec: 301.741\n",
      "INFO:tensorflow:loss = 0.25220641, step = 4601 (0.331 sec)\n",
      "INFO:tensorflow:global_step/sec: 329.316\n",
      "INFO:tensorflow:loss = 1.7911237, step = 4701 (0.304 sec)\n",
      "INFO:tensorflow:global_step/sec: 350.252\n",
      "INFO:tensorflow:loss = 0.3041039, step = 4801 (0.299 sec)\n",
      "INFO:tensorflow:global_step/sec: 315.391\n",
      "INFO:tensorflow:loss = 4.006593, step = 4901 (0.303 sec)\n",
      "INFO:tensorflow:global_step/sec: 308.479\n",
      "INFO:tensorflow:loss = 11.440742, step = 5001 (0.323 sec)\n",
      "INFO:tensorflow:global_step/sec: 319.924\n",
      "INFO:tensorflow:loss = 0.93186164, step = 5101 (0.313 sec)\n",
      "INFO:tensorflow:global_step/sec: 283.234\n",
      "INFO:tensorflow:loss = 1.113692, step = 5201 (0.367 sec)\n",
      "INFO:tensorflow:global_step/sec: 267.342\n",
      "INFO:tensorflow:loss = 1.8629832, step = 5301 (0.360 sec)\n",
      "INFO:tensorflow:global_step/sec: 318.645\n",
      "INFO:tensorflow:loss = 2.758434, step = 5401 (0.312 sec)\n",
      "INFO:tensorflow:global_step/sec: 327.631\n",
      "INFO:tensorflow:loss = 5.803428, step = 5501 (0.305 sec)\n",
      "INFO:tensorflow:global_step/sec: 317.453\n",
      "INFO:tensorflow:loss = 1.8450423, step = 5601 (0.317 sec)\n",
      "INFO:tensorflow:global_step/sec: 331.831\n",
      "INFO:tensorflow:loss = 2.044645, step = 5701 (0.299 sec)\n",
      "INFO:tensorflow:global_step/sec: 271.079\n",
      "INFO:tensorflow:loss = 0.16553625, step = 5801 (0.371 sec)\n",
      "INFO:tensorflow:global_step/sec: 314.712\n",
      "INFO:tensorflow:loss = 0.69580114, step = 5901 (0.331 sec)\n",
      "INFO:tensorflow:global_step/sec: 309.543\n",
      "INFO:tensorflow:loss = 0.76652163, step = 6001 (0.307 sec)\n",
      "INFO:tensorflow:global_step/sec: 243.529\n",
      "INFO:tensorflow:loss = 0.76544744, step = 6101 (0.414 sec)\n",
      "INFO:tensorflow:global_step/sec: 313.571\n",
      "INFO:tensorflow:loss = 2.715458, step = 6201 (0.316 sec)\n",
      "INFO:tensorflow:global_step/sec: 256.769\n",
      "INFO:tensorflow:loss = 1.4937918, step = 6301 (0.392 sec)\n",
      "INFO:tensorflow:global_step/sec: 288.983\n",
      "INFO:tensorflow:loss = 0.12821777, step = 6401 (0.343 sec)\n",
      "INFO:tensorflow:global_step/sec: 306.249\n",
      "INFO:tensorflow:loss = 1.029129, step = 6501 (0.327 sec)\n",
      "INFO:tensorflow:global_step/sec: 265.96\n",
      "INFO:tensorflow:loss = 0.9733986, step = 6601 (0.379 sec)\n",
      "INFO:tensorflow:global_step/sec: 336.206\n",
      "INFO:tensorflow:loss = 0.58823776, step = 6701 (0.294 sec)\n",
      "INFO:tensorflow:global_step/sec: 342.601\n",
      "INFO:tensorflow:loss = 2.4018426, step = 6801 (0.307 sec)\n",
      "INFO:tensorflow:global_step/sec: 310.893\n",
      "INFO:tensorflow:loss = 3.612802, step = 6901 (0.309 sec)\n",
      "INFO:tensorflow:global_step/sec: 276.401\n",
      "INFO:tensorflow:loss = 0.23617141, step = 7001 (0.362 sec)\n",
      "INFO:tensorflow:global_step/sec: 339.671\n",
      "INFO:tensorflow:loss = 1.2245857, step = 7101 (0.292 sec)\n",
      "INFO:tensorflow:global_step/sec: 328.598\n",
      "INFO:tensorflow:loss = 0.6718115, step = 7201 (0.306 sec)\n",
      "INFO:tensorflow:global_step/sec: 335.055\n",
      "INFO:tensorflow:loss = 0.33291692, step = 7301 (0.296 sec)\n",
      "INFO:tensorflow:global_step/sec: 247.225\n",
      "INFO:tensorflow:loss = 0.38439348, step = 7401 (0.409 sec)\n",
      "INFO:tensorflow:global_step/sec: 291.271\n",
      "INFO:tensorflow:loss = 1.5247939, step = 7501 (0.338 sec)\n",
      "INFO:tensorflow:global_step/sec: 331.771\n",
      "INFO:tensorflow:loss = 5.8704176, step = 7601 (0.301 sec)\n",
      "INFO:tensorflow:global_step/sec: 337.287\n",
      "INFO:tensorflow:loss = 1.3048997, step = 7701 (0.296 sec)\n",
      "INFO:tensorflow:global_step/sec: 284.008\n",
      "INFO:tensorflow:loss = 1.3587866, step = 7801 (0.352 sec)\n",
      "INFO:tensorflow:global_step/sec: 314.944\n",
      "INFO:tensorflow:loss = 0.2565527, step = 7901 (0.320 sec)\n",
      "INFO:tensorflow:global_step/sec: 318.26\n",
      "INFO:tensorflow:loss = 0.3808904, step = 8001 (0.312 sec)\n",
      "INFO:tensorflow:global_step/sec: 284.097\n",
      "INFO:tensorflow:loss = 0.5098706, step = 8101 (0.352 sec)\n",
      "INFO:tensorflow:global_step/sec: 314.907\n",
      "INFO:tensorflow:loss = 0.84212404, step = 8201 (0.318 sec)\n",
      "INFO:tensorflow:global_step/sec: 336.999\n",
      "INFO:tensorflow:loss = 1.1911765, step = 8301 (0.297 sec)\n",
      "INFO:tensorflow:global_step/sec: 260.977\n",
      "INFO:tensorflow:loss = 0.621176, step = 8401 (0.386 sec)\n",
      "INFO:tensorflow:global_step/sec: 270.437\n",
      "INFO:tensorflow:loss = 0.30834192, step = 8501 (0.367 sec)\n",
      "INFO:tensorflow:global_step/sec: 302.64\n",
      "INFO:tensorflow:loss = 0.25839424, step = 8601 (0.330 sec)\n",
      "INFO:tensorflow:global_step/sec: 323.119\n",
      "INFO:tensorflow:loss = 2.503171, step = 8701 (0.311 sec)\n",
      "INFO:tensorflow:global_step/sec: 334.486\n",
      "INFO:tensorflow:loss = 0.4627908, step = 8801 (0.297 sec)\n",
      "INFO:tensorflow:global_step/sec: 323.053\n",
      "INFO:tensorflow:loss = 0.45720044, step = 8901 (0.310 sec)\n",
      "INFO:tensorflow:global_step/sec: 281.044\n",
      "INFO:tensorflow:loss = 1.1823292, step = 9001 (0.358 sec)\n",
      "INFO:tensorflow:global_step/sec: 292.478\n",
      "INFO:tensorflow:loss = 0.2659555, step = 9101 (0.342 sec)\n",
      "INFO:tensorflow:global_step/sec: 329.592\n",
      "INFO:tensorflow:loss = 0.16091923, step = 9201 (0.304 sec)\n",
      "INFO:tensorflow:global_step/sec: 324.183\n",
      "INFO:tensorflow:loss = 0.3940344, step = 9301 (0.305 sec)\n",
      "INFO:tensorflow:global_step/sec: 325.351\n",
      "INFO:tensorflow:loss = 0.64414144, step = 9401 (0.307 sec)\n",
      "INFO:tensorflow:global_step/sec: 249.998\n",
      "INFO:tensorflow:loss = 2.7244775, step = 9501 (0.403 sec)\n",
      "INFO:tensorflow:global_step/sec: 275.405\n",
      "INFO:tensorflow:loss = 0.46533203, step = 9601 (0.365 sec)\n",
      "INFO:tensorflow:global_step/sec: 286.07\n",
      "INFO:tensorflow:loss = 0.39243847, step = 9701 (0.347 sec)\n",
      "INFO:tensorflow:global_step/sec: 330.347\n",
      "INFO:tensorflow:loss = 0.95770425, step = 9801 (0.301 sec)\n",
      "INFO:tensorflow:global_step/sec: 333.395\n",
      "INFO:tensorflow:loss = 0.8048396, step = 9901 (0.300 sec)\n",
      "INFO:tensorflow:global_step/sec: 311.409\n",
      "INFO:tensorflow:loss = 1.7472535, step = 10001 (0.321 sec)\n",
      "INFO:tensorflow:global_step/sec: 279.579\n",
      "INFO:tensorflow:loss = 0.6215954, step = 10101 (0.360 sec)\n",
      "INFO:tensorflow:global_step/sec: 319.856\n",
      "INFO:tensorflow:loss = 0.1463009, step = 10201 (0.311 sec)\n",
      "INFO:tensorflow:global_step/sec: 318.882\n",
      "INFO:tensorflow:loss = 0.2278531, step = 10301 (0.314 sec)\n",
      "INFO:tensorflow:global_step/sec: 321.302\n",
      "INFO:tensorflow:loss = 0.2577497, step = 10401 (0.311 sec)\n",
      "INFO:tensorflow:global_step/sec: 334.928\n",
      "INFO:tensorflow:loss = 0.63386154, step = 10501 (0.301 sec)\n",
      "INFO:tensorflow:global_step/sec: 324.423\n",
      "INFO:tensorflow:loss = 7.213933, step = 10601 (0.306 sec)\n",
      "INFO:tensorflow:global_step/sec: 223.93\n",
      "INFO:tensorflow:loss = 0.19391643, step = 10701 (0.449 sec)\n",
      "INFO:tensorflow:global_step/sec: 297.271\n",
      "INFO:tensorflow:loss = 0.7363667, step = 10801 (0.334 sec)\n",
      "INFO:tensorflow:global_step/sec: 332.963\n",
      "INFO:tensorflow:loss = 0.102908954, step = 10901 (0.316 sec)\n",
      "INFO:tensorflow:global_step/sec: 322.678\n",
      "INFO:tensorflow:loss = 0.58030325, step = 11001 (0.294 sec)\n",
      "INFO:tensorflow:global_step/sec: 342.042\n",
      "INFO:tensorflow:loss = 0.035707414, step = 11101 (0.292 sec)\n",
      "INFO:tensorflow:global_step/sec: 316.182\n",
      "INFO:tensorflow:loss = 0.2287913, step = 11201 (0.316 sec)\n",
      "INFO:tensorflow:global_step/sec: 284.865\n",
      "INFO:tensorflow:loss = 0.16885549, step = 11301 (0.355 sec)\n",
      "INFO:tensorflow:global_step/sec: 326.197\n",
      "INFO:tensorflow:loss = 0.025517749, step = 11401 (0.305 sec)\n",
      "INFO:tensorflow:global_step/sec: 320.714\n",
      "INFO:tensorflow:loss = 0.953476, step = 11501 (0.310 sec)\n",
      "INFO:tensorflow:global_step/sec: 332.883\n",
      "INFO:tensorflow:loss = 0.28196925, step = 11601 (0.300 sec)\n",
      "INFO:tensorflow:global_step/sec: 311.502\n",
      "INFO:tensorflow:loss = 0.24463204, step = 11701 (0.321 sec)\n",
      "INFO:tensorflow:global_step/sec: 274.93\n",
      "INFO:tensorflow:loss = 0.23435168, step = 11801 (0.381 sec)\n",
      "INFO:tensorflow:global_step/sec: 268.755\n",
      "INFO:tensorflow:loss = 0.18967754, step = 11901 (0.356 sec)\n",
      "INFO:tensorflow:global_step/sec: 312.9\n",
      "INFO:tensorflow:loss = 0.10355736, step = 12001 (0.320 sec)\n",
      "INFO:tensorflow:global_step/sec: 251.4\n",
      "INFO:tensorflow:loss = 0.5325237, step = 12101 (0.399 sec)\n",
      "INFO:tensorflow:global_step/sec: 270.482\n",
      "INFO:tensorflow:loss = 0.4846813, step = 12201 (0.367 sec)\n",
      "INFO:tensorflow:global_step/sec: 262.017\n",
      "INFO:tensorflow:loss = 0.19858545, step = 12301 (0.386 sec)\n",
      "INFO:tensorflow:global_step/sec: 271.895\n",
      "INFO:tensorflow:loss = 0.01518639, step = 12401 (0.367 sec)\n",
      "INFO:tensorflow:global_step/sec: 271.157\n",
      "INFO:tensorflow:loss = 0.24667053, step = 12501 (0.367 sec)\n",
      "INFO:tensorflow:global_step/sec: 266.448\n",
      "INFO:tensorflow:loss = 0.7655854, step = 12601 (0.376 sec)\n",
      "INFO:tensorflow:global_step/sec: 283.447\n",
      "INFO:tensorflow:loss = 0.31936035, step = 12701 (0.354 sec)\n",
      "INFO:tensorflow:global_step/sec: 262.716\n",
      "INFO:tensorflow:loss = 1.83112, step = 12801 (0.378 sec)\n",
      "INFO:tensorflow:global_step/sec: 316.923\n",
      "INFO:tensorflow:loss = 0.40786698, step = 12901 (0.331 sec)\n",
      "INFO:tensorflow:global_step/sec: 301.252\n",
      "INFO:tensorflow:loss = 0.22921959, step = 13001 (0.319 sec)\n",
      "INFO:tensorflow:global_step/sec: 321.687\n",
      "INFO:tensorflow:loss = 0.16589126, step = 13101 (0.311 sec)\n",
      "INFO:tensorflow:global_step/sec: 271.044\n",
      "INFO:tensorflow:loss = 0.69376206, step = 13201 (0.366 sec)\n",
      "INFO:tensorflow:global_step/sec: 283.463\n",
      "INFO:tensorflow:loss = 0.045898166, step = 13301 (0.355 sec)\n",
      "INFO:tensorflow:global_step/sec: 273.373\n",
      "INFO:tensorflow:loss = 0.1907439, step = 13401 (0.367 sec)\n",
      "INFO:tensorflow:global_step/sec: 271.165\n",
      "INFO:tensorflow:loss = 0.4235722, step = 13501 (0.370 sec)\n",
      "INFO:tensorflow:global_step/sec: 288.349\n",
      "INFO:tensorflow:loss = 0.18457326, step = 13601 (0.346 sec)\n",
      "INFO:tensorflow:global_step/sec: 264.003\n",
      "INFO:tensorflow:loss = 0.013200619, step = 13701 (0.378 sec)\n",
      "INFO:tensorflow:global_step/sec: 248.281\n",
      "INFO:tensorflow:loss = 0.12863249, step = 13801 (0.405 sec)\n",
      "INFO:tensorflow:global_step/sec: 250.143\n",
      "INFO:tensorflow:loss = 0.09190421, step = 13901 (0.398 sec)\n",
      "INFO:tensorflow:global_step/sec: 285.877\n",
      "INFO:tensorflow:loss = 0.41749164, step = 14001 (0.350 sec)\n",
      "INFO:tensorflow:global_step/sec: 329.259\n",
      "INFO:tensorflow:loss = 0.27272394, step = 14101 (0.302 sec)\n",
      "INFO:tensorflow:global_step/sec: 314.27\n",
      "INFO:tensorflow:loss = 0.076614924, step = 14201 (0.318 sec)\n",
      "INFO:tensorflow:global_step/sec: 333.108\n",
      "INFO:tensorflow:loss = 0.5218404, step = 14301 (0.300 sec)\n",
      "INFO:tensorflow:global_step/sec: 316.24\n",
      "INFO:tensorflow:loss = 0.15918571, step = 14401 (0.316 sec)\n",
      "INFO:tensorflow:global_step/sec: 303.193\n",
      "INFO:tensorflow:loss = 0.11084022, step = 14501 (0.333 sec)\n",
      "INFO:tensorflow:global_step/sec: 274.881\n",
      "INFO:tensorflow:loss = 0.20325717, step = 14601 (0.363 sec)\n",
      "INFO:tensorflow:global_step/sec: 330.458\n",
      "INFO:tensorflow:loss = 0.207778, step = 14701 (0.301 sec)\n",
      "INFO:tensorflow:global_step/sec: 303.215\n",
      "INFO:tensorflow:loss = 0.060600337, step = 14801 (0.333 sec)\n",
      "INFO:tensorflow:global_step/sec: 272.68\n",
      "INFO:tensorflow:loss = 0.06753541, step = 14901 (0.367 sec)\n",
      "INFO:tensorflow:global_step/sec: 278.711\n",
      "INFO:tensorflow:loss = 0.10640932, step = 15001 (0.357 sec)\n",
      "INFO:tensorflow:global_step/sec: 332.035\n",
      "INFO:tensorflow:loss = 0.011492984, step = 15101 (0.300 sec)\n",
      "INFO:tensorflow:global_step/sec: 290.501\n",
      "INFO:tensorflow:loss = 0.15888111, step = 15201 (0.346 sec)\n",
      "INFO:tensorflow:global_step/sec: 327.088\n",
      "INFO:tensorflow:loss = 0.056837384, step = 15301 (0.304 sec)\n",
      "INFO:tensorflow:global_step/sec: 316.463\n",
      "INFO:tensorflow:loss = 0.104069725, step = 15401 (0.316 sec)\n",
      "INFO:tensorflow:global_step/sec: 298.96\n",
      "INFO:tensorflow:loss = 0.11440698, step = 15501 (0.334 sec)\n",
      "INFO:tensorflow:global_step/sec: 333.671\n",
      "INFO:tensorflow:loss = 0.29308295, step = 15601 (0.300 sec)\n",
      "INFO:tensorflow:global_step/sec: 267.002\n",
      "INFO:tensorflow:loss = 0.07249571, step = 15701 (0.377 sec)\n",
      "INFO:tensorflow:global_step/sec: 290.019\n",
      "INFO:tensorflow:loss = 0.057936475, step = 15801 (0.346 sec)\n",
      "INFO:tensorflow:global_step/sec: 259.217\n",
      "INFO:tensorflow:loss = 0.847804, step = 15901 (0.386 sec)\n",
      "INFO:tensorflow:global_step/sec: 235.984\n",
      "INFO:tensorflow:loss = 0.03624999, step = 16001 (0.423 sec)\n",
      "INFO:tensorflow:global_step/sec: 273.383\n",
      "INFO:tensorflow:loss = 0.06259473, step = 16101 (0.366 sec)\n",
      "INFO:tensorflow:global_step/sec: 268.971\n",
      "INFO:tensorflow:loss = 0.07835274, step = 16201 (0.372 sec)\n",
      "INFO:tensorflow:global_step/sec: 263.308\n",
      "INFO:tensorflow:loss = 0.16941412, step = 16301 (0.382 sec)\n",
      "INFO:tensorflow:global_step/sec: 315.552\n",
      "INFO:tensorflow:loss = 2.1730573, step = 16401 (0.315 sec)\n",
      "INFO:tensorflow:global_step/sec: 274.882\n",
      "INFO:tensorflow:loss = 0.1593946, step = 16501 (0.366 sec)\n",
      "INFO:tensorflow:global_step/sec: 301.379\n",
      "INFO:tensorflow:loss = 0.18374616, step = 16601 (0.330 sec)\n",
      "INFO:tensorflow:global_step/sec: 317.215\n",
      "INFO:tensorflow:loss = 0.10226478, step = 16701 (0.313 sec)\n",
      "INFO:tensorflow:global_step/sec: 288.352\n",
      "INFO:tensorflow:loss = 0.06516072, step = 16801 (0.351 sec)\n",
      "INFO:tensorflow:global_step/sec: 299.572\n",
      "INFO:tensorflow:loss = 0.19193771, step = 16901 (0.334 sec)\n",
      "INFO:tensorflow:global_step/sec: 252.67\n",
      "INFO:tensorflow:loss = 0.013259938, step = 17001 (0.395 sec)\n",
      "INFO:tensorflow:global_step/sec: 263.309\n",
      "INFO:tensorflow:loss = 0.09845054, step = 17101 (0.379 sec)\n",
      "INFO:tensorflow:global_step/sec: 280.272\n",
      "INFO:tensorflow:loss = 0.14803255, step = 17201 (0.359 sec)\n",
      "INFO:tensorflow:global_step/sec: 317.765\n",
      "INFO:tensorflow:loss = 0.031508587, step = 17301 (0.311 sec)\n",
      "INFO:tensorflow:global_step/sec: 340.515\n",
      "INFO:tensorflow:loss = 0.20654431, step = 17401 (0.294 sec)\n",
      "INFO:tensorflow:global_step/sec: 333.421\n",
      "INFO:tensorflow:loss = 0.045559444, step = 17501 (0.300 sec)\n",
      "INFO:tensorflow:global_step/sec: 318.889\n",
      "INFO:tensorflow:loss = 0.027650606, step = 17601 (0.314 sec)\n",
      "INFO:tensorflow:global_step/sec: 307.353\n",
      "INFO:tensorflow:loss = 0.2659089, step = 17701 (0.328 sec)\n",
      "INFO:tensorflow:global_step/sec: 307.868\n",
      "INFO:tensorflow:loss = 0.043203935, step = 17801 (0.325 sec)\n",
      "INFO:tensorflow:global_step/sec: 275.64\n",
      "INFO:tensorflow:loss = 0.049498834, step = 17901 (0.362 sec)\n",
      "INFO:tensorflow:global_step/sec: 297.788\n",
      "INFO:tensorflow:loss = 0.083702005, step = 18001 (0.337 sec)\n",
      "INFO:tensorflow:global_step/sec: 273.281\n",
      "INFO:tensorflow:loss = 0.013095945, step = 18101 (0.363 sec)\n",
      "INFO:tensorflow:global_step/sec: 301.294\n",
      "INFO:tensorflow:loss = 0.21819592, step = 18201 (0.332 sec)\n",
      "INFO:tensorflow:global_step/sec: 306.806\n",
      "INFO:tensorflow:loss = 0.31217307, step = 18301 (0.328 sec)\n",
      "INFO:tensorflow:global_step/sec: 320.696\n",
      "INFO:tensorflow:loss = 0.052900113, step = 18401 (0.312 sec)\n",
      "INFO:tensorflow:global_step/sec: 318.094\n",
      "INFO:tensorflow:loss = 0.14631683, step = 18501 (0.314 sec)\n",
      "INFO:tensorflow:global_step/sec: 280.798\n",
      "INFO:tensorflow:loss = 0.003275036, step = 18601 (0.358 sec)\n",
      "INFO:tensorflow:global_step/sec: 342.165\n",
      "INFO:tensorflow:loss = 0.06036215, step = 18701 (0.304 sec)\n",
      "INFO:tensorflow:global_step/sec: 298.76\n",
      "INFO:tensorflow:loss = 0.04432611, step = 18801 (0.321 sec)\n",
      "INFO:tensorflow:global_step/sec: 327.852\n",
      "INFO:tensorflow:loss = 0.055013385, step = 18901 (0.305 sec)\n",
      "INFO:tensorflow:global_step/sec: 310.737\n",
      "INFO:tensorflow:loss = 0.08838765, step = 19001 (0.323 sec)\n",
      "INFO:tensorflow:global_step/sec: 309.694\n",
      "INFO:tensorflow:loss = 0.22461501, step = 19101 (0.320 sec)\n",
      "INFO:tensorflow:global_step/sec: 249.064\n",
      "INFO:tensorflow:loss = 0.03145126, step = 19201 (0.409 sec)\n",
      "INFO:tensorflow:global_step/sec: 273.953\n",
      "INFO:tensorflow:loss = 0.17571054, step = 19301 (0.360 sec)\n",
      "INFO:tensorflow:global_step/sec: 327.028\n",
      "INFO:tensorflow:loss = 0.39158666, step = 19401 (0.304 sec)\n",
      "INFO:tensorflow:global_step/sec: 334.325\n",
      "INFO:tensorflow:loss = 0.040967252, step = 19501 (0.299 sec)\n",
      "INFO:tensorflow:global_step/sec: 306.283\n",
      "INFO:tensorflow:loss = 0.20054844, step = 19601 (0.328 sec)\n",
      "INFO:tensorflow:global_step/sec: 326.935\n",
      "INFO:tensorflow:loss = 0.08043328, step = 19701 (0.306 sec)\n",
      "INFO:tensorflow:global_step/sec: 321.511\n",
      "INFO:tensorflow:loss = 0.042165846, step = 19801 (0.311 sec)\n",
      "INFO:tensorflow:global_step/sec: 314.642\n",
      "INFO:tensorflow:loss = 0.09890331, step = 19901 (0.319 sec)\n",
      "INFO:tensorflow:global_step/sec: 320.701\n",
      "INFO:tensorflow:loss = 0.32431126, step = 20001 (0.309 sec)\n",
      "INFO:tensorflow:global_step/sec: 276.957\n",
      "INFO:tensorflow:loss = 0.032446764, step = 20101 (0.363 sec)\n",
      "INFO:tensorflow:global_step/sec: 324.146\n",
      "INFO:tensorflow:loss = 0.08212304, step = 20201 (0.309 sec)\n",
      "INFO:tensorflow:global_step/sec: 269.697\n",
      "INFO:tensorflow:loss = 0.18221442, step = 20301 (0.372 sec)\n",
      "INFO:tensorflow:global_step/sec: 277.166\n",
      "INFO:tensorflow:loss = 0.029954683, step = 20401 (0.360 sec)\n",
      "INFO:tensorflow:global_step/sec: 311.705\n",
      "INFO:tensorflow:loss = 0.09457987, step = 20501 (0.321 sec)\n",
      "INFO:tensorflow:global_step/sec: 319.35\n",
      "INFO:tensorflow:loss = 0.06529947, step = 20601 (0.313 sec)\n",
      "INFO:tensorflow:global_step/sec: 319.001\n",
      "INFO:tensorflow:loss = 0.051009007, step = 20701 (0.315 sec)\n",
      "INFO:tensorflow:global_step/sec: 286.699\n",
      "INFO:tensorflow:loss = 0.017569717, step = 20801 (0.348 sec)\n",
      "INFO:tensorflow:global_step/sec: 325.029\n",
      "INFO:tensorflow:loss = 0.028415956, step = 20901 (0.305 sec)\n",
      "INFO:tensorflow:global_step/sec: 307.48\n",
      "INFO:tensorflow:loss = 0.057083, step = 21001 (0.327 sec)\n",
      "INFO:tensorflow:global_step/sec: 285.063\n",
      "INFO:tensorflow:loss = 0.012147564, step = 21101 (0.351 sec)\n",
      "INFO:tensorflow:global_step/sec: 314.646\n",
      "INFO:tensorflow:loss = 0.07647079, step = 21201 (0.318 sec)\n",
      "INFO:tensorflow:global_step/sec: 287.522\n",
      "INFO:tensorflow:loss = 0.028503276, step = 21301 (0.350 sec)\n",
      "INFO:tensorflow:global_step/sec: 259.888\n",
      "INFO:tensorflow:loss = 0.025719734, step = 21401 (0.384 sec)\n",
      "INFO:tensorflow:global_step/sec: 282.382\n",
      "INFO:tensorflow:loss = 0.1087678, step = 21501 (0.351 sec)\n",
      "INFO:tensorflow:global_step/sec: 319.237\n",
      "INFO:tensorflow:loss = 0.015327782, step = 21601 (0.315 sec)\n",
      "INFO:tensorflow:global_step/sec: 329.406\n",
      "INFO:tensorflow:loss = 0.047934197, step = 21701 (0.302 sec)\n",
      "INFO:tensorflow:global_step/sec: 271.217\n",
      "INFO:tensorflow:loss = 0.08197454, step = 21801 (0.373 sec)\n",
      "INFO:tensorflow:global_step/sec: 289.181\n",
      "INFO:tensorflow:loss = 0.196406, step = 21901 (0.344 sec)\n",
      "INFO:tensorflow:global_step/sec: 305.057\n",
      "INFO:tensorflow:loss = 0.029608464, step = 22001 (0.328 sec)\n",
      "INFO:tensorflow:global_step/sec: 291.712\n",
      "INFO:tensorflow:loss = 0.045423795, step = 22101 (0.343 sec)\n",
      "INFO:tensorflow:global_step/sec: 316.994\n",
      "INFO:tensorflow:loss = 0.0220997, step = 22201 (0.315 sec)\n",
      "INFO:tensorflow:global_step/sec: 335.709\n",
      "INFO:tensorflow:loss = 0.06828871, step = 22301 (0.296 sec)\n",
      "INFO:tensorflow:global_step/sec: 289.412\n",
      "INFO:tensorflow:loss = 0.095886275, step = 22401 (0.349 sec)\n",
      "INFO:tensorflow:global_step/sec: 264.004\n",
      "INFO:tensorflow:loss = 0.16202465, step = 22501 (0.379 sec)\n",
      "INFO:tensorflow:global_step/sec: 292.743\n",
      "INFO:tensorflow:loss = 0.09103072, step = 22601 (0.339 sec)\n",
      "INFO:tensorflow:global_step/sec: 316.533\n",
      "INFO:tensorflow:loss = 0.025910653, step = 22701 (0.318 sec)\n",
      "INFO:tensorflow:global_step/sec: 327.694\n",
      "INFO:tensorflow:loss = 0.021004504, step = 22801 (0.303 sec)\n",
      "INFO:tensorflow:global_step/sec: 306.658\n",
      "INFO:tensorflow:loss = 0.07806501, step = 22901 (0.328 sec)\n",
      "INFO:tensorflow:global_step/sec: 317.166\n",
      "INFO:tensorflow:loss = 0.03241463, step = 23001 (0.315 sec)\n",
      "INFO:tensorflow:global_step/sec: 317.642\n",
      "INFO:tensorflow:loss = 0.042591747, step = 23101 (0.315 sec)\n",
      "INFO:tensorflow:global_step/sec: 309.453\n",
      "INFO:tensorflow:loss = 0.04510494, step = 23201 (0.323 sec)\n",
      "INFO:tensorflow:global_step/sec: 329.421\n",
      "INFO:tensorflow:loss = 0.029292868, step = 23301 (0.302 sec)\n",
      "INFO:tensorflow:global_step/sec: 307.741\n",
      "INFO:tensorflow:loss = 0.024887871, step = 23401 (0.328 sec)\n",
      "INFO:tensorflow:global_step/sec: 321.921\n",
      "INFO:tensorflow:loss = 0.026107779, step = 23501 (0.308 sec)\n",
      "INFO:tensorflow:global_step/sec: 257.2\n",
      "INFO:tensorflow:loss = 0.030456983, step = 23601 (0.393 sec)\n",
      "INFO:tensorflow:global_step/sec: 285.067\n",
      "INFO:tensorflow:loss = 0.03446884, step = 23701 (0.350 sec)\n",
      "INFO:tensorflow:global_step/sec: 322.765\n",
      "INFO:tensorflow:loss = 0.042272404, step = 23801 (0.309 sec)\n",
      "INFO:tensorflow:global_step/sec: 313.658\n",
      "INFO:tensorflow:loss = 0.012376662, step = 23901 (0.319 sec)\n",
      "INFO:tensorflow:global_step/sec: 308.818\n",
      "INFO:tensorflow:loss = 0.03966806, step = 24001 (0.324 sec)\n",
      "INFO:tensorflow:global_step/sec: 310.738\n",
      "INFO:tensorflow:loss = 0.060012624, step = 24101 (0.322 sec)\n",
      "INFO:tensorflow:global_step/sec: 330.548\n",
      "INFO:tensorflow:loss = 0.046957772, step = 24201 (0.301 sec)\n",
      "INFO:tensorflow:global_step/sec: 304.089\n",
      "INFO:tensorflow:loss = 0.0191703, step = 24301 (0.332 sec)\n",
      "INFO:tensorflow:global_step/sec: 316.635\n",
      "INFO:tensorflow:loss = 0.09615418, step = 24401 (0.315 sec)\n",
      "INFO:tensorflow:global_step/sec: 310.737\n",
      "INFO:tensorflow:loss = 0.010516002, step = 24501 (0.321 sec)\n",
      "INFO:tensorflow:global_step/sec: 303.204\n",
      "INFO:tensorflow:loss = 0.07754654, step = 24601 (0.331 sec)\n",
      "INFO:tensorflow:global_step/sec: 240.522\n",
      "INFO:tensorflow:loss = 0.0014943181, step = 24701 (0.417 sec)\n",
      "INFO:tensorflow:global_step/sec: 273.379\n",
      "INFO:tensorflow:loss = 0.029815797, step = 24801 (0.368 sec)\n",
      "INFO:tensorflow:global_step/sec: 290.023\n",
      "INFO:tensorflow:loss = 0.16976552, step = 24901 (0.343 sec)\n",
      "INFO:tensorflow:global_step/sec: 295.153\n",
      "INFO:tensorflow:loss = 0.008138146, step = 25001 (0.338 sec)\n",
      "INFO:tensorflow:global_step/sec: 300.473\n",
      "INFO:tensorflow:loss = 0.032847743, step = 25101 (0.333 sec)\n",
      "INFO:tensorflow:global_step/sec: 306.924\n",
      "INFO:tensorflow:loss = 0.06523929, step = 25201 (0.326 sec)\n",
      "INFO:tensorflow:global_step/sec: 262.617\n",
      "INFO:tensorflow:loss = 0.0551019, step = 25301 (0.381 sec)\n",
      "INFO:tensorflow:global_step/sec: 301.299\n",
      "INFO:tensorflow:loss = 0.02621803, step = 25401 (0.330 sec)\n",
      "INFO:tensorflow:global_step/sec: 304.683\n",
      "INFO:tensorflow:loss = 0.007585174, step = 25501 (0.330 sec)\n",
      "INFO:tensorflow:global_step/sec: 316.634\n",
      "INFO:tensorflow:loss = 0.008654072, step = 25601 (0.316 sec)\n",
      "INFO:tensorflow:global_step/sec: 281.061\n",
      "INFO:tensorflow:loss = 0.013693995, step = 25701 (0.357 sec)\n",
      "INFO:tensorflow:global_step/sec: 255.248\n",
      "INFO:tensorflow:loss = 0.029335894, step = 25801 (0.391 sec)\n",
      "INFO:tensorflow:global_step/sec: 324.053\n",
      "INFO:tensorflow:loss = 0.013381727, step = 25901 (0.307 sec)\n",
      "INFO:tensorflow:global_step/sec: 304.009\n",
      "INFO:tensorflow:loss = 0.044429433, step = 26001 (0.332 sec)\n",
      "INFO:tensorflow:global_step/sec: 305.05\n",
      "INFO:tensorflow:loss = 0.028488623, step = 26101 (0.327 sec)\n",
      "INFO:tensorflow:global_step/sec: 309.776\n",
      "INFO:tensorflow:loss = 0.014427401, step = 26201 (0.323 sec)\n",
      "INFO:tensorflow:global_step/sec: 307.867\n",
      "INFO:tensorflow:loss = 0.033279244, step = 26301 (0.325 sec)\n",
      "INFO:tensorflow:global_step/sec: 312.678\n",
      "INFO:tensorflow:loss = 0.058359146, step = 26401 (0.320 sec)\n",
      "INFO:tensorflow:global_step/sec: 320.927\n",
      "INFO:tensorflow:loss = 0.02608341, step = 26501 (0.312 sec)\n",
      "INFO:tensorflow:global_step/sec: 309.773\n",
      "INFO:tensorflow:loss = 0.0067666983, step = 26601 (0.324 sec)\n",
      "INFO:tensorflow:global_step/sec: 304.126\n",
      "INFO:tensorflow:loss = 0.030802187, step = 26701 (0.329 sec)\n",
      "INFO:tensorflow:global_step/sec: 271.894\n",
      "INFO:tensorflow:loss = 0.013141214, step = 26801 (0.367 sec)\n",
      "INFO:tensorflow:global_step/sec: 232.691\n",
      "INFO:tensorflow:loss = 0.04531598, step = 26901 (0.432 sec)\n",
      "INFO:tensorflow:global_step/sec: 275.64\n",
      "INFO:tensorflow:loss = 0.054257076, step = 27001 (0.362 sec)\n",
      "INFO:tensorflow:global_step/sec: 277.935\n",
      "INFO:tensorflow:loss = 0.05418043, step = 27101 (0.359 sec)\n",
      "INFO:tensorflow:global_step/sec: 271.895\n",
      "INFO:tensorflow:loss = 0.013072925, step = 27201 (0.368 sec)\n",
      "INFO:tensorflow:global_step/sec: 271.894\n",
      "INFO:tensorflow:loss = 0.031681694, step = 27301 (0.368 sec)\n",
      "INFO:tensorflow:global_step/sec: 271.159\n",
      "INFO:tensorflow:loss = 0.029033896, step = 27401 (0.369 sec)\n",
      "INFO:tensorflow:global_step/sec: 265.403\n",
      "INFO:tensorflow:loss = 0.030100003, step = 27501 (0.379 sec)\n",
      "INFO:tensorflow:global_step/sec: 273.381\n",
      "INFO:tensorflow:loss = 0.008335559, step = 27601 (0.365 sec)\n",
      "INFO:tensorflow:global_step/sec: 239.946\n",
      "INFO:tensorflow:loss = 0.02329326, step = 27701 (0.416 sec)\n",
      "INFO:tensorflow:global_step/sec: 223.842\n",
      "INFO:tensorflow:loss = 0.036853604, step = 27801 (0.455 sec)\n",
      "INFO:tensorflow:global_step/sec: 235.983\n",
      "INFO:tensorflow:loss = 0.07556492, step = 27901 (0.416 sec)\n",
      "INFO:tensorflow:global_step/sec: 256.557\n",
      "INFO:tensorflow:loss = 0.03828002, step = 28001 (0.390 sec)\n",
      "INFO:tensorflow:global_step/sec: 267.531\n",
      "INFO:tensorflow:loss = 0.045651175, step = 28101 (0.375 sec)\n",
      "INFO:tensorflow:global_step/sec: 264.004\n",
      "INFO:tensorflow:loss = 0.011059683, step = 28201 (0.379 sec)\n",
      "INFO:tensorflow:global_step/sec: 268.972\n",
      "INFO:tensorflow:loss = 0.022825684, step = 28301 (0.371 sec)\n",
      "INFO:tensorflow:global_step/sec: 263.308\n",
      "INFO:tensorflow:loss = 0.04270538, step = 28401 (0.384 sec)\n",
      "INFO:tensorflow:global_step/sec: 265.404\n",
      "INFO:tensorflow:loss = 0.015630761, step = 28501 (0.373 sec)\n",
      "INFO:tensorflow:global_step/sec: 267.533\n",
      "INFO:tensorflow:loss = 0.051833976, step = 28601 (0.376 sec)\n",
      "INFO:tensorflow:global_step/sec: 255.248\n",
      "INFO:tensorflow:loss = 0.01680531, step = 28701 (0.391 sec)\n",
      "INFO:tensorflow:global_step/sec: 234.326\n",
      "INFO:tensorflow:loss = 0.0549723, step = 28801 (0.426 sec)\n",
      "INFO:tensorflow:global_step/sec: 290.863\n",
      "INFO:tensorflow:loss = 0.013206955, step = 28901 (0.345 sec)\n",
      "INFO:tensorflow:global_step/sec: 311.705\n",
      "INFO:tensorflow:loss = 0.019759076, step = 29001 (0.320 sec)\n",
      "INFO:tensorflow:global_step/sec: 306.922\n",
      "INFO:tensorflow:loss = 0.07369554, step = 29101 (0.327 sec)\n",
      "INFO:tensorflow:global_step/sec: 299.574\n",
      "INFO:tensorflow:loss = 0.029176109, step = 29201 (0.334 sec)\n",
      "INFO:tensorflow:global_step/sec: 319.6\n",
      "INFO:tensorflow:loss = 0.084672235, step = 29301 (0.310 sec)\n",
      "INFO:tensorflow:global_step/sec: 318.151\n",
      "INFO:tensorflow:loss = 0.031545475, step = 29401 (0.317 sec)\n",
      "INFO:tensorflow:global_step/sec: 308.818\n",
      "INFO:tensorflow:loss = 0.030005496, step = 29501 (0.323 sec)\n",
      "INFO:tensorflow:global_step/sec: 308.819\n",
      "INFO:tensorflow:loss = 0.07742635, step = 29601 (0.325 sec)\n",
      "INFO:tensorflow:global_step/sec: 330.339\n",
      "INFO:tensorflow:loss = 0.025081668, step = 29701 (0.300 sec)\n",
      "INFO:tensorflow:global_step/sec: 278.178\n",
      "INFO:tensorflow:loss = 0.0542585, step = 29801 (0.359 sec)\n",
      "INFO:tensorflow:global_step/sec: 260.356\n",
      "INFO:tensorflow:loss = 0.026728995, step = 29901 (0.386 sec)\n",
      "INFO:tensorflow:global_step/sec: 311.706\n",
      "INFO:tensorflow:loss = 0.05192495, step = 30001 (0.321 sec)\n",
      "INFO:tensorflow:global_step/sec: 314.644\n",
      "INFO:tensorflow:loss = 0.065029316, step = 30101 (0.318 sec)\n",
      "INFO:tensorflow:global_step/sec: 318.656\n",
      "INFO:tensorflow:loss = 0.0142825665, step = 30201 (0.315 sec)\n",
      "INFO:tensorflow:global_step/sec: 314.642\n",
      "INFO:tensorflow:loss = 0.04642711, step = 30301 (0.318 sec)\n",
      "INFO:tensorflow:global_step/sec: 314.648\n",
      "INFO:tensorflow:loss = 0.15887615, step = 30401 (0.317 sec)\n",
      "INFO:tensorflow:global_step/sec: 301.377\n",
      "INFO:tensorflow:loss = 0.01912181, step = 30501 (0.332 sec)\n",
      "INFO:tensorflow:global_step/sec: 308.819\n",
      "INFO:tensorflow:loss = 0.008215215, step = 30601 (0.325 sec)\n",
      "INFO:tensorflow:global_step/sec: 321.728\n",
      "INFO:tensorflow:loss = 0.034053672, step = 30701 (0.310 sec)\n",
      "INFO:tensorflow:global_step/sec: 308.818\n",
      "INFO:tensorflow:loss = 0.016394371, step = 30801 (0.325 sec)\n",
      "INFO:tensorflow:global_step/sec: 271.157\n",
      "INFO:tensorflow:loss = 0.057917066, step = 30901 (0.369 sec)\n",
      "INFO:tensorflow:global_step/sec: 265.403\n",
      "INFO:tensorflow:loss = 0.012590424, step = 31001 (0.375 sec)\n",
      "INFO:tensorflow:global_step/sec: 313.66\n",
      "INFO:tensorflow:loss = 0.080129884, step = 31101 (0.321 sec)\n",
      "INFO:tensorflow:global_step/sec: 318.653\n",
      "INFO:tensorflow:loss = 0.018138517, step = 31201 (0.314 sec)\n",
      "INFO:tensorflow:global_step/sec: 318.654\n",
      "INFO:tensorflow:loss = 0.0112710595, step = 31301 (0.313 sec)\n",
      "INFO:tensorflow:global_step/sec: 305.052\n",
      "INFO:tensorflow:loss = 0.06556987, step = 31401 (0.328 sec)\n",
      "INFO:tensorflow:global_step/sec: 313.659\n",
      "INFO:tensorflow:loss = 0.083278134, step = 31501 (0.319 sec)\n",
      "INFO:tensorflow:global_step/sec: 320.697\n",
      "INFO:tensorflow:loss = 0.038524088, step = 31601 (0.313 sec)\n",
      "INFO:tensorflow:global_step/sec: 306.91\n",
      "INFO:tensorflow:loss = 0.04540049, step = 31701 (0.325 sec)\n",
      "INFO:tensorflow:global_step/sec: 309.789\n",
      "INFO:tensorflow:loss = 0.027391098, step = 31801 (0.324 sec)\n",
      "INFO:tensorflow:global_step/sec: 318.654\n",
      "INFO:tensorflow:loss = 0.03179156, step = 31901 (0.314 sec)\n",
      "INFO:tensorflow:global_step/sec: 280.27\n",
      "INFO:tensorflow:loss = 0.0047290307, step = 32001 (0.357 sec)\n",
      "INFO:tensorflow:global_step/sec: 257.218\n",
      "INFO:tensorflow:loss = 0.023013934, step = 32101 (0.388 sec)\n",
      "INFO:tensorflow:global_step/sec: 313.659\n",
      "INFO:tensorflow:loss = 0.046587117, step = 32201 (0.319 sec)\n",
      "INFO:tensorflow:global_step/sec: 318.651\n",
      "INFO:tensorflow:loss = 0.040482484, step = 32301 (0.314 sec)\n",
      "INFO:tensorflow:global_step/sec: 311.706\n",
      "INFO:tensorflow:loss = 0.046570405, step = 32401 (0.321 sec)\n",
      "INFO:tensorflow:global_step/sec: 312.68\n",
      "INFO:tensorflow:loss = 0.03138489, step = 32501 (0.320 sec)\n",
      "INFO:tensorflow:global_step/sec: 313.658\n",
      "INFO:tensorflow:loss = 0.093612075, step = 32601 (0.319 sec)\n",
      "INFO:tensorflow:global_step/sec: 302.288\n",
      "INFO:tensorflow:loss = 0.04598008, step = 32701 (0.331 sec)\n",
      "INFO:tensorflow:global_step/sec: 315.637\n",
      "INFO:tensorflow:loss = 0.022406077, step = 32801 (0.318 sec)\n",
      "INFO:tensorflow:global_step/sec: 307.868\n",
      "INFO:tensorflow:loss = 0.009902868, step = 32901 (0.324 sec)\n",
      "INFO:tensorflow:global_step/sec: 311.705\n",
      "INFO:tensorflow:loss = 0.010503637, step = 33001 (0.321 sec)\n",
      "INFO:tensorflow:global_step/sec: 271.895\n",
      "INFO:tensorflow:loss = 0.008153386, step = 33101 (0.369 sec)\n",
      "INFO:tensorflow:global_step/sec: 268.25\n",
      "INFO:tensorflow:loss = 0.0242204, step = 33201 (0.373 sec)\n",
      "INFO:tensorflow:global_step/sec: 280.273\n",
      "INFO:tensorflow:loss = 0.045104966, step = 33301 (0.356 sec)\n",
      "INFO:tensorflow:global_step/sec: 310.735\n",
      "INFO:tensorflow:loss = 0.055412624, step = 33401 (0.322 sec)\n",
      "INFO:tensorflow:global_step/sec: 312.68\n",
      "INFO:tensorflow:loss = 0.027127815, step = 33501 (0.323 sec)\n",
      "INFO:tensorflow:global_step/sec: 306.925\n",
      "INFO:tensorflow:loss = 0.019180536, step = 33601 (0.323 sec)\n",
      "INFO:tensorflow:global_step/sec: 311.706\n",
      "INFO:tensorflow:loss = 0.016074212, step = 33701 (0.322 sec)\n",
      "INFO:tensorflow:global_step/sec: 307.868\n",
      "INFO:tensorflow:loss = 0.057565574, step = 33801 (0.324 sec)\n",
      "INFO:tensorflow:global_step/sec: 307.867\n",
      "INFO:tensorflow:loss = 0.045991756, step = 33901 (0.326 sec)\n",
      "INFO:tensorflow:global_step/sec: 311.706\n",
      "INFO:tensorflow:loss = 0.045432944, step = 34001 (0.321 sec)\n",
      "INFO:tensorflow:global_step/sec: 310.737\n",
      "INFO:tensorflow:loss = 0.031044275, step = 34101 (0.322 sec)\n",
      "INFO:tensorflow:global_step/sec: 266.11\n",
      "INFO:tensorflow:loss = 0.048024308, step = 34201 (0.376 sec)\n",
      "INFO:tensorflow:global_step/sec: 270.424\n",
      "INFO:tensorflow:loss = 0.01743571, step = 34301 (0.369 sec)\n",
      "INFO:tensorflow:global_step/sec: 315.639\n",
      "INFO:tensorflow:loss = 0.011522114, step = 34401 (0.318 sec)\n",
      "INFO:tensorflow:global_step/sec: 286.698\n",
      "INFO:tensorflow:loss = 0.059120677, step = 34501 (0.349 sec)\n",
      "INFO:tensorflow:global_step/sec: 305.052\n",
      "INFO:tensorflow:loss = 0.07258765, step = 34601 (0.328 sec)\n",
      "INFO:tensorflow:global_step/sec: 299.573\n",
      "INFO:tensorflow:loss = 0.025305383, step = 34701 (0.334 sec)\n",
      "INFO:tensorflow:global_step/sec: 313.658\n",
      "INFO:tensorflow:loss = 0.0017969636, step = 34801 (0.319 sec)\n",
      "INFO:tensorflow:global_step/sec: 315.637\n",
      "INFO:tensorflow:loss = 0.012230197, step = 34901 (0.317 sec)\n",
      "INFO:tensorflow:global_step/sec: 304.125\n",
      "INFO:tensorflow:loss = 0.0042318306, step = 35001 (0.328 sec)\n",
      "INFO:tensorflow:global_step/sec: 306.925\n",
      "INFO:tensorflow:loss = 0.019902552, step = 35101 (0.326 sec)\n",
      "INFO:tensorflow:global_step/sec: 303.202\n",
      "INFO:tensorflow:loss = 0.047129538, step = 35201 (0.331 sec)\n",
      "INFO:tensorflow:global_step/sec: 266.11\n",
      "INFO:tensorflow:loss = 0.012289593, step = 35301 (0.376 sec)\n",
      "INFO:tensorflow:global_step/sec: 280.273\n",
      "INFO:tensorflow:loss = 0.0081375465, step = 35401 (0.356 sec)\n",
      "INFO:tensorflow:global_step/sec: 312.679\n",
      "INFO:tensorflow:loss = 0.0071831895, step = 35501 (0.321 sec)\n",
      "INFO:tensorflow:global_step/sec: 306.925\n",
      "INFO:tensorflow:loss = 0.023916937, step = 35601 (0.335 sec)\n",
      "INFO:tensorflow:global_step/sec: 305.986\n",
      "INFO:tensorflow:loss = 0.0053602573, step = 35701 (0.318 sec)\n",
      "INFO:tensorflow:global_step/sec: 311.704\n",
      "INFO:tensorflow:loss = 0.026793573, step = 35801 (0.320 sec)\n",
      "INFO:tensorflow:global_step/sec: 305.05\n",
      "INFO:tensorflow:loss = 0.008112377, step = 35901 (0.328 sec)\n",
      "INFO:tensorflow:global_step/sec: 313.661\n",
      "INFO:tensorflow:loss = 0.007999109, step = 36001 (0.320 sec)\n",
      "INFO:tensorflow:global_step/sec: 308.816\n",
      "INFO:tensorflow:loss = 0.011636932, step = 36101 (0.324 sec)\n",
      "INFO:tensorflow:global_step/sec: 308.82\n",
      "INFO:tensorflow:loss = 0.018871466, step = 36201 (0.325 sec)\n",
      "INFO:tensorflow:global_step/sec: 297.791\n",
      "INFO:tensorflow:loss = 0.0063564354, step = 36301 (0.335 sec)\n",
      "INFO:tensorflow:global_step/sec: 262.616\n",
      "INFO:tensorflow:loss = 0.009504318, step = 36401 (0.381 sec)\n",
      "INFO:tensorflow:global_step/sec: 282.648\n",
      "INFO:tensorflow:loss = 0.039306246, step = 36501 (0.354 sec)\n",
      "INFO:tensorflow:global_step/sec: 305.052\n",
      "INFO:tensorflow:loss = 0.013517313, step = 36601 (0.327 sec)\n",
      "INFO:tensorflow:global_step/sec: 306.924\n",
      "INFO:tensorflow:loss = 0.022520568, step = 36701 (0.326 sec)\n",
      "INFO:tensorflow:global_step/sec: 311.702\n",
      "INFO:tensorflow:loss = 0.007569881, step = 36801 (0.324 sec)\n",
      "INFO:tensorflow:global_step/sec: 310.738\n",
      "INFO:tensorflow:loss = 0.009286598, step = 36901 (0.320 sec)\n",
      "INFO:tensorflow:global_step/sec: 313.661\n",
      "INFO:tensorflow:loss = 0.080708764, step = 37001 (0.318 sec)\n",
      "INFO:tensorflow:global_step/sec: 316.635\n",
      "INFO:tensorflow:loss = 0.028070025, step = 37101 (0.316 sec)\n",
      "INFO:tensorflow:global_step/sec: 299.573\n",
      "INFO:tensorflow:loss = 0.0060724905, step = 37201 (0.334 sec)\n",
      "INFO:tensorflow:global_step/sec: 313.66\n",
      "INFO:tensorflow:loss = 0.014565912, step = 37301 (0.319 sec)\n",
      "INFO:tensorflow:global_step/sec: 303.203\n",
      "INFO:tensorflow:loss = 0.026503686, step = 37401 (0.331 sec)\n",
      "INFO:tensorflow:global_step/sec: 261.929\n",
      "INFO:tensorflow:loss = 0.024665147, step = 37501 (0.382 sec)\n",
      "INFO:tensorflow:global_step/sec: 290.864\n",
      "INFO:tensorflow:loss = 0.008522366, step = 37601 (0.344 sec)\n",
      "INFO:tensorflow:global_step/sec: 309.775\n",
      "INFO:tensorflow:loss = 0.035612132, step = 37701 (0.322 sec)\n",
      "INFO:tensorflow:global_step/sec: 313.656\n",
      "INFO:tensorflow:loss = 0.033741776, step = 37801 (0.320 sec)\n",
      "INFO:tensorflow:global_step/sec: 314.65\n",
      "INFO:tensorflow:loss = 0.098513946, step = 37901 (0.317 sec)\n",
      "INFO:tensorflow:global_step/sec: 316.636\n",
      "INFO:tensorflow:loss = 0.0039267414, step = 38001 (0.317 sec)\n",
      "INFO:tensorflow:global_step/sec: 317.641\n",
      "INFO:tensorflow:loss = 0.012090231, step = 38101 (0.314 sec)\n",
      "INFO:tensorflow:global_step/sec: 309.775\n",
      "INFO:tensorflow:loss = 0.017174866, step = 38201 (0.324 sec)\n",
      "INFO:tensorflow:global_step/sec: 314.647\n",
      "INFO:tensorflow:loss = 0.028272625, step = 38301 (0.317 sec)\n",
      "INFO:tensorflow:global_step/sec: 316.636\n",
      "INFO:tensorflow:loss = 0.010637571, step = 38401 (0.317 sec)\n",
      "INFO:tensorflow:global_step/sec: 287.521\n",
      "INFO:tensorflow:loss = 0.01467517, step = 38501 (0.351 sec)\n",
      "INFO:tensorflow:global_step/sec: 260.565\n",
      "INFO:tensorflow:loss = 0.0316329, step = 38601 (0.381 sec)\n",
      "INFO:tensorflow:global_step/sec: 307.868\n",
      "INFO:tensorflow:loss = 0.017744789, step = 38701 (0.324 sec)\n",
      "INFO:tensorflow:global_step/sec: 305.986\n",
      "INFO:tensorflow:loss = 0.0016748887, step = 38801 (0.327 sec)\n",
      "INFO:tensorflow:global_step/sec: 306.924\n",
      "INFO:tensorflow:loss = 0.026879612, step = 38901 (0.326 sec)\n",
      "INFO:tensorflow:global_step/sec: 312.679\n",
      "INFO:tensorflow:loss = 0.002825636, step = 39001 (0.321 sec)\n",
      "INFO:tensorflow:global_step/sec: 309.774\n",
      "INFO:tensorflow:loss = 0.013494415, step = 39101 (0.323 sec)\n",
      "INFO:tensorflow:global_step/sec: 296.028\n",
      "INFO:tensorflow:loss = 0.0009228169, step = 39201 (0.337 sec)\n",
      "INFO:tensorflow:global_step/sec: 310.737\n",
      "INFO:tensorflow:loss = 0.014490133, step = 39301 (0.322 sec)\n",
      "INFO:tensorflow:global_step/sec: 312.678\n",
      "INFO:tensorflow:loss = 0.003362064, step = 39401 (0.320 sec)\n",
      "INFO:tensorflow:global_step/sec: 305.053\n",
      "INFO:tensorflow:loss = 0.04338023, step = 39501 (0.328 sec)\n",
      "INFO:tensorflow:global_step/sec: 259.216\n",
      "INFO:tensorflow:loss = 0.007867888, step = 39601 (0.387 sec)\n",
      "INFO:tensorflow:global_step/sec: 257.216\n",
      "INFO:tensorflow:loss = 0.023370536, step = 39701 (0.389 sec)\n",
      "INFO:tensorflow:global_step/sec: 305.052\n",
      "INFO:tensorflow:loss = 0.0021050232, step = 39801 (0.327 sec)\n",
      "INFO:tensorflow:global_step/sec: 307.869\n",
      "INFO:tensorflow:loss = 0.02911074, step = 39901 (0.325 sec)\n",
      "INFO:tensorflow:global_step/sec: 299.573\n",
      "INFO:tensorflow:loss = 0.00078155927, step = 40001 (0.334 sec)\n",
      "INFO:tensorflow:global_step/sec: 296.906\n",
      "INFO:tensorflow:loss = 0.004766836, step = 40101 (0.338 sec)\n",
      "INFO:tensorflow:global_step/sec: 305.051\n",
      "INFO:tensorflow:loss = 0.023831312, step = 40201 (0.327 sec)\n",
      "INFO:tensorflow:global_step/sec: 298.679\n",
      "INFO:tensorflow:loss = 0.010768136, step = 40301 (0.336 sec)\n",
      "INFO:tensorflow:global_step/sec: 277.936\n",
      "INFO:tensorflow:loss = 0.02719651, step = 40401 (0.359 sec)\n",
      "INFO:tensorflow:global_step/sec: 293.177\n",
      "INFO:tensorflow:loss = 0.02909667, step = 40501 (0.342 sec)\n",
      "INFO:tensorflow:global_step/sec: 283.445\n",
      "INFO:tensorflow:loss = 0.00999298, step = 40601 (0.353 sec)\n",
      "INFO:tensorflow:global_step/sec: 269.698\n",
      "INFO:tensorflow:loss = 0.015468199, step = 40701 (0.371 sec)\n",
      "INFO:tensorflow:global_step/sec: 263.308\n",
      "INFO:tensorflow:loss = 0.011007725, step = 40801 (0.379 sec)\n",
      "INFO:tensorflow:global_step/sec: 306.922\n",
      "INFO:tensorflow:loss = 0.00020359816, step = 40901 (0.327 sec)\n",
      "INFO:tensorflow:global_step/sec: 307.87\n",
      "INFO:tensorflow:loss = 0.08769333, step = 41001 (0.324 sec)\n",
      "INFO:tensorflow:global_step/sec: 305.052\n",
      "INFO:tensorflow:loss = 0.042994045, step = 41101 (0.328 sec)\n",
      "INFO:tensorflow:global_step/sec: 290.864\n",
      "INFO:tensorflow:loss = 0.057795696, step = 41201 (0.344 sec)\n",
      "INFO:tensorflow:global_step/sec: 300.473\n",
      "INFO:tensorflow:loss = 0.048708417, step = 41301 (0.333 sec)\n",
      "INFO:tensorflow:global_step/sec: 303.204\n",
      "INFO:tensorflow:loss = 0.041317504, step = 41401 (0.330 sec)\n",
      "INFO:tensorflow:global_step/sec: 319.671\n",
      "INFO:tensorflow:loss = 0.0045223865, step = 41501 (0.315 sec)\n",
      "INFO:tensorflow:global_step/sec: 315.637\n",
      "INFO:tensorflow:loss = 0.032584872, step = 41601 (0.315 sec)\n",
      "INFO:tensorflow:global_step/sec: 296.028\n",
      "INFO:tensorflow:loss = 0.026558138, step = 41701 (0.339 sec)\n",
      "INFO:tensorflow:global_step/sec: 262.617\n",
      "INFO:tensorflow:loss = 0.026443962, step = 41801 (0.381 sec)\n",
      "INFO:tensorflow:global_step/sec: 268.971\n",
      "INFO:tensorflow:loss = 0.03169131, step = 41901 (0.371 sec)\n",
      "INFO:tensorflow:global_step/sec: 308.818\n",
      "INFO:tensorflow:loss = 0.01790867, step = 42001 (0.324 sec)\n",
      "INFO:tensorflow:global_step/sec: 318.653\n",
      "INFO:tensorflow:loss = 0.0152394, step = 42101 (0.314 sec)\n",
      "INFO:tensorflow:global_step/sec: 314.646\n",
      "INFO:tensorflow:loss = 0.026577417, step = 42201 (0.318 sec)\n",
      "INFO:tensorflow:global_step/sec: 314.646\n",
      "INFO:tensorflow:loss = 0.12125861, step = 42301 (0.319 sec)\n",
      "INFO:tensorflow:global_step/sec: 317.642\n",
      "INFO:tensorflow:loss = 0.0232108, step = 42401 (0.315 sec)\n",
      "INFO:tensorflow:global_step/sec: 310.736\n",
      "INFO:tensorflow:loss = 0.0049465625, step = 42501 (0.322 sec)\n",
      "INFO:tensorflow:global_step/sec: 318.655\n",
      "INFO:tensorflow:loss = 0.020478416, step = 42601 (0.314 sec)\n",
      "INFO:tensorflow:global_step/sec: 314.645\n",
      "INFO:tensorflow:loss = 0.01595322, step = 42701 (0.317 sec)\n",
      "INFO:tensorflow:global_step/sec: 309.774\n",
      "INFO:tensorflow:loss = 0.018952638, step = 42801 (0.323 sec)\n",
      "INFO:tensorflow:global_step/sec: 263.309\n",
      "INFO:tensorflow:loss = 0.0024844366, step = 42901 (0.381 sec)\n",
      "INFO:tensorflow:global_step/sec: 285.878\n",
      "INFO:tensorflow:loss = 0.038171377, step = 43001 (0.350 sec)\n",
      "INFO:tensorflow:global_step/sec: 310.737\n",
      "INFO:tensorflow:loss = 0.013620295, step = 43101 (0.321 sec)\n",
      "INFO:tensorflow:global_step/sec: 308.817\n",
      "INFO:tensorflow:loss = 0.019763667, step = 43201 (0.325 sec)\n",
      "INFO:tensorflow:global_step/sec: 310.738\n",
      "INFO:tensorflow:loss = 0.0010194365, step = 43301 (0.321 sec)\n",
      "INFO:tensorflow:global_step/sec: 310.735\n",
      "INFO:tensorflow:loss = 0.0024939901, step = 43401 (0.326 sec)\n",
      "INFO:tensorflow:global_step/sec: 314.647\n",
      "INFO:tensorflow:loss = 0.007205924, step = 43501 (0.314 sec)\n",
      "INFO:tensorflow:global_step/sec: 311.705\n",
      "INFO:tensorflow:loss = 0.00799268, step = 43601 (0.322 sec)\n",
      "INFO:tensorflow:global_step/sec: 314.646\n",
      "INFO:tensorflow:loss = 0.020351749, step = 43701 (0.318 sec)\n",
      "INFO:tensorflow:global_step/sec: 308.818\n",
      "INFO:tensorflow:loss = 0.013508269, step = 43801 (0.323 sec)\n",
      "INFO:tensorflow:global_step/sec: 296.027\n",
      "INFO:tensorflow:loss = 0.009314031, step = 43901 (0.339 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 44000 into C:\\Users\\jongm\\AppData\\Local\\Temp\\tmp16z4p7tg\\model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.013671358.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.estimator.canned.dnn.DNNClassifier at 0x23442a061d0>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# TF Estimator\n",
    "feature_cols = [tf.feature_column.numeric_column(\"X\", shape=[28 * 28])]\n",
    "dnn_clf = tf.estimator.DNNClassifier(hidden_units=[300, 100], n_classes=10,\n",
    "                                     feature_columns=feature_cols)\n",
    "\n",
    "# input function\n",
    "input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "    x={\"X\": X_train}, y=y_train, num_epochs=40, batch_size=50, shuffle=True)\n",
    "\n",
    "# training\n",
    "dnn_clf.train(input_fn=input_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2018-09-12-17:43:39\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from C:\\Users\\jongm\\AppData\\Local\\Temp\\tmp16z4p7tg\\model.ckpt-44000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2018-09-12-17:43:39\n",
      "INFO:tensorflow:Saving dict for global step 44000: accuracy = 0.98, average_loss = 0.10493586, global_step = 44000, loss = 13.283021\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 44000: C:\\Users\\jongm\\AppData\\Local\\Temp\\tmp16z4p7tg\\model.ckpt-44000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.98,\n",
       " 'average_loss': 0.10493586,\n",
       " 'loss': 13.283021,\n",
       " 'global_step': 44000}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Test input function\n",
    "test_input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "    x={\"X\": X_test}, y=y_test, shuffle=False)\n",
    "\n",
    "# Test\n",
    "eval_results = dnn_clf.evaluate(input_fn=test_input_fn)\n",
    "eval_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10.3 텐서플로의 저수준 API로 심층 신경망 학습하기\n",
    "\n",
    "아래의 예제는 위에서 MNIST 분류기를 `Estimator`로 구현한 것을 이번에는 텐서플로 Low-Level로 구현한 것이다.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "# MNIST Dataset Load!\n",
    "(X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
    "\n",
    "# reshape : 28 x 28 -> 784\n",
    "X_train = X_train.astype(np.float32).reshape(-1, 28*28) / 255.0\n",
    "X_test = X_test.astype(np.float32).reshape(-1, 28*28) / 255.0\n",
    "y_train = y_train.astype(np.int32)\n",
    "y_test = y_test.astype(np.int32)\n",
    "\n",
    "# split validation set\n",
    "X_valid, X_train = X_train[:5000], X_train[5000:]\n",
    "y_valid, y_train = y_train[:5000], y_train[5000:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "################\n",
    "# layer params #\n",
    "################\n",
    "n_inputs = 28*28\n",
    "n_hidden1 = 300\n",
    "n_hidden2 = 100\n",
    "n_outputs = 10\n",
    "\n",
    "inputs = tf.placeholder(tf.float32, shape=[None, n_inputs], name=\"inputs\")\n",
    "labels = tf.placeholder(tf.int32, shape=[None], name=\"outputs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def neuron_layer(inputs, n_neurons, name, activation=None):\n",
    "    with tf.name_scope(name):\n",
    "        n_inputs = int(inputs.get_shape()[1])\n",
    "        stddev = 2 / np.sqrt(n_inputs)\n",
    "        init = tf.truncated_normal([n_inputs, n_neurons], stddev=stddev)\n",
    "        weights = tf.Variable(init, name=\"weights\")\n",
    "        bias = tf.Variable(tf.zeros([n_neurons]), name='bias')\n",
    "        Z = tf.matmul(inputs, weights) + bias\n",
    "        if activation:\n",
    "            return activation(Z)\n",
    "        else:\n",
    "            return Z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.name_scope('dnn'):\n",
    "    hidden1 = neuron_layer(inputs, n_hidden1, name='hidden1', \n",
    "                           activation=tf.nn.relu)\n",
    "    hidden2 = neuron_layer(hidden1, n_hidden2, name='hidden2',\n",
    "                           activation=tf.nn.relu)\n",
    "    logits = neuron_layer(hidden2, n_outputs, name='logits')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.name_scope('loss'):\n",
    "    cross_entropy = tf.reduce_mean(\n",
    "        tf.nn.sparse_softmax_cross_entropy_with_logits(labels=labels, logits=logits))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "################\n",
    "# Hyper-params #\n",
    "################\n",
    "learning_rate = 0.01\n",
    "n_epochs = 40\n",
    "batch_size = 50\n",
    "\n",
    "with tf.name_scope('train'):\n",
    "    optimizer = tf.train.GradientDescentOptimizer(learning_rate)\n",
    "    train_op = optimizer.minimize(cross_entropy)\n",
    "\n",
    "with tf.name_scope('eval'):\n",
    "    correct = tf.nn.in_top_k(predictions=logits, targets=labels, k=1)\n",
    "    accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def shuffle_batch(X, y, batch_size):\n",
    "    rnd_idx = np.random.permutation(len(X))\n",
    "    n_batches = len(X) // batch_size\n",
    "    for batch_idx in np.array_split(rnd_idx, n_batches):\n",
    "        X_batch, y_batch = X[batch_idx], y[batch_idx]\n",
    "        yield X_batch, y_batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 000, bacth acc: 0.9000, valid acc: 0.9164\n",
      "epoch: 001, bacth acc: 0.9000, valid acc: 0.9364\n",
      "epoch: 002, bacth acc: 0.9000, valid acc: 0.9476\n",
      "epoch: 003, bacth acc: 0.9600, valid acc: 0.9526\n",
      "epoch: 004, bacth acc: 0.9600, valid acc: 0.9528\n",
      "epoch: 005, bacth acc: 0.9600, valid acc: 0.9586\n",
      "epoch: 006, bacth acc: 1.0000, valid acc: 0.9612\n",
      "epoch: 007, bacth acc: 0.9400, valid acc: 0.9632\n",
      "epoch: 008, bacth acc: 1.0000, valid acc: 0.9638\n",
      "epoch: 009, bacth acc: 1.0000, valid acc: 0.9660\n",
      "epoch: 010, bacth acc: 1.0000, valid acc: 0.9684\n",
      "epoch: 011, bacth acc: 0.9800, valid acc: 0.9698\n",
      "epoch: 012, bacth acc: 1.0000, valid acc: 0.9698\n",
      "epoch: 013, bacth acc: 1.0000, valid acc: 0.9706\n",
      "epoch: 014, bacth acc: 0.9800, valid acc: 0.9710\n",
      "epoch: 015, bacth acc: 0.9800, valid acc: 0.9734\n",
      "epoch: 016, bacth acc: 0.9800, valid acc: 0.9722\n",
      "epoch: 017, bacth acc: 0.9800, valid acc: 0.9742\n",
      "epoch: 018, bacth acc: 0.9400, valid acc: 0.9752\n",
      "epoch: 019, bacth acc: 1.0000, valid acc: 0.9752\n",
      "epoch: 020, bacth acc: 0.9400, valid acc: 0.9746\n",
      "epoch: 021, bacth acc: 0.9400, valid acc: 0.9756\n",
      "epoch: 022, bacth acc: 0.9800, valid acc: 0.9750\n",
      "epoch: 023, bacth acc: 0.9600, valid acc: 0.9762\n",
      "epoch: 024, bacth acc: 0.9600, valid acc: 0.9774\n",
      "epoch: 025, bacth acc: 0.9800, valid acc: 0.9792\n",
      "epoch: 026, bacth acc: 1.0000, valid acc: 0.9768\n",
      "epoch: 027, bacth acc: 0.9800, valid acc: 0.9776\n",
      "epoch: 028, bacth acc: 0.9800, valid acc: 0.9786\n",
      "epoch: 029, bacth acc: 1.0000, valid acc: 0.9774\n",
      "epoch: 030, bacth acc: 0.9800, valid acc: 0.9792\n",
      "epoch: 031, bacth acc: 1.0000, valid acc: 0.9782\n",
      "epoch: 032, bacth acc: 0.9800, valid acc: 0.9782\n",
      "epoch: 033, bacth acc: 0.9800, valid acc: 0.9786\n",
      "epoch: 034, bacth acc: 1.0000, valid acc: 0.9786\n",
      "epoch: 035, bacth acc: 1.0000, valid acc: 0.9802\n",
      "epoch: 036, bacth acc: 1.0000, valid acc: 0.9792\n",
      "epoch: 037, bacth acc: 1.0000, valid acc: 0.9784\n",
      "epoch: 038, bacth acc: 0.9800, valid acc: 0.9798\n",
      "epoch: 039, bacth acc: 1.0000, valid acc: 0.9780\n"
     ]
    }
   ],
   "source": [
    "# train\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    for epoch in range(n_epochs):\n",
    "        for X_batch, y_batch in shuffle_batch(X_train, y_train, batch_size):\n",
    "            sess.run(train_op, feed_dict={inputs: X_batch,\n",
    "                                          labels: y_batch})\n",
    "        \n",
    "        acc_batch = accuracy.eval(feed_dict={inputs: X_batch, labels: y_batch})\n",
    "        acc_valid = accuracy.eval(feed_dict={inputs: X_valid, labels: y_valid})\n",
    "        print('epoch: {:03d}, bacth acc: {:.4f}, valid acc: {:.4f}'.format(epoch,\n",
    "                                                                          acc_batch,\n",
    "                                                                          acc_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `neuron_layer()` 대신 `tf.layers.dense()` 사용"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "# MNIST Dataset Load!\n",
    "(X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
    "\n",
    "# reshape : 28 x 28 -> 784\n",
    "X_train = X_train.astype(np.float32).reshape(-1, 28*28) / 255.0\n",
    "X_test = X_test.astype(np.float32).reshape(-1, 28*28) / 255.0\n",
    "y_train = y_train.astype(np.int32)\n",
    "y_test = y_test.astype(np.int32)\n",
    "\n",
    "# split validation set\n",
    "X_valid, X_train = X_train[:5000], X_train[5000:]\n",
    "y_valid, y_train = y_train[:5000], y_train[5000:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "################\n",
    "# layer params #\n",
    "################\n",
    "n_inputs = 28*28\n",
    "n_hidden1 = 300\n",
    "n_hidden2 = 100\n",
    "n_outputs = 10\n",
    "\n",
    "inputs = tf.placeholder(tf.float32, shape=[None, n_inputs], name=\"inputs\")\n",
    "labels = tf.placeholder(tf.int32, shape=[None], name=\"outputs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.name_scope('dnn'):\n",
    "    hidden1 = tf.layers.dense(inputs=inputs, units=n_hidden1,\n",
    "                              activation=tf.nn.relu, name='hidden1',\n",
    "                              kernel_initializer=tf.truncated_normal_initializer(stddev=0.1))\n",
    "    hidden2 = tf.layers.dense(hidden1, n_hidden2,\n",
    "                              activation=tf.nn.relu, name='hidden2',\n",
    "                              kernel_initializer=tf.truncated_normal_initializer(stddev=0.1))\n",
    "    logits = tf.layers.dense(hidden2, n_outputs, name='logits')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.name_scope('loss'):\n",
    "    cross_entropy = tf.reduce_mean(\n",
    "        tf.nn.sparse_softmax_cross_entropy_with_logits(labels=labels, logits=logits))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "################\n",
    "# Hyper-params #\n",
    "################\n",
    "learning_rate = 0.01\n",
    "n_epochs = 40\n",
    "batch_size = 50\n",
    "\n",
    "with tf.name_scope('train'):\n",
    "    optimizer = tf.train.GradientDescentOptimizer(learning_rate)\n",
    "    train_op = optimizer.minimize(cross_entropy)\n",
    "\n",
    "with tf.name_scope('eval'):\n",
    "    correct = tf.nn.in_top_k(predictions=logits, targets=labels, k=1)\n",
    "    accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def shuffle_batch(X, y, batch_size):\n",
    "    rnd_idx = np.random.permutation(len(X))\n",
    "    n_batches = len(X) // batch_size\n",
    "    for batch_idx in np.array_split(rnd_idx, n_batches):\n",
    "        X_batch, y_batch = X[batch_idx], y[batch_idx]\n",
    "        yield X_batch, y_batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 000, bacth acc: 0.8600, valid acc: 0.9106\n",
      "epoch: 001, bacth acc: 0.9200, valid acc: 0.9256\n",
      "epoch: 002, bacth acc: 0.9600, valid acc: 0.9358\n",
      "epoch: 003, bacth acc: 0.9000, valid acc: 0.9418\n",
      "epoch: 004, bacth acc: 0.9600, valid acc: 0.9458\n",
      "epoch: 005, bacth acc: 0.9600, valid acc: 0.9484\n",
      "epoch: 006, bacth acc: 1.0000, valid acc: 0.9544\n",
      "epoch: 007, bacth acc: 0.9400, valid acc: 0.9582\n",
      "epoch: 008, bacth acc: 0.9600, valid acc: 0.9592\n",
      "epoch: 009, bacth acc: 0.9600, valid acc: 0.9624\n",
      "epoch: 010, bacth acc: 0.9400, valid acc: 0.9660\n",
      "epoch: 011, bacth acc: 0.9800, valid acc: 0.9640\n",
      "epoch: 012, bacth acc: 0.9800, valid acc: 0.9632\n",
      "epoch: 013, bacth acc: 0.9800, valid acc: 0.9662\n",
      "epoch: 014, bacth acc: 1.0000, valid acc: 0.9668\n",
      "epoch: 015, bacth acc: 0.9400, valid acc: 0.9678\n",
      "epoch: 016, bacth acc: 0.9800, valid acc: 0.9706\n",
      "epoch: 017, bacth acc: 1.0000, valid acc: 0.9682\n",
      "epoch: 018, bacth acc: 0.9800, valid acc: 0.9706\n",
      "epoch: 019, bacth acc: 0.9800, valid acc: 0.9706\n",
      "epoch: 020, bacth acc: 1.0000, valid acc: 0.9710\n",
      "epoch: 021, bacth acc: 1.0000, valid acc: 0.9720\n",
      "epoch: 022, bacth acc: 0.9600, valid acc: 0.9716\n",
      "epoch: 023, bacth acc: 0.9800, valid acc: 0.9728\n",
      "epoch: 024, bacth acc: 0.9800, valid acc: 0.9728\n",
      "epoch: 025, bacth acc: 1.0000, valid acc: 0.9732\n",
      "epoch: 026, bacth acc: 0.9400, valid acc: 0.9724\n",
      "epoch: 027, bacth acc: 1.0000, valid acc: 0.9728\n",
      "epoch: 028, bacth acc: 0.9400, valid acc: 0.9720\n",
      "epoch: 029, bacth acc: 0.9600, valid acc: 0.9738\n",
      "epoch: 030, bacth acc: 0.9800, valid acc: 0.9720\n",
      "epoch: 031, bacth acc: 0.9800, valid acc: 0.9734\n",
      "epoch: 032, bacth acc: 0.9600, valid acc: 0.9746\n",
      "epoch: 033, bacth acc: 0.9800, valid acc: 0.9736\n",
      "epoch: 034, bacth acc: 0.9800, valid acc: 0.9740\n",
      "epoch: 035, bacth acc: 0.9800, valid acc: 0.9730\n",
      "epoch: 036, bacth acc: 0.9800, valid acc: 0.9726\n",
      "epoch: 037, bacth acc: 1.0000, valid acc: 0.9742\n",
      "epoch: 038, bacth acc: 0.9800, valid acc: 0.9742\n",
      "epoch: 039, bacth acc: 1.0000, valid acc: 0.9736\n"
     ]
    }
   ],
   "source": [
    "# train\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    for epoch in range(n_epochs):\n",
    "        for X_batch, y_batch in shuffle_batch(X_train, y_train, batch_size):\n",
    "            sess.run(train_op, feed_dict={inputs: X_batch,\n",
    "                                          labels: y_batch})\n",
    "        \n",
    "        acc_batch = accuracy.eval(feed_dict={inputs: X_batch, labels: y_batch})\n",
    "        acc_valid = accuracy.eval(feed_dict={inputs: X_valid, labels: y_valid})\n",
    "        print('epoch: {:03d}, bacth acc: {:.4f}, valid acc: {:.4f}'.format(epoch,\n",
    "                                                                          acc_batch,\n",
    "                                                                          acc_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\r\n",
       " -->\r\n",
       "<!-- Title: G Pages: 1 -->\r\n",
       "<svg width=\"438pt\" height=\"291pt\"\r\n",
       " viewBox=\"0.00 0.00 437.65 291.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 287)\">\r\n",
       "<title>G</title>\r\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-287 433.646,-287 433.646,4 -4,4\"/>\r\n",
       "<!-- inputs -->\r\n",
       "<g id=\"node1\" class=\"node\"><title>inputs</title>\r\n",
       "<ellipse fill=\"white\" stroke=\"#aaaaaa\" stroke-width=\"2\" cx=\"133.646\" cy=\"-265\" rx=\"32.4942\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"133.646\" y=\"-261.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\" fill=\"#414141\">inputs</text>\r\n",
       "</g>\r\n",
       "<!-- dnn -->\r\n",
       "<g id=\"node4\" class=\"node\"><title>dnn</title>\r\n",
       "<ellipse fill=\"white\" stroke=\"#aaaaaa\" stroke-width=\"2\" cx=\"140.646\" cy=\"-178\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"140.646\" y=\"-174.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\" fill=\"#414141\">dnn</text>\r\n",
       "</g>\r\n",
       "<!-- inputs&#45;&gt;dnn -->\r\n",
       "<g id=\"edge3\" class=\"edge\"><title>inputs&#45;&gt;dnn</title>\r\n",
       "<path fill=\"none\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" d=\"M135.063,-246.799C135.974,-235.739 137.181,-221.084 138.239,-208.23\"/>\r\n",
       "<polygon fill=\"#aaaaaa\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" points=\"142.433,-208.48 139.232,-196.175 134.061,-207.79 142.433,-208.48\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"155.146\" y=\"-217.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\" fill=\"#414141\">?×784</text>\r\n",
       "</g>\r\n",
       "<!-- train -->\r\n",
       "<g id=\"node8\" class=\"node\"><title>train</title>\r\n",
       "<ellipse fill=\"white\" stroke=\"#aaaaaa\" stroke-width=\"2\" cx=\"127.646\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"127.646\" y=\"-14.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\" fill=\"#414141\">train</text>\r\n",
       "</g>\r\n",
       "<!-- inputs&#45;&gt;train -->\r\n",
       "<g id=\"edge11\" class=\"edge\"><title>inputs&#45;&gt;train</title>\r\n",
       "<path fill=\"none\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" d=\"M123.323,-247.911C108.345,-222.671 83.152,-172 90.6465,-127 95.3499,-98.7586 106.526,-68.0015 115.373,-46.5598\"/>\r\n",
       "<polygon fill=\"#aaaaaa\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" points=\"119.291,-48.0784 120.119,-35.3915 111.56,-44.7931 119.291,-48.0784\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"108.146\" y=\"-130.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\" fill=\"#414141\">?×784</text>\r\n",
       "</g>\r\n",
       "<!-- outputs -->\r\n",
       "<g id=\"node2\" class=\"node\"><title>outputs</title>\r\n",
       "<ellipse fill=\"white\" stroke=\"#aaaaaa\" stroke-width=\"2\" cx=\"223.646\" cy=\"-178\" rx=\"38.1938\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"223.646\" y=\"-174.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\" fill=\"#414141\">outputs</text>\r\n",
       "</g>\r\n",
       "<!-- loss -->\r\n",
       "<g id=\"node7\" class=\"node\"><title>loss</title>\r\n",
       "<ellipse fill=\"white\" stroke=\"#aaaaaa\" stroke-width=\"2\" cx=\"162.646\" cy=\"-91\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"162.646\" y=\"-87.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\" fill=\"#414141\">loss</text>\r\n",
       "</g>\r\n",
       "<!-- outputs&#45;&gt;loss -->\r\n",
       "<g id=\"edge5\" class=\"edge\"><title>outputs&#45;&gt;loss</title>\r\n",
       "<path fill=\"none\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" d=\"M211.885,-160.611C203.112,-148.386 190.962,-131.455 180.926,-117.472\"/>\r\n",
       "<polygon fill=\"#aaaaaa\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" points=\"184.164,-114.78 173.755,-107.479 177.339,-119.677 184.164,-114.78\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"201.146\" y=\"-130.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\" fill=\"#414141\">?</text>\r\n",
       "</g>\r\n",
       "<!-- eval -->\r\n",
       "<g id=\"node9\" class=\"node\"><title>eval</title>\r\n",
       "<ellipse fill=\"white\" stroke=\"#aaaaaa\" stroke-width=\"2\" cx=\"234.646\" cy=\"-91\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"234.646\" y=\"-87.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\" fill=\"#414141\">eval</text>\r\n",
       "</g>\r\n",
       "<!-- outputs&#45;&gt;eval -->\r\n",
       "<g id=\"edge13\" class=\"edge\"><title>outputs&#45;&gt;eval</title>\r\n",
       "<path fill=\"none\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" d=\"M225.872,-159.799C227.304,-148.739 229.2,-134.084 230.864,-121.23\"/>\r\n",
       "<polygon fill=\"#aaaaaa\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" points=\"235.049,-121.615 232.424,-109.175 226.718,-120.537 235.049,-121.615\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"234.146\" y=\"-130.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\" fill=\"#414141\">?</text>\r\n",
       "</g>\r\n",
       "<!-- hidden1 -->\r\n",
       "<g id=\"node3\" class=\"node\"><title>hidden1</title>\r\n",
       "<ellipse fill=\"white\" stroke=\"#aaaaaa\" stroke-width=\"2\" cx=\"39.6465\" cy=\"-265\" rx=\"39.7935\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"39.6465\" y=\"-261.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\" fill=\"#414141\">hidden1</text>\r\n",
       "</g>\r\n",
       "<!-- hidden1&#45;&gt;dnn -->\r\n",
       "<g id=\"edge4\" class=\"edge\"><title>hidden1&#45;&gt;dnn</title>\r\n",
       "<path fill=\"none\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" d=\"M57.7033,-248.804C73.6935,-235.347 97.2094,-215.556 115.122,-200.481\"/>\r\n",
       "<polygon fill=\"#aaaaaa\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" points=\"117.937,-203.601 124.414,-192.661 112.528,-197.175 117.937,-203.601\"/>\r\n",
       "</g>\r\n",
       "<!-- hidden1&#45;&gt;train -->\r\n",
       "<g id=\"edge12\" class=\"edge\"><title>hidden1&#45;&gt;train</title>\r\n",
       "<path fill=\"none\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" d=\"M42.1927,-246.993C47.7357,-212.802 62.6967,-133.811 90.6465,-73 95.2353,-63.0159 101.636,-52.8706 107.825,-44.1101\"/>\r\n",
       "<polygon fill=\"#aaaaaa\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" points=\"111.382,-46.3628 115.133,-34.2148 104.626,-41.3722 111.382,-46.3628\"/>\r\n",
       "</g>\r\n",
       "<!-- dnn&#45;&gt;loss -->\r\n",
       "<g id=\"edge6\" class=\"edge\"><title>dnn&#45;&gt;loss</title>\r\n",
       "<path fill=\"none\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" d=\"M144.993,-160.207C147.897,-148.987 151.792,-133.94 155.184,-120.833\"/>\r\n",
       "<polygon fill=\"#aaaaaa\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" points=\"159.293,-121.719 158.234,-109.049 151.161,-119.614 159.293,-121.719\"/>\r\n",
       "</g>\r\n",
       "<!-- dnn&#45;&gt;train -->\r\n",
       "<g id=\"edge9\" class=\"edge\"><title>dnn&#45;&gt;train</title>\r\n",
       "<path fill=\"none\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" d=\"M136.17,-160.101C132.792,-146.432 128.442,-126.627 126.646,-109 125.025,-93.0824 126.498,-88.9993 126.646,-73 126.721,-64.9746 126.857,-56.2791 127.006,-48.2587\"/>\r\n",
       "<polygon fill=\"#aaaaaa\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" points=\"131.21,-48.105 127.251,-36.0234 122.811,-47.937 131.21,-48.105\"/>\r\n",
       "</g>\r\n",
       "<!-- dnn&#45;&gt;eval -->\r\n",
       "<g id=\"edge14\" class=\"edge\"><title>dnn&#45;&gt;eval</title>\r\n",
       "<path fill=\"none\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" d=\"M152.854,-161.767C161.49,-151.427 173.628,-137.737 185.646,-127 191.786,-121.515 198.785,-116.093 205.534,-111.214\"/>\r\n",
       "<polygon fill=\"#aaaaaa\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" points=\"208.34,-114.377 215.774,-104.062 203.531,-107.49 208.34,-114.377\"/>\r\n",
       "</g>\r\n",
       "<!-- hidden2 -->\r\n",
       "<g id=\"node5\" class=\"node\"><title>hidden2</title>\r\n",
       "<ellipse fill=\"white\" stroke=\"#aaaaaa\" stroke-width=\"2\" cx=\"317.646\" cy=\"-265\" rx=\"39.7935\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"317.646\" y=\"-261.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\" fill=\"#414141\">hidden2</text>\r\n",
       "</g>\r\n",
       "<!-- hidden2&#45;&gt;dnn -->\r\n",
       "<g id=\"edge2\" class=\"edge\"><title>hidden2&#45;&gt;dnn</title>\r\n",
       "<path fill=\"none\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" d=\"M291.148,-251.275C259.66,-236.153 207.128,-210.926 173.197,-194.632\"/>\r\n",
       "<polygon fill=\"#aaaaaa\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" points=\"174.672,-190.681 162.036,-189.272 171.035,-198.253 174.672,-190.681\"/>\r\n",
       "</g>\r\n",
       "<!-- hidden2&#45;&gt;train -->\r\n",
       "<g id=\"edge8\" class=\"edge\"><title>hidden2&#45;&gt;train</title>\r\n",
       "<path fill=\"none\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" d=\"M319.635,-247.014C322.901,-211.84 326.046,-129.594 292.646,-73 267.572,-30.5132 207.111,-20.281 166.763,-18.4266\"/>\r\n",
       "<polygon fill=\"#aaaaaa\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" points=\"166.807,-14.2263 154.695,-18.091 166.574,-22.623 166.807,-14.2263\"/>\r\n",
       "</g>\r\n",
       "<!-- logits -->\r\n",
       "<g id=\"node6\" class=\"node\"><title>logits</title>\r\n",
       "<ellipse fill=\"white\" stroke=\"#aaaaaa\" stroke-width=\"2\" cx=\"219.646\" cy=\"-265\" rx=\"30.5947\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"219.646\" y=\"-261.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\" fill=\"#414141\">logits</text>\r\n",
       "</g>\r\n",
       "<!-- logits&#45;&gt;dnn -->\r\n",
       "<g id=\"edge1\" class=\"edge\"><title>logits&#45;&gt;dnn</title>\r\n",
       "<path fill=\"none\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" d=\"M206.819,-248.37C198.447,-238.35 187.159,-225.174 176.646,-214 172.779,-209.889 168.565,-205.619 164.427,-201.532\"/>\r\n",
       "<polygon fill=\"#aaaaaa\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" points=\"167.189,-198.36 155.654,-193.014 161.338,-204.387 167.189,-198.36\"/>\r\n",
       "</g>\r\n",
       "<!-- logits&#45;&gt;train -->\r\n",
       "<g id=\"edge7\" class=\"edge\"><title>logits&#45;&gt;train</title>\r\n",
       "<path fill=\"none\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" d=\"M235.283,-249.224C247.557,-236.473 263.693,-216.86 270.646,-196 287.934,-144.139 301.733,-117.968 270.646,-73 247.43,-39.4167 200.473,-26.6956 166.695,-21.8907\"/>\r\n",
       "<polygon fill=\"#aaaaaa\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" points=\"166.889,-17.6836 154.471,-20.4098 165.878,-26.0226 166.889,-17.6836\"/>\r\n",
       "</g>\r\n",
       "<!-- loss&#45;&gt;train -->\r\n",
       "<g id=\"edge10\" class=\"edge\"><title>loss&#45;&gt;train</title>\r\n",
       "<path fill=\"none\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" d=\"M154.529,-73.5327C150.553,-65.4677 145.66,-55.542 141.143,-46.379\"/>\r\n",
       "<polygon fill=\"#aaaaaa\" stroke=\"#aaaaaa\" stroke-width=\"2.5\" points=\"144.793,-44.2842 135.72,-35.378 137.259,-47.9983 144.793,-44.2842\"/>\r\n",
       "</g>\r\n",
       "<!-- init -->\r\n",
       "<g id=\"node10\" class=\"node\"><title>init</title>\r\n",
       "<ellipse fill=\"white\" stroke=\"#aaaaaa\" stroke-width=\"2\" cx=\"402.646\" cy=\"-265\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"402.646\" y=\"-261.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\" fill=\"#414141\">init</text>\r\n",
       "</g>\r\n",
       "</g>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x23e88ac9048>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import graphviz\n",
    "import tfgraphviz as tfg\n",
    "\n",
    "tf_graph = tfg.board(tf.get_default_graph())\n",
    "tf_graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10.4 신경망 하이퍼파라미터 튜닝하기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.4.1 은닉층의 수\n",
    "\n",
    "1989년 George Cybenko는 뉴런 수만 무한하다면 은닉층 하나로 어떠한 함수도 근사할 수 있다는 것을 밝혔는데, 이를 **시벤코 정리**(Cybenko's theorem)이라고 한다. 실제로도 많은 문제들이 하나의 은닉층에 뉴런 수만 충분하다면 모델링이 가능하다. 하지만, 심층 신경망(DNN)이 얕은 신경망보다 **파라미터 효율성**이 훨씬 좋으며 복잡한 함수를 모델링하는 데에 있어 얕은 신경망보다 훨씬 적은 수의 뉴런을 사용하기 때문에 학습 시간이 더 빠르다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.4.2 은닉층의 뉴런 수\n",
    "\n",
    "위의 MNIST 분류기 예제 코드에서는 은닉층(hidden layer)의 뉴런수를 첫 번째 은닉층(`hidden1`)은 300개, 두 번째 은닉층(`hidden2`)은 100개로 구성하였다. 이처럼 은닉층의 뉴런 수는 일반적으로 각 층의 뉴런을 점점 줄여가는 방법으로 모델링한다. 그 이유는 저수준의 많은 특성(ex. 눈, 코, 입)이 고수준의 적은 특성(ex.얼굴 전체) 으로 합쳐질 수 있기 때문이다.\n",
    "\n",
    "하지만, 요즘에는 모든 은닉층에 동일한 뉴런 수를 사용하며 모델이 오버피팅되기 전까지 점진적으로 뉴런 수를 늘린다. 일반적으로 층의 뉴런 수를 늘리는 것보다 층(layer) 개수를 늘리는 것이 성능이 더 좋아진다.\n",
    "\n",
    "이처럼 최적의 은닉층의 수와 뉴런 수를 찾기란 매우 어려운 일이다. 따라서 실제 필요한 개수 보다 더 많은 층과 뉴런으로 모델링한 후에 오버피팅 되지않도록 조정해나가는 방식을 사용하는 데 이를 '스트레치 팬츠'(stratch pants) 방식이라고 한다(교재에서는 그 이유를 자신에게 알맞는 바지 사이즈를 찾는 데 시간을 낭비하는 대신, 큰 스트레치 팬츠를 사고 알맞게 줄이는게 더 낫기 때문). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tensorflow",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}