Chapter6-Logical-Regression-and-Maximum-Entropy-Model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第6章 逻辑斯谛回归与最大熵模型\n",
    "## 6.1 预备知识（逻辑斯谛回归）\n",
    "逻辑斯谛回归模型<b>假定随机变量X的数据分布符合逻辑斯谛回归</b>,有:  \n",
    ">逻辑斯谛回归分布函数：\n",
    "$$F(x)=P(X \\leq x)=\\frac{1}{1+e^\\left(-\\frac{x-\\mu}{\\gamma}\\right)}$$\n",
    "\n",
    ">密度函数：\n",
    "$$f(x)=F'(x)=\\frac{e^\\left(-\\frac{x-\\mu}{\\gamma}\\right)}{\\gamma(1+e^\\left(-\\frac{x-\\mu}{\\gamma}\\right))^2}$$\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.2 二项逻辑斯谛回归模型\n",
    "抛开上述关于逻辑斯谛回归的预备知识，这里给出二项逻辑斯谛回归模型的<b>定义</b>：\n",
    "><b>给定模型参数$w$和一个样本$x$，逻辑斯谛逻辑回归模型</b>\n",
    "$$P(Y=1|x) = \\frac{e^\\left(wx+b\\right)}{1+e^\\left(wx+b\\right)}$$  \n",
    "$$P(Y=0|x) = \\frac{1}{1+e^\\left(wx+b\\right)}$$\n",
    "\n",
    "其中，$Y \\in \\{0,1\\}$为二分类的类别标签空间，$x$是数据的一条样本记录，通过上述的两个式子得到的结果就是在给定了$x$的情况下，分别得到类别标签为$0$和$1$的概率值，所以，有心的读者可以发现，这个思维和朴素贝叶斯的思路很像，都是通过给定样本$x$得到不同类别标签的概率值，并选择类别概率值最大的类别作为预测类别。那~ 我们为了得到参数$w$和$b$（实际可以理解为只需要求解$w$，如下式），需要做哪些工作呢？\n",
    ">$$P(Y=1|x) = \\frac{e^\\left(wx\\right)}{1+e^\\left(wx\\right)}$$  \n",
    "$$P(Y=0|x) = \\frac{1}{1+e^\\left(wx\\right)}$$\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.3 求解逻辑斯谛回归的参数$w$\n",
    "### 6.3.1 转化问题为似然函数求解问题\n",
    "根据6.2中的式子可以得知\n",
    "$$P(Y=1|x) = \\frac{e^\\left(wx\\right)}{1+e^\\left(wx\\right)}$$\n",
    "$$ = 1 - P(Y=0|x)$$\n",
    "\n",
    "令\n",
    "$$P(Y=1|x) = \\pi(x)$$\n",
    "\n",
    "则有\n",
    "$$P(Y=0|x) = 1- \\pi(x)$$\n",
    "\n",
    "(其中，$\\pi(x)$是由参数$w$决定的函数)\n",
    "$$\\pi(x) = \\frac{e^\\left(wx\\right)}{1+e^\\left(wx\\right)} $$\n",
    ">(1)此时可以直接使用似然函数求解得到$w$的值，其中，作为补充，这里给出似然函数的表达，即有：\n",
    "$$L(w) = \\prod_{i=1}^N[\\pi(x_i)]^{y_i} [1-\\pi(x_i)]^{1-y_i}$$\n",
    "(2)写出对数似然函数\n",
    "$$\\log(L(w)) = \\sum_{i=1}^N[y_i \\log \\pi(x_i) + (1-y_i) \\log(1-\\pi(x_i))]$$\n",
    "(3)此时我们的目的，是求解<b>最大化</b>$L(w)$时的$w$值,即\n",
    "$$w^* = \\arg\\underset{w}{\\max} (\\log(L(w))$$\n",
    "\n",
    "那，如何求解最大化对数函数的参数值呢？处理这种高维度参数的问题，实际上都是使用凸优化的通用的通法（梯度下降或者拟牛顿法来求解极值）了，所以这里使用梯度下降方法来求解参数$w$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.3.2 求解似然函数极值\n",
    "使用<b>梯度下降方法</b>（这里是梯度上升）求解极（大）值\n",
    ">结合之前在感知机模型中关于梯度下降方法的描述，这里首先对$L(w)$求解关于$w$的偏导数，即有\n",
    "$$\\nabla_w L(w)\n",
    "= \\nabla_w (\\sum_{i=1}^N[y_i \\log \\pi(x_i) + (1-y_i) \\log(1-\\pi(x_i))])$$\n",
    "$$= \\sum_{i=1}^N(\\frac{y_i}{\\pi(x_i)} - (1-y_i) \\frac{1}{1-\\pi(x_i)}) \\nabla_w (\\pi(x_i))$$\n",
    "其中\n",
    "$$\\nabla_w (\\pi(x_i))= \\frac{(1 + e^{w x_i}) e^{w x_i} x_i - e^{wx_i}e^{wx_i}x_i}{(1+e^{wx})^2} = x_i \\pi(x_i) (1-\\pi(x_i))$$\n",
    "上式变为\n",
    "$$= \\sum_{i=1}^N (\\frac{y_i}{\\pi(x_i)} - (1-y_i) \\frac{1}{1-\\pi(x_i)}) \\pi(x_i) (1-\\pi(x_i)) x_i$$\n",
    "化简为\n",
    "$$= \\sum_{i=1}^N y_i (1-\\pi(x_i)) x_i - (1-y_i)\\pi(x_i)x_i$$\n",
    "$$= \\sum_{i=1}^N (y_i- \\pi(x_i))x_i$$\n",
    "当给定了步长$\\eta$的值时，所以每次更新参数$w$的操作为：\n",
    ">$$\\Delta w \\leftarrow \\eta \\sum_{i=1}^N (y_i- \\pi(x_i))x_i$$\n",
    "也即有\n",
    "$$w_{t+1} = w_{t} + \\Delta w$$\n",
    "$$ = w_{t} + \\eta \\sum_{i=1}^N (y_i- \\frac{e^\\left(w_t x_i\\right)}{1+e^\\left(w_t x_i\\right)})x_i$$\n",
    "至此，目标函数已经得到，为了求解逻辑斯谛回归的参数$w^*$。接下来将对以上的思路进行求解并编写代码实现：\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.4 如何实现\n",
    "### 6.4.1 自定义数据集\n",
    ">首先定义数据$X$和$Y$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADIJJREFUeJzt3V9onfUdx/HPp21EW2u9sHOdtcYLUWSyuR0KQxiZdVL/\nzW0wUKpXg3MzxbGBKLkYXoSxG+nNLhZUtrFMEbTMP0zXVZNSmH9OtNrWzlGcqaWyRpydpeBa/e4i\nJ0ujifnz/DxP+j3vFxyac/rk93x50Dc/n5wTHRECAOSxrO4BAABlEXYASIawA0AyhB0AkiHsAJAM\nYQeAZFaUWMT225I+lPSxpJMR0SixLgBg4YqEve07EfFewfUAAIvArRgASMYlPnlq+5+S/i0pJP0m\nIgZnOKYpqSlJq1at+uZll11W+bwA0E1GR0ffi4i1cx1XKuxfiYjDtr8kabukOyNi52zHNxqNaLVa\nlc8LAN3E9uh8foZZ5FZMRBxu/3lE0jZJG0usCwBYuMpht73K9urJryVdK2lv1XUBAItT4l0x50va\nZntyvT9GxDMF1gUALELlsEfEW5K+VmAWAEABvN0RAJIh7ACQDGEHgGQIOwAkQ9gBIBnCDgDJEHYA\nSIawA0AyhB0AkiHsAJAMYQeAZAg7ACRD2AEgGcIOAMkQdgBIhrADQDKEHQCSIewAkAxhB4BkCDsA\nJEPYASAZwg4AyRB2AEiGsANAMoQdAJIh7ACQDGEHgGSKhd32ctuv2n6q1JoAgIUruWO/S9L+gusB\n3amvb+IBLFKRsNteL+kGSQ+UWA8AsHgrCq2zVdLdklYXWg/oPpO79JGR6c+Hh2sYBqezyjt22zdK\nOhIRo3Mc17Tdst0aHx+veloAwCwcEdUWsH8p6XZJJyWdKekcSY9HxG2zfU+j0YhWq1XpvEBa7NQx\nC9ujEdGY67jKO/aIuDci1kdEr6RbJD33eVEHAHyxSt1jB1AKO3VUVDTsETEsabjkmgCAheGTpwCQ\nDGEHgGQIOwAkQ9gBIBnCDgDJEHYASIawA0AyhB0AkiHsAJAMYQeAZAg7ACRD2AEgGcIOAMkQdgBI\nhrADQDKEHQCSIewAkAxhB4BkCDsAJEPYASAZwg4AyRB2AEiGsANAMoQdAJIh7ACQDGEHgGQIOwAk\nQ9gBIJnKYbd9pu2XbL9me5/t+0oMBgDq65t4YEFWFFjjI0lXR8Qx2z2Sdtn+c0S8UGBtAMACVQ57\nRISkY+2nPe1HVF0XQBeb3KWPjEx/PjxcwzCnnyL32G0vt71b0hFJ2yPixRmOadpu2W6Nj4+XOC0A\nYAae2HAXWsw+V9I2SXdGxN7Zjms0GtFqtYqdF0BS7NSnsT0aEY25jiv6rpiI+EDSsKTNJdcFAMxf\n5XvsttdKOhERH9g+S9I1kn5VeTIAYKe+KCXeFbNO0u9sL9fEfwE8GhFPFVgXALAIJd4V87qkKwvM\nAgAogE+eAkAyhB0AkiHsAJAMYQeAZAg7ACRD2AEgGcIOAMkQdgBIhrADQDKEHQCSIewAkAxhB4Bk\nCDsAJEPYASAZwg4AyRB2AEiGsANAMoQdAJIh7ACQDGEHgGQIOwAkQ9gBIBnCDgDJEHYASIawA0Ay\nhB0AkiHsAJBM5bDbvtD287b3295n+64SgwGA+vomHliQFQXWOCnp5xHxiu3VkkZtb4+INwqsDQBY\noMphj4h3Jb3b/vpD2/slXSCJsANYnMld+sjI9OfDwzUMc/opeo/ddq+kKyW9OMPfNW23bLfGx8dL\nnhYAcApHRJmF7LMljUgaiIjHP+/YRqMRrVaryHkBJMZOfRrboxHRmOu4Ijt22z2SHpM0NFfUAQBf\nrMr32G1b0oOS9kfE/dVHAoA2duqLUmLHfpWk2yVdbXt3+3F9gXUBAItQ4l0xuyS5wCwAgAL45CkA\nJEPYASAZwg4AyRB2AEiGsANAMoQdAJIh7ACQDGEHgGQIOwAkQ9gBIBnCDgDJEHYASIawA0AyhB0A\nkiHsAJAMYQeAZAg7ACRD2AEgGcIOAMkQdgBIhrADQDKEHQCSIewAkAxhB4BkCDsAJEPYASAZwg4A\nyRQJu+2HbB+xvbfEet1gaM+Qerf2atl9y9S7tVdDe4bqHqk2XAugrFI79t9K2lxorfSG9gyp+WRT\nY0fHFAqNHR1T88lmVwaNawGUVyTsEbFT0vsl1uoG/Tv6dfzE8WmvHT9xXP07+muaqD5cC6C8jt1j\nt9203bLdGh8f79Rpl6SDRw8u6PXMuBZAeR0Le0QMRkQjIhpr167t1GmXpA1rNizo9cy4FkB5vCum\nBgObBrSyZ+W011b2rNTApoGaJqoP1wIoj7DXYMsVWzR406AuWnORLOuiNRdp8KZBbbliS92jdRzX\nAijPEVF9EfthSX2SzpP0L0m/iIgHZzu+0WhEq9WqfF4A6Ca2RyOiMddxK0qcLCJuLbEOAKA6bsUA\nQDKEHQCSIewAkAxhB4BkCDsAJEPYASAZwg4AyRB2AEiGsANAMoQdAJIh7ACQDGEHgGQIOwAkQ9gB\nIBnCDgDJEHYASIawA0AyhB0AkiHsAJAMYQeAZAg7ACRD2AEgGcIOAMkQdgBIhrADQDKEHQCSIewA\nkEyRsNvebPtN2wds31NiTQDA4lQOu+3lkn4t6TpJl0u61fblVdfNbmjPkHq39mrZfcvUu7VXQ3uG\n6h4JWHr6+iYeWJAVBdbYKOlARLwlSbYfkXSzpDcKrJ3S0J4hNZ9s6viJ45KksaNjaj7ZlCRtuWJL\nnaMBSKDErZgLJL1zyvND7dcwi/4d/f+P+qTjJ46rf0d/TRMBS8zkTn1kZOLBzn1BSoTdM7wWnznI\nbtpu2W6Nj48XOO3p6+DRgwt6HQAWosStmEOSLjzl+XpJhz99UEQMShqUpEaj8Znwd5MNazZo7OjY\njK8DkDQ8PPHn5C598jnmpcSO/WVJl9i+2PYZkm6R9ESBddMa2DSglT0rp722smelBjYN1DQRgEwq\n79gj4qTtOyQ9K2m5pIciYl/lyRKb/AFp/45+HTx6UBvWbNDApgF+cAp8Gjv1RXFE5++KNBqNaLVa\nHT8vAJzObI9GRGOu4/jkKQAkQ9gBIBnCDgDJEHYASIawA0AyhB0AkiHsAJAMYQeAZAg7ACRD2AEg\nGcIOAMkQdgBIhrADQDKEHQCSIewAkAxhB4BkCDsAJEPYASAZwg4AyRB2AEiGsANAMoQdAJIh7ACQ\nDGEHgGQIOwAkQ9gBIBnCDgDJEHYASKZS2G3/yPY+25/YbpQaCgCweFV37Hsl/VDSzgKzAAAKWFHl\nmyNivyTZLjMNAKCySmFfCNtNSc32049s7+3UuZe48yS9V/cQSwTXYgrXYgrXYsql8zlozrDb/quk\nL8/wV/0R8af5ThMRg5IG22u2IoJ78uJanIprMYVrMYVrMcV2az7HzRn2iLim+jgAgE7h7Y4AkEzV\ntzv+wPYhSd+S9LTtZ+f5rYNVzpsM12IK12IK12IK12LKvK6FI+KLHgQA0EHcigGAZAg7ACTT0bDb\n3mz7TdsHbN/TyXMvNbYfsn2k29/Pb/tC28/b3t/+9RR31T1TXWyfafsl26+1r8V9dc9UN9vLbb9q\n+6m6Z6mT7bdt77G9ez5veezYPXbbyyX9Q9J3JR2S9LKkWyPijY4MsMTY/rakY5J+HxFfrXueuthe\nJ2ldRLxie7WkUUnf78Z/LjzxEe5VEXHMdo+kXZLuiogXah6tNrZ/Jqkh6ZyIuLHueepi+21JjYiY\n1we1Orlj3yjpQES8FRH/lfSIpJs7eP4lJSJ2Snq/7jnqFhHvRsQr7a8/lLRf0gX1TlWPmHCs/bSn\n/ejadzfYXi/pBkkP1D3L6aaTYb9A0junPD+kLv0XGDOz3SvpSkkv1jtJfdq3HnZLOiJpe0R07bWQ\ntFXS3ZI+qXuQJSAk/cX2aPvXs3yuToZ9pt8U1rW7EUxn+2xJj0n6aUT8p+556hIRH0fE1yWtl7TR\ndlfeprN9o6QjETFa9yxLxFUR8Q1J10n6SftW7qw6GfZDki485fl6SYc7eH4sUe37yY9JGoqIx+ue\nZymIiA8kDUvaXPModblK0vfa95YfkXS17T/UO1J9IuJw+88jkrZp4tb2rDoZ9pclXWL7YttnSLpF\n0hMdPD+WoPYPDB+UtD8i7q97njrZXmv73PbXZ0m6RtLf652qHhFxb0Ssj4heTbTiuYi4reaxamF7\nVfuNBbK9StK1mvh/YcyqY2GPiJOS7pD0rCZ+QPZoROzr1PmXGtsPS/qbpEttH7L947pnqslVkm7X\nxI5sd/txfd1D1WSdpOdtv66JjdD2iOjqt/lBknS+pF22X5P0kqSnI+KZz/sGfqUAACTDJ08BIBnC\nDgDJEHYASIawA0AyhB0AkiHsAJAMYQeAZP4HFm8ToAVuLksAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2126f191f98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "x = np.array([[3,4],[1,1],[4,3],[1,0],[2,1],[4,2],[4,0]]) # 定义三个数据向量(8行_2列,数组)\n",
    "y = np.array([1,0,1,0,0,1,1]).T # 定义类别标签(8行_1列,向量)\n",
    "\n",
    "posindex = x[y[:] == 1] # 绘图用，标记正样本id\n",
    "negindex = x[y[:] == 0] # 绘图用，标记负样本id\n",
    "\n",
    "# 绘制散点图\n",
    "import matplotlib.pyplot as plt\n",
    "plt.scatter(posindex[:,0], posindex[:,1], c='red', alpha=1, marker='+', label='pickup') \n",
    "plt.scatter(negindex[:,0], negindex[:,1], c='green', alpha=1, marker='o', label='pickup') \n",
    "plt.axis([0,5,-1,5])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">此时我们的目标是求解参数$w$  \n",
    "(注意，这里由于需要得到$wx$的值，即$w$和$x$应该可以相乘，所以，这就隐式地要求$w$的参数为两个数，需要定义为一个包含两个参数值得参数向量：$w = (w_1,w_2)$)。但是这里我们这样设置也不合理，因为在上面我们已经将$wx+b$变相处理为了$wx$，所以这里统一对输入的X添加一个纵列，即$np.ones(len(X))$，即$wx = w_0x_0+w_1x_1+w_2x_2$，因为$x_0$都是1，所以可以理解为$wx = w_0+w_1x_1+w_2x_2$，这样$w_0$就相当于参数$b$了。\n",
    "\n",
    "### 6.4.2 定义自己的逻辑回归分类案发，编写损失计算函数、梯度上升求解w的函数、预测函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 计算损失\n",
    "def calLoss(w,X,Y):\n",
    "    return np.sum((np.exp(np.dot(w,X.T))/(np.exp(np.dot(w,X.T)) + 1) - Y)**2)/2\n",
    "\n",
    "# 预测，带入式子π(x)，得到y_，若y_大于0.5，则预测y为1，否则为0\n",
    "def predict(w,X):\n",
    "    result = []\n",
    "    X = np.column_stack([np.ones(len(X)),X])\n",
    "    y_ = np.exp(np.dot(w,X.T))/(np.exp(np.dot(w,X.T)) + 1.0)\n",
    "    y = y_\n",
    "    y[y>0.5] = 1  \n",
    "    y[y<=0.5] = 0\n",
    "    return y\n",
    "\n",
    "# 梯度上升算法求解最大化似然函数的时候的参数w\n",
    "def gradAscent(X,Y,eta,iter):\n",
    "    row, col = X.shape\n",
    "    X = np.column_stack([np.ones(len(x)),X])\n",
    "    w = np.ones((1,col + 1))\n",
    "    for i in range(iter):\n",
    "        ## 以下三步更新参数w\n",
    "        grad = np.dot((Y - np.exp(np.dot(w,X.T))/(1 + np.exp(np.dot(w,X.T)))),X) # step 1 \n",
    "        delta = eta * grad # step 2\n",
    "        w = w + delta # step 3\n",
    "# 取消以下注释可以不显示每一帧变化\n",
    "        if i%5 == 0:\n",
    "            print('==============================iteration'+ str(i)+'==============================')\n",
    "            print('weight:\\t\\t',w)\n",
    "            print('Loss:\\t\\t', calLoss(w,X,Y))\n",
    "            print('pred labels:\\t',predict(w,x))\n",
    "# 取消以下注释可以绘制每一帧变化以及保存图像           \n",
    "#             x_ = np.arange(-1,5,0.1)\n",
    "#             y_ = np.arange(-1,5,0.1)\n",
    "#             x_, y_ = np.meshgrid(x_, y_)   \n",
    "#             plt.figure(i)\n",
    "#             f = w[0,0] + w[0,1]*x_ + w[0,2]*y_\n",
    "#             plt.scatter(posindex[:,0], posindex[:,1], c='red', alpha=1, marker='+', label='pickup') \n",
    "#             plt.scatter(negindex[:,0], negindex[:,1], c='green', alpha=1, marker='o', label='pickup') \n",
    "#             plt.contour(x_, y_, f,0)\n",
    "#             plt.savefig(\"\".join([str(i),\".jpg\"]))\n",
    "#             plt.show()\n",
    "            \n",
    "# 取消以下注释可以“早停”\n",
    "#         if sum(abs(predict(w,x)[0].flatten() - y)) == 0:\n",
    "#             print('=============================end at iter:'+ str(i)+'=============================')\n",
    "#             print 'weight:\\t\\t',w\n",
    "#             print 'Loss:\\t\\t', calLoss(w,X,Y)\n",
    "#             print 'pred labels:\\t',predict(w,x)\n",
    "            \n",
    "#             x_ = np.arange(-1,5,0.1)\n",
    "#             y_ = np.arange(-1,5,0.1)\n",
    "#             x_, y_ = np.meshgrid(x_, y_)\n",
    "#             plt.figure(i)\n",
    "#             f = w[0,0] + w[0,1]*x_ + w[0,2]*y_\n",
    "#             plt.scatter(posindex[:,0], posindex[:,1], c='red', alpha=1, marker='+', label='pickup') \n",
    "#             plt.scatter(negindex[:,0], negindex[:,1], c='green', alpha=1, marker='o', label='pickup') \n",
    "#             plt.contour(x_, y_, f,0)\n",
    "#             plt.show()\n",
    "#             break # 如果满足所有的类别标签已经预测正确，则可以直接早停并返回参数w\n",
    "    return w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==============================iteration0==============================\n",
      "weight:\t\t [[0.83157338 0.77412186 0.8841749 ]]\n",
      "Loss:\t\t 1.2371932361860227\n",
      "pred labels:\t [[1. 1. 1. 1. 1. 1. 1.]]\n",
      "==============================iteration5==============================\n",
      "weight:\t\t [[0.19092714 0.1005751  0.47067337]]\n",
      "Loss:\t\t 0.7332237864212623\n",
      "pred labels:\t [[1. 1. 1. 1. 1. 1. 1.]]\n",
      "==============================iteration10==============================\n",
      "weight:\t\t [[-0.14660199  0.21201419  0.410074  ]]\n",
      "Loss:\t\t 0.6294973683904217\n",
      "pred labels:\t [[1. 1. 1. 1. 1. 1. 1.]]\n",
      "==============================iteration15==============================\n",
      "weight:\t\t [[-0.45365548  0.32725663  0.38175783]]\n",
      "Loss:\t\t 0.5460589069943628\n",
      "pred labels:\t [[1. 1. 1. 0. 1. 1. 1.]]\n",
      "==============================iteration20==============================\n",
      "weight:\t\t [[-0.7379793   0.42948221  0.36742371]]\n",
      "Loss:\t\t 0.4760452571202963\n",
      "pred labels:\t [[1. 1. 1. 0. 1. 1. 1.]]\n",
      "==============================iteration25==============================\n",
      "weight:\t\t [[-1.00117515  0.52129389  0.36166423]]\n",
      "Loss:\t\t 0.4175259023104088\n",
      "pred labels:\t [[1. 0. 1. 0. 1. 1. 1.]]\n",
      "==============================iteration30==============================\n",
      "weight:\t\t [[-1.24504627  0.60460239  0.36114593]]\n",
      "Loss:\t\t 0.3687044061969869\n",
      "pred labels:\t [[1. 0. 1. 0. 1. 1. 1.]]\n",
      "==============================iteration35==============================\n",
      "weight:\t\t [[-1.47143339  0.68082019  0.36382727]]\n",
      "Loss:\t\t 0.32794487570821285\n",
      "pred labels:\t [[1. 0. 1. 0. 1. 1. 1.]]\n",
      "==============================iteration40==============================\n",
      "weight:\t\t [[-1.68210559  0.75101819  0.3684553 ]]\n",
      "Loss:\t\t 0.29381420364059524\n",
      "pred labels:\t [[1. 0. 1. 0. 1. 1. 1.]]\n",
      "==============================iteration45==============================\n",
      "weight:\t\t [[-1.87870012  0.81603208  0.37425549]]\n",
      "Loss:\t\t 0.26509884949389056\n",
      "pred labels:\t [[1. 0. 1. 0. 1. 1. 1.]]\n",
      "==============================iteration50==============================\n",
      "weight:\t\t [[-2.06269681  0.87653135  0.38074443]]\n",
      "Loss:\t\t 0.24079620825920617\n",
      "pred labels:\t [[1. 0. 1. 0. 1. 1. 1.]]\n",
      "==============================iteration55==============================\n",
      "weight:\t\t [[-2.23541364  0.93306419  0.38761731]]\n",
      "Loss:\t\t 0.22009115937723775\n",
      "pred labels:\t [[1. 0. 1. 0. 1. 1. 1.]]\n",
      "==============================iteration60==============================\n",
      "weight:\t\t [[-2.39801426  0.98608715  0.39468013]]\n",
      "Loss:\t\t 0.20232721252943067\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration65==============================\n",
      "weight:\t\t [[-2.55152108  1.03598554  0.40180841]]\n",
      "Loss:\t\t 0.18697801516534718\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration70==============================\n",
      "weight:\t\t [[-2.69683056  1.08308786  0.40892202]]\n",
      "Loss:\t\t 0.17362198223215292\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration75==============================\n",
      "weight:\t\t [[-2.83472851  1.12767647  0.41596937]]\n",
      "Loss:\t\t 0.16192095674485835\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration80==============================\n",
      "weight:\t\t [[-2.96590437  1.1699957   0.42291758]]\n",
      "Loss:\t\t 0.15160284961914317\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration85==============================\n",
      "weight:\t\t [[-3.09096414  1.21025817  0.42974607]]\n",
      "Loss:\t\t 0.14244779345445155\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration90==============================\n",
      "weight:\t\t [[-3.21044171  1.24864981  0.43644255]]\n",
      "Loss:\t\t 0.13427722412058998\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration95==============================\n",
      "weight:\t\t [[-3.32480871  1.28533395  0.44300024]]\n",
      "Loss:\t\t 0.12694532219796162\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration100==============================\n",
      "weight:\t\t [[-3.43448298  1.32045465  0.44941614]]\n",
      "Loss:\t\t 0.1203323198168644\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration105==============================\n",
      "weight:\t\t [[-3.53983581  1.35413947  0.45568978]]\n",
      "Loss:\t\t 0.1143392655702795\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration110==============================\n",
      "weight:\t\t [[-3.64119816  1.38650175  0.46182244]]\n",
      "Loss:\t\t 0.10888392232172754\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration115==============================\n",
      "weight:\t\t [[-3.73886591  1.41764253  0.46781652]]\n",
      "Loss:\t\t 0.10389754305363426\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration120==============================\n",
      "weight:\t\t [[-3.83310441  1.44765223  0.4736752 ]]\n",
      "Loss:\t\t 0.0993223271422951\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration125==============================\n",
      "weight:\t\t [[-3.92415232  1.47661193  0.47940214]]\n",
      "Loss:\t\t 0.09510940471132606\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration130==============================\n",
      "weight:\t\t [[-4.01222492  1.50459465  0.48500128]]\n",
      "Loss:\t\t 0.09121723190608502\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration135==============================\n",
      "weight:\t\t [[-4.09751692  1.53166626  0.49047673]]\n",
      "Loss:\t\t 0.08761030701827191\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration140==============================\n",
      "weight:\t\t [[-4.18020493  1.5578864   0.49583265]]\n",
      "Loss:\t\t 0.08425813812615121\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration145==============================\n",
      "weight:\t\t [[-4.26044956  1.58330916  0.50107318]]\n",
      "Loss:\t\t 0.08113440875240806\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration150==============================\n",
      "weight:\t\t [[-4.33839721  1.60798377  0.5062024 ]]\n",
      "Loss:\t\t 0.07821630013230808\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration155==============================\n",
      "weight:\t\t [[-4.41418168  1.63195512  0.51122432]]\n",
      "Loss:\t\t 0.07548393792601034\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration160==============================\n",
      "weight:\t\t [[-4.48792551  1.65526423  0.51614282]]\n",
      "Loss:\t\t 0.0729199382873065\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration165==============================\n",
      "weight:\t\t [[-4.55974121  1.6779487   0.52096165]]\n",
      "Loss:\t\t 0.0705090336385019\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration170==============================\n",
      "weight:\t\t [[-4.62973228  1.70004303  0.52568443]]\n",
      "Loss:\t\t 0.06823776269221252\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration175==============================\n",
      "weight:\t\t [[-4.69799412  1.72157899  0.53031464]]\n",
      "Loss:\t\t 0.06609421250334774\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration180==============================\n",
      "weight:\t\t [[-4.76461487  1.74258584  0.53485562]]\n",
      "Loss:\t\t 0.06406780285320804\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration185==============================\n",
      "weight:\t\t [[-4.82967608  1.76309062  0.53931057]]\n",
      "Loss:\t\t 0.06214910523217002\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration190==============================\n",
      "weight:\t\t [[-4.89325338  1.78311836  0.54368255]]\n",
      "Loss:\t\t 0.060329690226287674\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "==============================iteration195==============================\n",
      "weight:\t\t [[-4.95541698  1.80269224  0.54797449]]\n",
      "Loss:\t\t 0.05860199832373426\n",
      "pred labels:\t [[1. 0. 1. 0. 0. 1. 1.]]\n",
      "参数 w =  [-5.00417406  1.8180391   0.5513523 ]\n",
      "决策函数为 f =  -5.004174063397151 + 1.818039096100689 * x1 +  0.551352302103199 * x2\n"
     ]
    }
   ],
   "source": [
    "w = gradAscent(x,y,0.06,200)\n",
    "r = predict(w,x)\n",
    "print('参数 w = ',w.flatten())\n",
    "print('决策函数为 f = ',w[0,0],'+', w[0,1],'* x1 + ', w[0,2],'* x2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<b>动图演示上述计算过程</b>\n",
    ">![动图演示](img/chap61.gif)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}