第6.1章-Logistic回归.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第6.1章 Logistic回归"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Logistic Regression，一种经典的__分类__方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 一. Logistic分布"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "设$X$是连续随机变量，$X$服从Logistic分布是指，$X$具有以下的分布函数：\n",
    "\n",
    "$$F(x) = P(X \\le x) = \\frac{1}{1+e^{-(x-\\mu)/\\gamma}}$$\n",
    "\n",
    "它的概率密度函数为：\n",
    "\n",
    "$$f(x) = F'(x) = \\frac{e^{-(x-\\mu)/\\gamma}}{\\gamma(1+e^{-(x-\\mu)/\\gamma})^2}$$\n",
    "\n",
    "式中，$\\mu$为位置参数，$\\gamma>0$为形状参数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$F(x)$是一条S形曲线，被称为sigmoid curve。该曲线以$(\\mu,\\frac{1}{2})$中心对称，且在中心附近增长最快，在两段增长较慢——$\\gamma$的值越小，曲线在中心附近增长得越快。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![logistic分布](./pic/ch6-1-1.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二. 二元分类中的Logistic回归模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "输入$x \\in \\mathbf{R}^{n+1}$(加入第0维$x^{(0)}=1$)，输出$y \\in \\mathcal{Y} = \\{1,0\\}$，一般1代表正例，0代表负例。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Logistic回归模型用条件概率分布$P(Y|X)$表示，具体的：\n",
    "\n",
    "$$P(Y=1|x) = \\frac{1}{1+e^{-w·x}}$$\n",
    "\n",
    "$$P(Y=0|x) = 1-P(Y=1|x) = \\frac{e^{-w·x}}{1+e^{-w·x}}$$\n",
    "\n",
    "式中$w \\in \\mathbf{R}^{n+1}$，是需要学习的模型参数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "预测时，模型将根据输入$x$，计算$P(Y=1|x)$和$P(Y=0|x)$，哪个概率大，则把输入分到相应的一类中去。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Why: Intuition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__理解一__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从线性模型(线性回归)的思想出发，我们希望为每一个输入特征赋一个权重(当然还有一个bias)，算加权和，从而得到最终的结果。然而，$w·x$可能从负无穷到正无穷，而我们想要的是概率(条件概率也是概率)，取值范围是$[0,1]$。因此，我们自然会想到将$w·x$通过一个函数，转化到$[0,1]$的区间上。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这样的函数可能是“单位跃阶函数”：\n",
    "\n",
    "\\begin{equation}\n",
    "f(w·x)=\n",
    "    \\begin{cases}\n",
    "        1& \\text{if w·x > 0}\\\\\n",
    "        \\frac{1}{2}& \\text{if w·x = 0}\\\\\n",
    "        0& \\text{if w·x < 0}\\\\\n",
    "    \\end{cases}\n",
    "\\end{equation}\n",
    "\n",
    "但单位跃阶函数在$w·x=0$处不连续，因此不易优化。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因此，我们选择logistic函数(logistic function)：\n",
    "\n",
    "$$f(w·x) = \\frac{1}{1+e^{-w·x}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![单位跃阶函数与logistic函数](./pic/ch6-1-2.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__理解二__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们假设：__输出为$Y=1$的对数几率（log odds）为输出$x$的线性函数__。从这个假设，我们能推出logistic模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__几率(odds)__：若某件事发生的概率为$p$，则该事件发生的几率(odd)为$\\frac{p}{1-p}$，对数几率为$log\\frac{p}{1-p}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由上面的假设，有：\n",
    "\n",
    "$$log\\frac{P(Y=1|x)}{1-P(Y=1|x)}=w·x$$\n",
    "\n",
    "化简得：\n",
    "\n",
    "$$P(Y=1|x) = \\frac{1}{1+e^{-w·x}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后，logistic回归模型有一个特别的好处，就是能够顺便输出预测的__概率__~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. How: 模型参数的估计方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一般采用__最大似然估计(MLE)__。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "给定训练集$T=\\{(x_1,y_1),(x_2,y_2),\\cdots,(x_N,y_N)\\}$，$x_i \\in \\mathbf{R}^{n+1}$(加入第0维$x^{(0)}=1$)，输出$y_i \\in \\mathcal{Y} = \\{1,0\\}$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "设：\n",
    "\n",
    "$$P(Y=1|x)=h(x)=\\frac{1}{1+e^{-w·x}}$$\n",
    "\n",
    "则：\n",
    "\n",
    "$$P(Y=0|x)=1-h(x)=\\frac{e^{-w·x}}{1+e^{-w·x}}=\\frac{1}{1+e^{w·x}}=h(-x)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因此，对于任一对$(x,y)$，有：\n",
    "\n",
    "$$P(y|x)=[h(x)]^y[1-h(x)]^{1-y}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "似然函数为：\n",
    "\n",
    "$$l(w) = \\prod_{i=1}^N P(y_i|x_i) = \\prod_{i=1}^N [h(x_i)]^{y_i}[1-h(x_i)]^{1-y_i}$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对数似然函数为：\n",
    "\n",
    "$$L(w) = \\sum_{i=1}^N \\Big[y_i·log\\ h(x_i)+(1-y_i)·log(1-h(x_i))\\Big]$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "之后再采用__梯度下降法__或__拟牛顿法__解$L(w)$的最优化问题即可。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 三. 多元分类中的Logistic回归模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "假设输出$Y$的取值集合$\\mathcal{Y}=\\{1,2,\\cdots,K\\}$，则模型如下：\n",
    "\n",
    "$$P(Y=m\\ |\\ x)=\\frac{e^{w_m·x}}{1+\\sum_{k=1}^{K-1} e^{w_k·x}}$$\n",
    "\n",
    "其中，$m=1,2,\\cdots,K-1$，因此需要学习$K-1$个参数。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}