第00章 补充的数学知识.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 说明\n",
    "\n",
    "书中涉及到大量的数学分析和矩阵论的知识。在非数学专业本科学的内容不能完全覆盖书中需要的知识点。下面是我的补充。\n",
    "\n",
    "### 一、矩阵求导规则\n",
    "\n",
    "书上用到了很多向量和矩阵的求导，这里介绍一下：\n",
    "\n",
    "#### 1. 对向量求导\n",
    "\n",
    "$$\\begin{aligned}\n",
    "\\frac{\\partial \\mathbf{Ax}}{\\partial \\mathbf{x}} &= \\mathbf{A}^\\text{T}\\\\\n",
    "\\frac{\\partial \\mathbf{Ax}}{\\partial \\mathbf{x}^\\text{T}} &= \\mathbf{A}\\\\\n",
    "\\frac{\\partial \\mathbf{x}^\\text{T}\\mathbf{A}}{\\partial \\mathbf{x}} &= \\mathbf{A}\\\\\n",
    "\\frac{\\partial \\mathbf{x}^\\text{T}\\mathbf{A}\\mathbf{x}}{\\partial \\mathbf{x}} &= (\\mathbf{A}^\\text{T} + \\mathbf{A})\\mathbf{x} \\\\\n",
    "\\frac{\\partial \\mathbf{x\\cdot x}}{\\partial \\mathbf{x}} &= \\frac{\\mathbf{x}^\\text{T}\\mathbf{x}}{\\partial \\mathbf{x}} = 2\\mathbf{x}\\\\\n",
    "\\frac{\\partial \\mathbf{w\\cdot x}}{\\partial \\mathbf{x}} &= \\frac{\\partial \\mathbf{w}^\\text{T}\\mathbf{x}}{\\partial \\mathbf{x}} = \\mathbf{w}\n",
    "\\end{aligned}$$\n",
    "\n",
    "#### 2. 对矩阵求导\n",
    "\n",
    "对矩阵求导规则比较复杂，这篇文章写的很好：[【必读】3分钟带你了解标量对矩阵求导方法](https://zhuanlan.zhihu.com/p/279209775)。\n",
    "\n",
    "另外我再总结一下书中证明要用到的的公式（如果你按照上面文章的方法学会了对矩阵求导的通用方法，可以很容易自己证明出来）\n",
    "\n",
    "$$\\begin{aligned}\n",
    "\\\\d(\\mathbf{AX}) &= \\mathbf{A}d\\mathbf{X}\\\\\n",
    "\\frac{\\partial tr(\\mathbf{X}^\\text{T}\\mathbf{A}\\mathbf{X})}{\\partial \\mathbf{X}} &= (\\mathbf{A}^\\text{T} + \\mathbf{A})\\mathbf{X} \\\\\n",
    "\\frac{\\partial \\ln|\\mathbf{X}|}{\\partial \\mathbf{X}} &= \\mathbf{X}^{-1}\\\\\n",
    "\\frac{\\partial \\mathbf{w}^{\\text{T}}\\mathbf{X}^{-1}\\mathbf{w}}{\\partial\\mathbf{X}} &= -\\mathbf{X}^{-1}\\mathbf{w}\\mathbf{w}^{\\text{T}}\\mathbf{X}^{-1}，若\\mathbf{X}为对称阵\n",
    "\\end{aligned}$$\n",
    "\n",
    "### 二、多维随机变量的统计\n",
    "\n",
    "假设$\\mathbf{x}_1,\\mathbf{x}_2,\\cdots,\\mathbf{x}_N$是$d$维随机变量的样本，则\n",
    "\n",
    "#### 1. 样本均值\n",
    "\n",
    "$$\n",
    "\\mathbf{m} = \\frac{1}{N}\\sum_{i=1}^N\\mathbf{x}_i\n",
    "$$\n",
    "\n",
    "#### 2. 样本协方差矩阵（注意这个公式不求平均）\n",
    "\n",
    "$$\n",
    "\\mathbf{S}_{\\text{w}} = \\sum_{i=1}^N (\\mathbf{x}_i - \\mathbf{m})(\\mathbf{x}_i - \\mathbf{m})^{\\text{T}} = \\mathbf{X}^{\\text{T}}\\mathbf{X} - N\\mathbf{m}\\mathbf{m}^\\text{T}\n",
    "$$\n",
    "\n",
    "其中\n",
    "\n",
    "$$\n",
    "\\mathbf{X}^{\\text{T}}\\mathbf{X}=\n",
    "\\left[\\begin{matrix} \n",
    "\\mathbf{x}_1,\\mathbf{x}_2,\\cdots,\\mathbf{x}_N\n",
    "\\end{matrix}\\right]\n",
    "\\left[\\begin{matrix}\n",
    "\\mathbf{x}_1^{\\text{T}} \\\\\n",
    "\\mathbf{x}_2^{\\text{T}} \\\\\n",
    "\\vdots \\\\\n",
    "\\mathbf{x}_N^{\\text{T}}\n",
    "\\end{matrix}\\right]\n",
    "$$\n",
    "\n",
    "#### 3. 样本二阶矩阵\n",
    "\n",
    "$$\n",
    "\\sum_{i=1}^N \\mathbf{x}_i\\mathbf{x}_i^{\\text{T}} = \\mathbf{X}^{\\text{T}}\\mathbf{X}\n",
    "$$\n",
    "\n",
    "### 三、超空间几何\n",
    "\n",
    "#### 1. 点到超平面距离\n",
    "\n",
    "假设在$n$维空间中，有超平面$\\mathbf{w}\\cdot \\mathbf{x} + b = 0$，有一点$\\mathbf{x}_0$。\n",
    "\n",
    "则点到超平面的距离为：\n",
    "\n",
    "$$d = \\frac{|\\mathbf{w} \\cdot \\mathbf{x}_0 + b|}{||\\mathbf{w}||}$$\n",
    "\n",
    "#### 2. 平行的超平面的距离\n",
    "\n",
    "假设在$n$维空间中，有两个平行的超平面\n",
    "\n",
    "$$\\mathbf{w}\\cdot \\mathbf{x} + b_1 = 0\\\\\n",
    " \\mathbf{w}\\cdot \\mathbf{x} + b_2 = 0$$\n",
    "\n",
    "因为平行的超平面的权向量（$\\mathbf{w}$向量）一定成比例，所以上面的公式把两个超平面的权向量化成一样的了，这时就只有$b$不同。\n",
    "\n",
    "则两个超平面之间的距离为：\n",
    "\n",
    "$$d = \\frac{|b_1-b_2|}{||\\mathbf{w}||}$$\n",
    "\n",
    "### 四、进阶版拉格朗日算子法（求多元函数条件极值）\n",
    "\n",
    "#### 1. 自变量是向量的多元函数\n",
    "\n",
    "高数中学的拉格朗日算子法的条件只能是等式，下面给出条件有不等式的求法：\n",
    "\n",
    "假设在约束条件$g(\\mathbf{x})\\leqslant 0$的情况下，求多元函数$f(\\mathbf{x})$的极值。\n",
    "\n",
    "1. 写出拉格朗日函数\n",
    "\n",
    "$$\n",
    "\\mathscr{L}(\\mathbf{x}) = f(\\mathbf{x}) + \\lambda g(\\mathbf{x})\n",
    "$$\n",
    "\n",
    "2. 写出条件不等式（KKT条件）\n",
    "\n",
    "$$\\begin{align}\n",
    "\\frac{\\partial\\mathscr{L}}{\\partial\\mathbf{x}} = \\frac{\\partial f}{\\partial\\mathbf{x}} + \\lambda\\frac{\\partial g}{\\partial\\mathbf{x}} &= 0 \\\\\n",
    "\\frac{\\partial\\mathscr{L}}{\\partial \\lambda} = g(\\mathbf{x}) &\\leqslant 0 \\\\\n",
    "\\lambda &\\geqslant 0 \\\\\n",
    "\\lambda g(\\mathbf{x}) &= 0\n",
    "\\end{align}$$\n",
    "\n",
    "这组不等式也叫KKT条件\n",
    "\n",
    "3. 求解KKT条件\n",
    "\n",
    "如果有多个约束条件也是同理，把KKT条件扩展一下就行。\n",
    "\n",
    "#### 2. 自变量是矩阵的函数\n",
    "\n",
    "自变量是矩阵可能不好理解，但是教材162页的公式用到了这个方法。\n",
    "\n",
    "自变量是矩阵时就把上面的公式中的常数$\\lambda$换成对角阵$\\boldsymbol{\\Lambda}$，自变量向量$\\mathbf{x}$换成矩阵$\\mathbf{X}$即可，其他都不变。这里就不再赘述了。\n",
    "\n",
    "### 五、欧拉积分\n",
    "\n",
    "贝叶斯估计可能会用到欧拉积分。\n",
    "\n",
    "#### 1. 伽马函数\n",
    "\n",
    "$$\\begin{aligned}\n",
    "\\Gamma(s) &= \\int_0^{+\\infty }x^{s}e^{-x} \\text{d}x \\\\\n",
    "\\Gamma(s) &= s\\Gamma(s-1) \\\\\n",
    "\\Gamma(s) &= s! \\quad若s为正整数\n",
    "\\end{aligned}$$\n",
    "\n",
    "#### 2. 贝塔函数\n",
    "\n",
    "$$\\begin{aligned}\n",
    "\\beta(x,y) &= \\int_0^1P^{x}(1-P)^{y}\\text{d}P \\\\\n",
    "\\beta(x,y) &= \\frac{x!y!}{(x+y+1)!} \\quad若x,y为正整数\n",
    "\\end{aligned}$$"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}