PyTorch_rnn_LSTM_GRU.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTorch 中的循环神经网络模块\n",
    "前面我们讲了循环神经网络的基础知识和网络结构，下面我们教大家如何在 pytorch 下构建循环神经网络，因为 pytorch 的动态图机制，使得循环神经网络非常方便。\n",
    "\n",
    "## 一般的 RNN\n",
    "\n",
    "![](https://ws1.sinaimg.cn/large/006tKfTcly1fmt9xz889xj30kb07nglo.jpg)\n",
    "\n",
    "对于最简单的 RNN，我们可以使用下面两种方式去调用，分别是 `torch.nn.RNNCell()` 和 `torch.nn.RNN()`，这两种方式的区别在于 `RNNCell()` 只能接受序列中单步的输入，且必须传入隐藏状态，而 `RNN()` 可以接受一个序列的输入，默认会传入全 0 的隐藏状态，也可以自己申明隐藏状态传入。\n",
    "\n",
    "`RNN()` 里面的参数有\n",
    "\n",
    "input_size 表示输入 $x_t$ 的特征维度\n",
    "\n",
    "hidden_size 表示输出的特征维度\n",
    "\n",
    "num_layers 表示网络的层数\n",
    "\n",
    "nonlinearity 表示选用的非线性激活函数，默认是 'tanh'\n",
    "\n",
    "bias 表示是否使用偏置，默认使用\n",
    "\n",
    "batch_first 表示输入数据的形式，默认是 False，就是这样形式，(seq, batch, feature)，也就是将序列长度放在第一位，batch 放在第二位\n",
    "\n",
    "dropout 表示是否在输出层应用 dropout\n",
    "\n",
    "bidirectional 表示是否使用双向的 rnn，默认是 False\n",
    "\n",
    "对于 `RNNCell()`，里面的参数就少很多，只有 input_size，hidden_size，bias 以及 nonlinearity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.autograd import Variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定义一个单步的rnn\n",
    "rnn_single = nn.RNNCell(input_size=100, hidden_size=200)   # RNNCell()里面参数只有 input_size, hidden_size, bias, nonlinearity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[ 0.0372,  0.0532,  0.0556,  ...,  0.0182, -0.0300, -0.0476],\n",
       "        [ 0.0641, -0.0329,  0.0089,  ..., -0.0439,  0.0679,  0.0170],\n",
       "        [-0.0379, -0.0687,  0.0458,  ..., -0.0027,  0.0582,  0.0053],\n",
       "        ...,\n",
       "        [ 0.0498, -0.0198,  0.0464,  ...,  0.0130,  0.0651, -0.0542],\n",
       "        [ 0.0397,  0.0049, -0.0477,  ...,  0.0263,  0.0463,  0.0191],\n",
       "        [-0.0321, -0.0107,  0.0350,  ...,  0.0075, -0.0290,  0.0395]],\n",
       "       requires_grad=True)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 访问其中的参数\n",
    "rnn_single.weight_hh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 构造一序列，长为6， batch为5， 特征是100\n",
    "x = Variable(torch.randn(6, 5, 100))     # 这是rnn的输入格式！记住就好\n",
    "# 定义初始的记忆状态\n",
    "h_t = Variable(torch.zeros(5, 200))\n",
    "\n",
    "# 传入 rnn\n",
    "out = []\n",
    "for i in range(6):\n",
    "    h_t = rnn_single(x[i], h_t)\n",
    "    out.append(h_t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([5, 200])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out[0].shape # 每个输出的维度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到经过了 rnn 之后，隐藏状态的值已经被改变了，因为网络记忆了序列中的信息，同时输出 6 个结果\n",
    "\n",
    "下面我们看看直接使用 `RNN`的情况"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "rnn_seq = nn.RNN(100, 200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 访问其中的参数\n",
    "# rnn_seq.weight_hh_l0\n",
    "\n",
    "out, h_t = rnn_seq(x) # 使用默认的全 0 隐藏状态"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# h_t\n",
    "len(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 自己定义初始的隐藏状态\n",
    "h_0 = Variable(torch.randn(1, 5, 200))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([6, 5, 200])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 这里的隐藏状态的大小有三个维度，分别是 (num_layers * num_direction, batch, hidden_size)\n",
    "out, h_t = rnn_seq(x, h_0)\n",
    "out.shape                      # 输出为 （seq, batch, feature)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一般情况下我们都是用 nn.RNN() 而不是 nn.RNNCell()，因为 nn.RNN() 能够避免我们手动写循环，非常方便，同时如果不特别说明，我们也会选择使用默认的全 0 初始化隐藏状态\n",
    "\n",
    "\n",
    "## LSTM\n",
    "\n",
    "![](https://ws1.sinaimg.cn/large/006tKfTcly1fmt9qj3uhmj30iz07ct90.jpg)\n",
    "\n",
    "LSTM 和基本的 RNN 是一样的，他的参数也是相同的，同时他也有 `nn.LSTMCell()` 和 `nn.LSTM()` 两种形式，跟前面讲的都是相同的，我们就不再赘述了，下面直接举个小例子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[ 0.0548, -0.0224,  0.0546,  ..., -0.0194,  0.0035,  0.0039],\n",
       "        [ 0.0983,  0.0367,  0.0212,  ...,  0.0353,  0.0811, -0.0880],\n",
       "        [-0.0187,  0.0015, -0.0970,  ..., -0.0562,  0.0408, -0.0796],\n",
       "        ...,\n",
       "        [-0.0719, -0.0961,  0.0613,  ..., -0.0242,  0.0252, -0.0661],\n",
       "        [-0.0091, -0.0892,  0.0817,  ...,  0.0388, -0.0996, -0.0159],\n",
       "        [ 0.0732,  0.0468,  0.0928,  ...,  0.0093, -0.0677, -0.0431]],\n",
       "       requires_grad=True)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lstm_seq = nn.LSTM(50, 100, num_layers=2)   # 输入维度100，输出维度200， 两层\n",
    "# lstm_seq.weight_hh_10.shape\n",
    "lstm_seq.weight_hh_l0 # 第一层的 h_t 权重"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([400, 100])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lstm_seq.weight_hh_l0.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "lstm_input = Variable(torch.randn(10, 3, 50)) # 序列 10，batch 是 3，输入维度 50\n",
    "out, (h, c) = lstm_seq(lstm_input) # 使用默认的全 0 隐藏状态"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意这里 LSTM 输出的隐藏状态有两个，h 和 c，就是上图中的每个 cell 之间的两个箭头，这两个隐藏状态的大小都是相同的，(num_layers * direction, batch, feature)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 我们可以不使用默认的隐藏状态，这是需要传入两个张量\n",
    "h_init = Variable(torch.randn(2, 3, 100))\n",
    "c_init = Variable(torch.randn(2, 3, 100))\n",
    "\n",
    "out, (h, c) = lstm_seq(lstm_input, (h_init, c_init))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "h.shape torch.Size([2, 3, 100]) \t c.shape torch.Size([2, 3, 100]) \t out.shape torch.Size([10, 3, 100])\n"
     ]
    }
   ],
   "source": [
    "print('h.shape',h.shape,'\\t c.shape',c.shape,'\\t out.shape',out.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GRU\n",
    "![](https://ws3.sinaimg.cn/large/006tKfTcly1fmtaj38y9sj30io06bmxc.jpg)\n",
    "\n",
    "GRU 和前面讲的这两个是同样的道理，就不再细说，还是演示一下例子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "gru_seq = nn.GRU(10, 20)\n",
    "gru_input = Variable(torch.randn(3, 32, 10))\n",
    "\n",
    "out, h = gru_seq(gru_input)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[-0.1367, -0.1534,  0.1861,  ..., -0.0772,  0.1176,  0.0775],\n",
       "        [ 0.1510, -0.0243,  0.1811,  ...,  0.0915,  0.1027,  0.0209],\n",
       "        [ 0.1624,  0.1065,  0.1454,  ...,  0.0628, -0.1073, -0.1305],\n",
       "        ...,\n",
       "        [ 0.1948, -0.0937, -0.1434,  ...,  0.2168,  0.0837,  0.1831],\n",
       "        [-0.0576,  0.0960, -0.1009,  ...,  0.1947, -0.0544, -0.1055],\n",
       "        [-0.1230,  0.0755,  0.1718,  ..., -0.1473, -0.0738,  0.1418]],\n",
       "       requires_grad=True)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gru_seq.weight_hh_l0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "h.shape torch.Size([1, 32, 20]) \t out.shape torch.Size([3, 32, 20])\n"
     ]
    }
   ],
   "source": [
    "print('h.shape',h.shape, '\\t out.shape',out.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}