1. Return States and Sequences.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "recorded-characteristic",
   "metadata": {},
   "source": [
    "Each LSTM cell will output one hidden state h for each input.\n",
    "We can demonstrate this in Keras with a very small model with a single LSTM layer that itself contains a single LSTM cell.\n",
    "\n",
    "In this example, we will have one input sample with 3 time steps and one feature observed at each time step:\n",
    "# Return Sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "every-catch",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define model\n",
    "inputs1 = Input(shape=(3, 1))\n",
    "lstm1 = LSTM(1)(inputs1)\n",
    "model = Model(inputs=inputs1, outputs=lstm1)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "sublime-wrestling",
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Model\n",
    "from keras.layers import Input\n",
    "from keras.layers import LSTM\n",
    "from numpy import array\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "indie-rough",
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs1 = Input(shape=(3, 1))\n",
    "lstm1 = LSTM(1)(inputs1)\n",
    "model = Model(inputs=inputs1, outputs=lstm1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "indian-center",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-0.03685613]]\n"
     ]
    }
   ],
   "source": [
    "# define input data\n",
    "data = array([0.1, 0.2, 0.3]).reshape((1,3,1))\n",
    "# make and show prediction\n",
    "print(model.predict(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "induced-lewis",
   "metadata": {},
   "source": [
    "It is possible to access the hidden state output for each input time step.\n",
    "\n",
    "This can be done by setting the return_sequences attribute to True when defining the LSTM layer, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "asian-arcade",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[-0.00273301]\n",
      "  [-0.00730167]\n",
      "  [-0.01297203]]]\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Model\n",
    "from keras.layers import Input\n",
    "from keras.layers import LSTM\n",
    "from numpy import array\n",
    "# define model\n",
    "inputs1 = Input(shape=(3, 1))\n",
    "lstm1 = LSTM(1, return_sequences=True)(inputs1)\n",
    "model = Model(inputs=inputs1, outputs=lstm1)\n",
    "# define input data\n",
    "data = array([0.1, 0.2, 0.3]).reshape((1,3,1))\n",
    "# make and show prediction\n",
    "print(model.predict(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "alive-uruguay",
   "metadata": {},
   "source": [
    "Running the example returns a sequence of 3 values, one hidden state output for each input time step for the single LSTM cell in the layer."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "polish-manhattan",
   "metadata": {},
   "source": [
    "You must set return_sequences=True when stacking LSTM layers so that the second LSTM layer has a three-dimensional sequence input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "historical-questionnaire",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "sticky-buffalo",
   "metadata": {},
   "source": [
    "# Return States"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "outer-tiger",
   "metadata": {},
   "source": [
    "The output of an LSTM cell or layer of cells is called the hidden state.\n",
    "\n",
    "This is confusing, because each LSTM cell retains an internal state that is not output, called the cell state, or c.\n",
    "\n",
    "Generally, we do not need to access the cell state unless we are developing sophisticated models where subsequent layers may need to have their cell state initialized with the final cell state of another layer, such as in an encoder-decoder model.\n",
    "\n",
    "Keras provides the return_state argument to the LSTM layer that will provide access to the hidden state output (state_h) and the cell state (state_c). For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "consolidated-senate",
   "metadata": {},
   "outputs": [],
   "source": [
    "# lstm1, state_h, state_c = LSTM(1, return_state=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "graphic-license",
   "metadata": {},
   "source": [
    "This may look confusing because both lstm1 and state_h refer to the same hidden state output. The reason for these two tensors being separate will become clear in the next section.\n",
    "\n",
    "We can demonstrate access to the hidden and cell states of the cells in the LSTM layer with a worked example listed below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "intermediate-worcester",
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Model\n",
    "from keras.layers import Input\n",
    "from keras.layers import LSTM\n",
    "from numpy import array\n",
    "# define model\n",
    "inputs1 = Input(shape=(3, 1))\n",
    "lstm1, state_h, state_c = LSTM(1, return_state=True)(inputs1)\n",
    "model = Model(inputs=inputs1, outputs=[lstm1, state_h, state_c])\n",
    "# define input data\n",
    "data = array([0.1, 0.2, 0.3]).reshape((1,3,1))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "olympic-devices",
   "metadata": {},
   "source": [
    "Running the example returns 3 arrays:\n",
    "\n",
    "The LSTM hidden state output for the last time step.\n",
    "\n",
    "The LSTM hidden state output for the last time step (again).\n",
    "\n",
    "The LSTM cell state for the last time step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "periodic-privilege",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[-0.00273301]\n",
      "  [-0.00730167]\n",
      "  [-0.01297203]]]\n"
     ]
    }
   ],
   "source": [
    "# make and show prediction\n",
    "print(model.predict(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "domestic-lunch",
   "metadata": {},
   "source": [
    "The hidden state and the cell state could in turn be used to initialize the states of another LSTM layer with the same number of cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "tribal-disposal",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "varying-convention",
   "metadata": {},
   "source": [
    "# Return States and Sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "north-exclusion",
   "metadata": {},
   "source": [
    "We can access both the sequence of hidden state and the cell states at the same time.\n",
    "\n",
    "This can be done by configuring the LSTM layer to both return sequences and return states.\n",
    "\n",
    "lstm1, state_h, state_c = LSTM(1, return_sequences=True, return_state=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "compatible-denmark",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[array([[[-0.01833471],\n",
      "        [-0.05499296],\n",
      "        [-0.10949519]]], dtype=float32), array([[-0.10949519]], dtype=float32), array([[-0.21264875]], dtype=float32)]\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Model\n",
    "from keras.layers import Input\n",
    "from keras.layers import LSTM\n",
    "from numpy import array\n",
    "# define model\n",
    "inputs1 = Input(shape=(3, 1))\n",
    "lstm1, state_h, state_c = LSTM(1, return_sequences=True, return_state=True)(inputs1)\n",
    "model = Model(inputs=inputs1, outputs=[lstm1, state_h, state_c])\n",
    "# define input data\n",
    "data = array([0.1, 0.2, 0.3]).reshape((1,3,1))\n",
    "# make and show prediction\n",
    "print(model.predict(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "formed-requirement",
   "metadata": {},
   "source": [
    "Specifically, we learned:\n",
    "\n",
    "That return sequences return the hidden state output for each input time step.\n",
    "\n",
    "That return state returns the hidden state output and cell state for the last input time step.\n",
    "\n",
    "That return sequences and return state can be used at the same time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "agreed-logic",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}