dlnd_language_translationv2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Language Translation\n",
    "In this project, I am going to build language translation model called `seq2seq model or encoder-decoder model` in TensorFlow. The objective of the model is translating English sentences to French sentences. I am going to show the detailed steps, and they will answer to the questions like `how to preprocess the dataset`, `how to define inputs`, `how to define encoder model`, `how to define decoder model`, `how to build the entire seq2seq model`, `how to calculate the loss and clip gradients`, and `how to train and get prediction`.\n",
    "\n",
    "This is a part of Udacity's Deep Learning Nanodegree. Some codes/functions (save, load, measuring accuracy, etc) are provided by Udacity. However, majority part is implemented by myself along with much richer explanations and references on each section.  \n",
    "\n",
    "## Get the Data\n",
    "While I am running this project on my labtop computer, I cannot handle a huge dataset. Rather I am going to use the reduced size of the original dataset ([WMT10 French-English corpus](http://www.statmt.org/wmt10/training-giga-fren.tar)). This version of data is provided by Udacity. If you have GPU machines, feel free to run the codes. There is no other configuration needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pickle\n",
    "import copy\n",
    "import numpy as np\n",
    "\n",
    "def load_data(path):\n",
    "    input_file = os.path.join(path)\n",
    "    with open(input_file, 'r', encoding='utf-8') as f:\n",
    "        data = f.read()\n",
    "\n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "source_path = 'data/small_vocab_en'\n",
    "target_path = 'data/small_vocab_fr'\n",
    "source_text = load_data(source_path)\n",
    "target_text = load_data(target_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the Data\n",
    "\n",
    "The two datasets store bunch of sentences in different language, and that is something we don't have to explore for now. You probably already know how your data looks like when you decided to download this one. **However**, it is worthwhile to explore how complex the datasets are. The complexity could suggest how we should approach to get the right result still considering some of restrictions. \n",
    "\n",
    "`note: ` The two files exactly contains the same number of lines. Each i-th line in both files has the same meaning but expressed in different languages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset Brief Stats\n",
      "* number of unique words in English sample sentences: 227        [this is roughly measured/without any preprocessing]\n",
      "\n",
      "* English sentences\n",
      "\t- number of sentences: 137861\n",
      "\t- avg. number of words in a sentence: 13.225277634719028\n",
      "* French sentences\n",
      "\t- number of sentences: 137861 [data integrity check / should have the same number]\n",
      "\t- avg. number of words in a sentence: 14.226612312401622\n",
      "\n",
      "* Sample sentences range from 0 to 5\n",
      "[1-th] sentence\n",
      "\tEN: new jersey is sometimes quiet during autumn , and it is snowy in april .\n",
      "\tFR: new jersey est parfois calme pendant l' automne , et il est neigeux en avril .\n",
      "\n",
      "[2-th] sentence\n",
      "\tEN: the united states is usually chilly during july , and it is usually freezing in november .\n",
      "\tFR: les états-unis est généralement froid en juillet , et il gèle habituellement en novembre .\n",
      "\n",
      "[3-th] sentence\n",
      "\tEN: california is usually quiet during march , and it is usually hot in june .\n",
      "\tFR: california est généralement calme en mars , et il est généralement chaud en juin .\n",
      "\n",
      "[4-th] sentence\n",
      "\tEN: the united states is sometimes mild during june , and it is cold in september .\n",
      "\tFR: les états-unis est parfois légère en juin , et il fait froid en septembre .\n",
      "\n",
      "[5-th] sentence\n",
      "\tEN: your least liked fruit is the grape , but my least liked is the apple .\n",
      "\tFR: votre moins aimé fruit est le raisin , mais mon moins aimé est la pomme .\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from collections import Counter\n",
    "\n",
    "print('Dataset Brief Stats')\n",
    "print('* number of unique words in English sample sentences: {}\\\n",
    "        [this is roughly measured/without any preprocessing]'.format(len(Counter(source_text.split()))))\n",
    "print()\n",
    "\n",
    "english_sentences = source_text.split('\\n')\n",
    "print('* English sentences')\n",
    "print('\\t- number of sentences: {}'.format(len(english_sentences)))\n",
    "print('\\t- avg. number of words in a sentence: {}'.format(np.average([len(sentence.split()) for sentence in english_sentences])))\n",
    "\n",
    "french_sentences = target_text.split('\\n')\n",
    "print('* French sentences')\n",
    "print('\\t- number of sentences: {} [data integrity check / should have the same number]'.format(len(french_sentences)))\n",
    "print('\\t- avg. number of words in a sentence: {}'.format(np.average([len(sentence.split()) for sentence in french_sentences])))\n",
    "print()\n",
    "\n",
    "sample_sentence_range = (0, 5)\n",
    "side_by_side_sentences = list(zip(english_sentences, french_sentences))[sample_sentence_range[0]:sample_sentence_range[1]]\n",
    "print('* Sample sentences range from {} to {}'.format(sample_sentence_range[0], sample_sentence_range[1]))\n",
    "\n",
    "for index, sentence in enumerate(side_by_side_sentences):\n",
    "    en_sent, fr_sent = sentence\n",
    "    print('[{}-th] sentence'.format(index+1))\n",
    "    print('\\tEN: {}'.format(en_sent))\n",
    "    print('\\tFR: {}'.format(fr_sent))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing\n",
    "\n",
    "Here are brief overview what steps will be done in this section\n",
    "\n",
    "- **create lookup tables** \n",
    "  - create two mapping tables \n",
    "      - (key, value) == (unique word string, its unique index)     - `(1)`\n",
    "      - (key, value) == (its unique index, unique word string)     - `(2)`\n",
    "      - `(1)` is used in the next step, and (2) is used later for prediction step\n",
    "      \n",
    "      \n",
    "- **text to word ids**\n",
    "  - convert each string word in the list of sentences to the index\n",
    "  - `(1)` is used for converting process\n",
    "  \n",
    "  \n",
    "- **save the pre-processed data**\n",
    "  - create two `(1)` mapping tables for English and French\n",
    "  - using the mapping tables, replace strings in the original source and target dataset with indicies\n",
    "\n",
    "### Create Lookup Tables\n",
    "\n",
    "As mentioned breifly, I am going to implement a function to create lookup tables. Since every models are mathmatically represented, the input and the output(prediction) should also be represented as numbers. That is why this step is necessary for NLP problem because human readable text is not machine readable. This function takes a list of sentences and returns two mapping tables (dictionary data type). Along with the list of sentences, there are special tokens, `<PAD>`, `<EOS>`, `<UNK>`, and `<GO>` to be added in the mapping tables. \n",
    "\n",
    "- (key, value) == (unique word string, its unique index)     - `(1)`\n",
    "- (key, value) == (its unique index, unique word string)     - `(2)`\n",
    "\n",
    "`(1)` will be used in the next step, `test to word ids`, to find a match between word and its index. `(2)` is not used in pre-processing step, but `(2)` will be used later. After making a prediction, the sequences of words in the output sentence will be represented as their indicies. The predicted output is machine readable but not human readable. That is why we need `(2)` to convert each indicies of words back into human readable words in string.\n",
    "\n",
    "<br/>\n",
    "<img src='./lookup.png' alt='Drawing' width='70%'>\n",
    "\n",
    "#### References\n",
    "- [Why special tokens?](https://datascience.stackexchange.com/questions/26947/why-do-we-need-to-add-start-s-end-s-symbols-when-using-recurrent-neural-n)\n",
    "- [Python `enumerate`](https://docs.python.org/3/library/functions.html#enumerate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "CODES = {'<PAD>': 0, '<EOS>': 1, '<UNK>': 2, '<GO>': 3 }\n",
    "\n",
    "def create_lookup_tables(text):\n",
    "    # make a list of unique words\n",
    "    vocab = set(text.split())\n",
    "\n",
    "    # (1)\n",
    "    # starts with the special tokens\n",
    "    vocab_to_int = copy.copy(CODES)\n",
    "\n",
    "    # the index (v_i) will starts from 4 (the 2nd arg in enumerate() specifies the starting index)\n",
    "    # since vocab_to_int already contains special tokens\n",
    "    for v_i, v in enumerate(vocab, len(CODES)):\n",
    "        vocab_to_int[v] = v_i\n",
    "\n",
    "    # (2)\n",
    "    int_to_vocab = {v_i: v for v, v_i in vocab_to_int.items()}\n",
    "\n",
    "    return vocab_to_int, int_to_vocab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text to Word Ids\n",
    "\n",
    "Two `(1)` lookup tables will be provided in `text_to_ids` functions as arguments. They will be used in the converting process for English(source) and French(target) respectively. This part is more like a programming part, so there are not much to mention. I will just go over few minor things to remember before jumping in.\n",
    "\n",
    "- original(raw) source & target datas contain a list of sentences\n",
    "  - they are represented as a string \n",
    "\n",
    "- the number of sentences are the same for English and French\n",
    " \n",
    "- by accessing each sentences, need to convert word into the corresponding index.\n",
    "  - each word should be stored in a list\n",
    "  - this makes the resuling list as a 2-D array ( row: sentence, column: word index )\n",
    "  \n",
    "- for every target sentences, special token, `<EOS>` should be inserted at the end\n",
    "  - this token suggests when to stop creating a sequence\n",
    "  \n",
    "<br/>\n",
    "<img src='./conversion.png' alt='Drawing' width='70%'>\n",
    "<br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def text_to_ids(source_text, target_text, source_vocab_to_int, target_vocab_to_int):\n",
    "    \"\"\"\n",
    "        1st, 2nd args: raw string text to be converted\n",
    "        3rd, 4th args: lookup tables for 1st and 2nd args respectively\n",
    "    \n",
    "        return: A tuple of lists (source_id_text, target_id_text) converted\n",
    "    \"\"\"\n",
    "    # empty list of converted sentences\n",
    "    source_text_id = []\n",
    "    target_text_id = []\n",
    "    \n",
    "    # make a list of sentences (extraction)\n",
    "    source_sentences = source_text.split(\"\\n\")\n",
    "    target_sentences = target_text.split(\"\\n\")\n",
    "    \n",
    "    max_source_sentence_length = max([len(sentence.split(\" \")) for sentence in source_sentences])\n",
    "    max_target_sentence_length = max([len(sentence.split(\" \")) for sentence in target_sentences])\n",
    "    \n",
    "    # iterating through each sentences (# of sentences in source&target is the same)\n",
    "    for i in range(len(source_sentences)):\n",
    "        # extract sentences one by one\n",
    "        source_sentence = source_sentences[i]\n",
    "        target_sentence = target_sentences[i]\n",
    "        \n",
    "        # make a list of tokens/words (extraction) from the chosen sentence\n",
    "        source_tokens = source_sentence.split(\" \")\n",
    "        target_tokens = target_sentence.split(\" \")\n",
    "        \n",
    "        # empty list of converted words to index in the chosen sentence\n",
    "        source_token_id = []\n",
    "        target_token_id = []\n",
    "        \n",
    "        for index, token in enumerate(source_tokens):\n",
    "            if (token != \"\"):\n",
    "                source_token_id.append(source_vocab_to_int[token])\n",
    "        \n",
    "        for index, token in enumerate(target_tokens):\n",
    "            if (token != \"\"):\n",
    "                target_token_id.append(target_vocab_to_int[token])\n",
    "                \n",
    "        # put <EOS> token at the end of the chosen target sentence\n",
    "        # this token suggests when to stop creating a sequence\n",
    "        target_token_id.append(target_vocab_to_int['<EOS>'])\n",
    "            \n",
    "        # add each converted sentences in the final list\n",
    "        source_text_id.append(source_token_id)\n",
    "        target_text_id.append(target_token_id)\n",
    "    \n",
    "    return source_text_id, target_text_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess and Save Data\n",
    "\n",
    "`create_lookup_tables`, `text_to_ids` are generalized functions. It can  be used for other languages too. In this particular project, the target languages are English and French, so those languages have to fed into `create_lookup_tables`, `text_to_ids` functions to generate pre-processed dataset for this project. Here is the steps to do it.\n",
    "\n",
    "- Load data(text) from the original file for English and French\n",
    "- Make them lower case letters\n",
    "- Create lookup tables for both English and French\n",
    "- Convert the original data into the list of sentences whose words are represented in index\n",
    "- Finally, save the preprocessed data to the external file (checkpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_and_save_data(source_path, target_path, text_to_ids):\n",
    "    # Preprocess\n",
    "    \n",
    "    # load original data (English, French)\n",
    "    source_text = load_data(source_path)\n",
    "    target_text = load_data(target_path)\n",
    "\n",
    "    # to the lower case\n",
    "    source_text = source_text.lower()\n",
    "    target_text = target_text.lower()\n",
    "\n",
    "    # create lookup tables for English and French data\n",
    "    source_vocab_to_int, source_int_to_vocab = create_lookup_tables(source_text)\n",
    "    target_vocab_to_int, target_int_to_vocab = create_lookup_tables(target_text)\n",
    "\n",
    "    # create list of sentences whose words are represented in index\n",
    "    source_text, target_text = text_to_ids(source_text, target_text, source_vocab_to_int, target_vocab_to_int)\n",
    "\n",
    "    # Save data for later use\n",
    "    pickle.dump((\n",
    "        (source_text, target_text),\n",
    "        (source_vocab_to_int, target_vocab_to_int),\n",
    "        (source_int_to_vocab, target_int_to_vocab)), open('preprocess.p', 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocess_and_save_data(source_path, target_path, text_to_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Check Point\n",
    " This project uses a small set of sentences. However, in general, NLP requires a huge amount of raw text data. It would take quite a long time to preprocess, so it is recommended to avoid whenever possible. In practice, save the preprocessed data to the external file could speed up your job and let you focus more on building a model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "def load_preprocess():\n",
    "    with open('preprocess.p', mode='rb') as in_file:\n",
    "        return pickle.load(in_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = load_preprocess()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the Version of TensorFlow and Access to GPU\n",
    "Since the Recurrent Neural Networks is kind of heavy model to train, it is recommended to train the model in GPU environment. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow Version: 1.7.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda/envs/test/lib/python3.6/site-packages/ipykernel_launcher.py:12: UserWarning: No GPU found. Please use a GPU to train your neural network.\n",
      "  if sys.path[0] == '':\n"
     ]
    }
   ],
   "source": [
    "from distutils.version import LooseVersion\n",
    "import warnings\n",
    "import tensorflow as tf\n",
    "from tensorflow.python.layers.core import Dense\n",
    "\n",
    "# Check TensorFlow Version\n",
    "assert LooseVersion(tf.__version__) >= LooseVersion('1.1'), 'Please use TensorFlow version 1.1 or newer'\n",
    "print('TensorFlow Version: {}'.format(tf.__version__))\n",
    "\n",
    "# Check for a GPU\n",
    "if not tf.test.gpu_device_name():\n",
    "    warnings.warn('No GPU found. Please use a GPU to train your neural network.')\n",
    "else:\n",
    "    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build the Neural Network\n",
    "\n",
    " In this notebook, I am going to build a special kind of model called 'sequence to sequence' (seq2seq in short). You can separate the entire model into 2 small sub-models. The first sub-model is called as __[E]__ Encoder, and the second sub-model is called as __[D]__ Decoder. __[E]__ takes a raw input text data just like any other RNN architectures do. At the end, __[E]__ outputs a neural representation. This is a very typical work, but you need to pay attention what this output really is. The output of __[E]__ is going to be the input data for __[D]__.\n",
    "\n",
    "That is why we call __[E]__ as Encoder and __[D]__ as Decoder. __[E]__ makes an output encoded in neural representational form, and we don't know what it really is. It is somewhat encrypted. __[D]__ has the ability to look inside the __[E]__'s output, and it will create a totally different output data (translated in French in this case). \n",
    "\n",
    "In order to build such a model, there are 6 steps to do overall. I noted what functions to be implemented are related to those steps.\n",
    "- __(1)__ define input parameters to the encoder model\n",
    "  - `enc_dec_model_inputs`\n",
    "- __(2)__ build encoder model\n",
    "  - `encoding_layer`\n",
    "- __(3)__ define input parameters to the decoder model\n",
    "  - `enc_dec_model_inputs`, `process_decoder_input`, `decoding_layer`\n",
    "- __(4)__ build decoder model for training\n",
    "  - `decoding_layer_train`\n",
    "- __(5)__ build decoder model for inference\n",
    "  - `decoding_layer_infer`\n",
    "- __(6)__ put (4) and (5) together \n",
    "  - `decoding_layer`\n",
    "- __(7)__ connect encoder and decoder models\n",
    "  - `seq2seq_model`\n",
    "- __(8)__ train and estimate loss and accuracy\n",
    "\n",
    "<img src=\"./training_phase.png\" style=\"width:400px;\"/>\n",
    "<div style=\"text-align:center;\">Fig 1. Neural Machine Translation / Training Phase</div>\n",
    "<br/>\n",
    "The figure above is borrowed from Thang Luong's thesis ['Neural Machine Translation'](https://github.com/lmthang/thesis/blob/master/thesis.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Input (1), (3)\n",
    "\n",
    "`enc_dec_model_inputs` function creates and returns parameters (TF placeholders) related to building model. \n",
    "- inputs placeholder will be fed with English sentence data, and its shape is `[None, None]`. The first `None` means the batch size, and the batch size is unknown since user can set it. The second `None` means the lengths of sentences. The maximum length of setence is different from batch to batch, so it cannot be set with the exact number. \n",
    "  - One option is to set the lengths of every sentences to the maximum length across all sentences in every batch. No matter which method you choose, you need to add special character, `<PAD>` in empty positions. However, with the latter option, there could be unnecessarily more `<PAD>` characters.\n",
    "  \n",
    "\n",
    "- targets placeholder is similar to inputs placeholder except that it will be fed with French sentence data.\n",
    "\n",
    "\n",
    "- target_sequence_length placeholder represents the lengths of each sentences, so the shape is `None`, a column tensor, which is the same number to the batch size. This particular value is required as an argument of TrainerHelper to build decoder model for training. We will see in (4).\n",
    "\n",
    "\n",
    "- max_target_len gets the maximum value out of lengths of all the target sentences(sequences). As you know, we have the lengths of all the sentences in target_sequence_length parameter. The way to get the maximum value from it is to use [tf.reduce_max](https://www.tensorflow.org/api_docs/python/tf/reduce_max). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def enc_dec_model_inputs():\n",
    "    inputs = tf.placeholder(tf.int32, [None, None], name='input')\n",
    "    targets = tf.placeholder(tf.int32, [None, None], name='targets') \n",
    "    \n",
    "    target_sequence_length = tf.placeholder(tf.int32, [None], name='target_sequence_length')\n",
    "    max_target_len = tf.reduce_max(target_sequence_length)    \n",
    "    \n",
    "    return inputs, targets, target_sequence_length, max_target_len"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`hyperparam_inputs` function creates and returns parameters (TF placeholders) related to hyper-parameters to the model. \n",
    "- lr_rate is learning rate\n",
    "- keep_prob is the keep probability for Dropouts\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def hyperparam_inputs():\n",
    "    lr_rate = tf.placeholder(tf.float32, name='lr_rate')\n",
    "    keep_prob = tf.placeholder(tf.float32, name='keep_prob')\n",
    "    \n",
    "    return lr_rate, keep_prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Process Decoder Input (3)\n",
    "<br/>\n",
    "<img src=\"./go_insert.png\" style=\"width:600px;\"/>\n",
    "<div style=\"text-align:center;\">Fig 2. `<GO>` insertion</div>\n",
    "<br/>\n",
    "\n",
    "On the decoder side, we need two different kinds of input for training and inference purposes repectively. While training phase, the input is provided as target label, but they still need to be embeded. On the inference phase, however, the output of each time step will be the input for the next time step. They also need to be embeded and embedding vector should be shared between two different phases.\n",
    "\n",
    "In this section, I am going to preprocess the target label data for the training phase. It is nothing special task. What all you need to do is add `<GO>` special token in front of all target data. `<GO>` token is a kind of guide token as saying like \"this is the start of the translation\". For this process, you need to know three libraries from TensorFlow.\n",
    "- [TF strided_slice](https://www.tensorflow.org/api_docs/python/tf/strided_slice)\n",
    "  - extracts a strided slice of a tensor (generalized python array indexing).\n",
    "  - can be thought as splitting into multiple tensors with the striding window size from begin to end\n",
    "  - arguments: TF Tensor, Begin, End, Strides\n",
    "- [TF fill](https://www.tensorflow.org/api_docs/python/tf/concat)\n",
    "  - creates a tensor filled with a scalar value.\n",
    "  - arguments: TF Tensor (must be int32/int64), value to fill\n",
    "- [TF concat](https://www.tensorflow.org/api_docs/python/tf/fill)\n",
    "  - concatenates tensors along one dimension.\n",
    "  - arguments: a list of TF Tensor (tf.fill and after_slice in this case), axis=1\n",
    "    \n",
    "After preprocessing the target label data, we will embed it later when implementing decoding_layer function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_decoder_input(target_data, target_vocab_to_int, batch_size):\n",
    "    \"\"\"\n",
    "    Preprocess target data for encoding\n",
    "    :return: Preprocessed target data\n",
    "    \"\"\"\n",
    "    # get '<GO>' id\n",
    "    go_id = target_vocab_to_int['<GO>']\n",
    "    \n",
    "    after_slice = tf.strided_slice(target_data, [0, 0], [batch_size, -1], [1, 1])\n",
    "    after_concat = tf.concat( [tf.fill([batch_size, 1], go_id), after_slice], 1)\n",
    "    \n",
    "    return after_concat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding (2)\n",
    "\n",
    "<img src=\"./encoding_model.png\" style=\"width:400px;\"/>\n",
    "<div style=\"text-align:center;\">Fig 3. Encoding model highlighted - Embedding/RNN layers</div>\n",
    "<br/>\n",
    "\n",
    "As depicted in Fig 3, the encoding model consists of two different parts. The first part is the embedding layer. Each word in a sentence will be represented with the number of features specified as `encoding_embedding_size`. This layer gives much richer representative power for the words [useful explanation](https://stackoverflow.com/questions/40784656/tf-contrib-layers-embed-sequence-is-for-what/44280918#44280918). The second part is the RNN layer(s). You can make use of any kind of RNN related techniques or algorithms. For example, in this project, multiple LSTM cells are stacked together after dropout technique is applied. You can use different kinds of RNN cells such as GRU.\n",
    "\n",
    "Embedding layer\n",
    "- [TF contrib.layers.embed_sequence](https://www.tensorflow.org/api_docs/python/tf/contrib/layers/embed_sequence)\n",
    "\n",
    "RNN layers\n",
    "- [TF contrib.rnn.LSTMCell](https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/LSTMCell)\n",
    "  - simply specifies how many internal units it has\n",
    "- [TF contrib.rnn.DropoutWrapper](https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/DropoutWrapper)\n",
    "  - wraps a cell with keep probability value \n",
    "- [TF contrib.rnn.MultiRNNCell](https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/MultiRNNCell)\n",
    "  - stacks multiple RNN (type) cells\n",
    "  - [how this API is used in action?](https://github.com/tensorflow/tensorflow/blob/6947f65a374ebf29e74bb71e36fd82760056d82c/tensorflow/docs_src/tutorials/recurrent.md#stacking-multiple-lstms)\n",
    "  \n",
    "Encoding model\n",
    "- [TF nn.dynamic_rnn](https://www.tensorflow.org/api_docs/python/tf/nn/dynamic_rnn)\n",
    "  - put Embedding layer and RNN layer(s) all together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob, \n",
    "                   source_vocab_size, \n",
    "                   encoding_embedding_size):\n",
    "    \"\"\"\n",
    "    :return: tuple (RNN output, RNN state)\n",
    "    \"\"\"\n",
    "    embed = tf.contrib.layers.embed_sequence(rnn_inputs, \n",
    "                                             vocab_size=source_vocab_size, \n",
    "                                             embed_dim=encoding_embedding_size)\n",
    "    \n",
    "    stacked_cells = tf.contrib.rnn.MultiRNNCell([tf.contrib.rnn.DropoutWrapper(tf.contrib.rnn.LSTMCell(rnn_size), keep_prob) for _ in range(num_layers)])\n",
    "    \n",
    "    outputs, state = tf.nn.dynamic_rnn(stacked_cells, \n",
    "                                       embed, \n",
    "                                       dtype=tf.float32)\n",
    "    return outputs, state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decoding - Training process (4)\n",
    "\n",
    "Decoding model can be thought of two separate processes, training and inference. It is not they have different architecture, but they share the same architecture and its parameters. It is that they have different strategy to feed the shared model.\n",
    "\n",
    "For this(training) and the next(inference) section, Fig 4 shows clearly shows what they are.\n",
    "\n",
    "<img src=\"./decoder_shift.png\" style=\"width:700px;\"/>\n",
    "<div style=\"text-align:center;\">Fig 4. Decoder shifted inputs</div>\n",
    "<br/>\n",
    "\n",
    "While encoder uses [TF contrib.layers.embed_sequence](https://www.tensorflow.org/api_docs/python/tf/contrib/layers/embed_sequence), it is not applicable to decoder even though it may require its input embeded. That is because the same embedding vector should be shared via training and inferece phases. [TF contrib.layers.embed_sequence](https://www.tensorflow.org/api_docs/python/tf/contrib/layers/embed_sequence) can only embed the prepared dataset before running. What needed for inference process is dynamic embedding capability. It is impossible to embed the output from the inference process before running the model because the output of the current time step will be the input of the next time step.\n",
    "\n",
    "How we can embed? We will see soon. However, for now, what you need to remember is training and inference processes share the same embedding parameters. For the training part, embeded input should be delivered. On the inference part, only embedding parameters used in the training part should be delivered.\n",
    "\n",
    "Let's see the training part first. \n",
    "- [`tf.contrib.seq2seq.TrainingHelper`](https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/TrainingHelper)\n",
    "  - TrainingHelper is where we pass the embeded input. As the name indicates, this is only a helper instance. This instance should be delivered to the BasicDecoder, which is the actual process of building the decoder model.\n",
    "- [`tf.contrib.seq2seq.BasicDecoder`](https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/BasicDecoder)\n",
    "  - BasicDecoder builds the decoder model. It means it connects the RNN layer(s) on the decoder side and the input prepared by TrainingHelper.\n",
    "- [`tf.contrib.seq2seq.dynamic_decode`](https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/dynamic_decode)\n",
    "  - dynamic_decode unrolls the decoder model so that actual prediction can be retrieved by BasicDecoder for each time steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decoding_layer_train(encoder_state, dec_cell, dec_embed_input, \n",
    "                         target_sequence_length, max_summary_length, \n",
    "                         output_layer, keep_prob):\n",
    "    \"\"\"\n",
    "    Create a training process in decoding layer \n",
    "    :return: BasicDecoderOutput containing training logits and sample_id\n",
    "    \"\"\"\n",
    "    dec_cell = tf.contrib.rnn.DropoutWrapper(dec_cell, \n",
    "                                             output_keep_prob=keep_prob)\n",
    "    \n",
    "    # for only input layer\n",
    "    helper = tf.contrib.seq2seq.TrainingHelper(dec_embed_input, \n",
    "                                               target_sequence_length)\n",
    "    \n",
    "    decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell, \n",
    "                                              helper, \n",
    "                                              encoder_state, \n",
    "                                              output_layer)\n",
    "\n",
    "    # unrolling the decoder layer\n",
    "    outputs, _, _ = tf.contrib.seq2seq.dynamic_decode(decoder, \n",
    "                                                      impute_finished=True, \n",
    "                                                      maximum_iterations=max_summary_length)\n",
    "    return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decoding - Inference process (5)\n",
    "\n",
    "- [`tf.contrib.seq2seq.GreedyEmbeddingHelper`](https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/GreedyEmbeddingHelper)\n",
    "  - GreedyEmbeddingHelper dynamically takes the output of the current step and give it to the next time step's input. In order to embed the each input result dynamically, embedding parameter(just bunch of weight values) should be provided. Along with it, GreedyEmbeddingHelper asks to give the `start_of_sequence_id` for the same amount as the batch size and `end_of_sequence_id`.\n",
    "- [`tf.contrib.seq2seq.BasicDecoder`](https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/BasicDecoder)\n",
    "  - same as described in the training process section\n",
    "- [`tf.contrib.seq2seq.dynamic_decode`](https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/dynamic_decode)\n",
    "  - same as described in the training process section"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id,\n",
    "                         end_of_sequence_id, max_target_sequence_length,\n",
    "                         vocab_size, output_layer, batch_size, keep_prob):\n",
    "    \"\"\"\n",
    "    Create a inference process in decoding layer \n",
    "    :return: BasicDecoderOutput containing inference logits and sample_id\n",
    "    \"\"\"\n",
    "    dec_cell = tf.contrib.rnn.DropoutWrapper(dec_cell, \n",
    "                                             output_keep_prob=keep_prob)\n",
    "    \n",
    "    helper = tf.contrib.seq2seq.GreedyEmbeddingHelper(dec_embeddings, \n",
    "                                                      tf.fill([batch_size], start_of_sequence_id), \n",
    "                                                      end_of_sequence_id)\n",
    "    \n",
    "    decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell, \n",
    "                                              helper, \n",
    "                                              encoder_state, \n",
    "                                              output_layer)\n",
    "    \n",
    "    outputs, _, _ = tf.contrib.seq2seq.dynamic_decode(decoder, \n",
    "                                                      impute_finished=True, \n",
    "                                                      maximum_iterations=max_target_sequence_length)\n",
    "    return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the Decoding Layer (3), (6)\n",
    "\n",
    "__Embed the target sequences__\n",
    "\n",
    "- [TF contrib.layers.embed_sequence](https://www.tensorflow.org/api_docs/python/tf/contrib/layers/embed_sequence) creates internal representation of embedding parameter, so we cannot look into or retrieve it. Rather, you need to create a embedding parameter manually by [TF Variable](https://www.tensorflow.org/api_docs/python/tf/Variable). \n",
    "\n",
    "- Manually created embedding parameter is used for training phase to convert provided target data(sequence of sentence) by [TF nn.embedding_lookup](https://www.tensorflow.org/api_docs/python/tf/nn/embedding_lookup) before the training is run. [TF nn.embedding_lookup](https://www.tensorflow.org/api_docs/python/tf/nn/embedding_lookup) with manually created embedding parameters returns the similar result to the [TF contrib.layers.embed_sequence](https://www.tensorflow.org/api_docs/python/tf/contrib/layers/embed_sequence). For the inference process, whenever the output of the current time step is calculated via decoder, it will be embeded by the shared embedding parameter and become the input for the next time step. You only need to provide the embedding parameter to the GreedyEmbeddingHelper, then it will help the process.\n",
    "\n",
    "- [How embedding_lookup works?](https://stackoverflow.com/questions/34870614/what-does-tf-nn-embedding-lookup-function-do)\n",
    "  - In short, it selects specified rows\n",
    "  \n",
    "- Note: Please be careful about setting the variable scope. As mentioned previously, parameters/variables are shared between training and inference processes. Sharing can be specified via [tf.variable_scope](https://www.tensorflow.org/api_docs/python/tf/variable_scope).\n",
    "\n",
    "__Construct the decoder RNN layer(s)__\n",
    "- As depicted in Fig 3 and Fig 4, the number of RNN layer in the decoder model has to be equal to the number of RNN layer(s) in the encoder model.\n",
    "\n",
    "__Create an output layer to map the outputs of the decoder to the elements of our vocabulary__\n",
    "- This is just a fully connected layer to get probabilities of occurance of each words at the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decoding_layer(dec_input, encoder_state,\n",
    "                   target_sequence_length, max_target_sequence_length,\n",
    "                   rnn_size,\n",
    "                   num_layers, target_vocab_to_int, target_vocab_size,\n",
    "                   batch_size, keep_prob, decoding_embedding_size):\n",
    "    \"\"\"\n",
    "    Create decoding layer\n",
    "    :return: Tuple of (Training BasicDecoderOutput, Inference BasicDecoderOutput)\n",
    "    \"\"\"\n",
    "    target_vocab_size = len(target_vocab_to_int)\n",
    "    dec_embeddings = tf.Variable(tf.random_uniform([target_vocab_size, decoding_embedding_size]))\n",
    "    dec_embed_input = tf.nn.embedding_lookup(dec_embeddings, dec_input)\n",
    "    \n",
    "    cells = tf.contrib.rnn.MultiRNNCell([tf.contrib.rnn.LSTMCell(rnn_size) for _ in range(num_layers)])\n",
    "    \n",
    "    with tf.variable_scope(\"decode\"):\n",
    "        output_layer = tf.layers.Dense(target_vocab_size)\n",
    "        train_output = decoding_layer_train(encoder_state, \n",
    "                                            cells, \n",
    "                                            dec_embed_input, \n",
    "                                            target_sequence_length, \n",
    "                                            max_target_sequence_length, \n",
    "                                            output_layer, \n",
    "                                            keep_prob)\n",
    "\n",
    "    with tf.variable_scope(\"decode\", reuse=True):\n",
    "        infer_output = decoding_layer_infer(encoder_state, \n",
    "                                            cells, \n",
    "                                            dec_embeddings, \n",
    "                                            target_vocab_to_int['<GO>'], \n",
    "                                            target_vocab_to_int['<EOS>'], \n",
    "                                            max_target_sequence_length, \n",
    "                                            target_vocab_size, \n",
    "                                            output_layer,\n",
    "                                            batch_size,\n",
    "                                            keep_prob)\n",
    "\n",
    "    return (train_output, infer_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the Seq2Seq model (7)\n",
    "\n",
    "In this section, previously defined functions, `encoding_layer`, `process_decoder_input`, and `decoding_layer` are put together to build the big picture, Sequence to Sequence model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def seq2seq_model(input_data, target_data, keep_prob, batch_size,\n",
    "                  target_sequence_length,\n",
    "                  max_target_sentence_length,\n",
    "                  source_vocab_size, target_vocab_size,\n",
    "                  enc_embedding_size, dec_embedding_size,\n",
    "                  rnn_size, num_layers, target_vocab_to_int):\n",
    "    \"\"\"\n",
    "    Build the Sequence-to-Sequence model\n",
    "    :return: Tuple of (Training BasicDecoderOutput, Inference BasicDecoderOutput)\n",
    "    \"\"\"\n",
    "    enc_outputs, enc_states = encoding_layer(input_data, \n",
    "                                             rnn_size, \n",
    "                                             num_layers, \n",
    "                                             keep_prob, \n",
    "                                             source_vocab_size, \n",
    "                                             enc_embedding_size)\n",
    "    \n",
    "    dec_input = process_decoder_input(target_data, \n",
    "                                      target_vocab_to_int, \n",
    "                                      batch_size)\n",
    "    \n",
    "    train_output, infer_output = decoding_layer(dec_input,\n",
    "                                               enc_states, \n",
    "                                               target_sequence_length, \n",
    "                                               max_target_sentence_length,\n",
    "                                               rnn_size,\n",
    "                                              num_layers,\n",
    "                                              target_vocab_to_int,\n",
    "                                              target_vocab_size,\n",
    "                                              batch_size,\n",
    "                                              keep_prob,\n",
    "                                              dec_embedding_size)\n",
    "    \n",
    "    return train_output, infer_output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network Training\n",
    "### Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_step = 300\n",
    "\n",
    "epochs = 13\n",
    "batch_size = 128\n",
    "\n",
    "rnn_size = 128\n",
    "num_layers = 3\n",
    "\n",
    "encoding_embedding_size = 200\n",
    "decoding_embedding_size = 200\n",
    "\n",
    "learning_rate = 0.001\n",
    "keep_probability = 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the Graph\n",
    "`seq2seq_model` function creates the model. It defines how the feedforward and backpropagation should flow. The last step for this model to be trainable is deciding and applying what optimization algorithms to use. In this section, [TF contrib.seq2seq.sequence_loss](https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/sequence_loss) is used to calculate the loss, then [TF train.AdamOptimizer](https://www.tensorflow.org/api_docs/python/tf/train/AdamOptimizer) is applied to calculate the gradient descent on the loss. Let's go over eatch steps in the code cell below.\n",
    "\n",
    "__load data from the checkpoint__\n",
    "- (source_int_text, target_int_text) are the input data, and (source_vocab_to_int, target_vocab_to_int) is the dictionary to lookup the index number of each words.\n",
    "- max_target_sentence_length is the length of the longest sentence from the source input data. This will be used for GreedyEmbeddingHelper when building inference process in the decoder mode.\n",
    "\n",
    "__create inputs__\n",
    "- inputs (input_data, targets, target_sequence_length, max_target_sequence_length) from enc_dec_model_inputs function\n",
    "- inputs (lr, keep_prob) from hyperparam_inputs function\n",
    "\n",
    "__build seq2seq model__\n",
    "- build the model by seq2seq_model function. It will return train_logits(logits to calculate the loss) and inference_logits(logits from prediction).\n",
    "\n",
    "__cost function__\n",
    "- [TF contrib.seq2seq.sequence_loss](https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/sequence_loss) is used. This loss function is just a weighted softmax cross entropy loss function, but it is particularly designed to be applied in time series model (RNN). Weights should be explicitly provided as an argument, and it can be created by [TF sequence_mask](https://www.tensorflow.org/api_docs/python/tf/sequence_mask). In this project, [TF sequence_mask](https://www.tensorflow.org/api_docs/python/tf/sequence_mask) creates \\[batch_size, max_target_sequence_length\\] size of variable, then maks only the first target_sequence_length number of elements to 1. It means <PAD\\> parts will have less weight than others.\n",
    "\n",
    "__Optimizer__\n",
    "- [TF train.AdamOptimizer](https://www.tensorflow.org/api_docs/python/tf/train/AdamOptimizer) is used, and this is where the learning rate should be specified. You can choose other algorithms as well, this is just a choice.\n",
    "\n",
    "__Gradient Clipping__\n",
    "- Since recurrent neural networks is notorious about vanishing/exploding gradient, gradient clipping technique is believed to improve the issues. \n",
    "- The concept is really easy. You decide thresholds to keep the gradient to be in a certain boundary. In this project, the range of the threshold is between -1 and 1.\n",
    "- Now, you need to apply this conceptual knowledge to the TensorFlow code. Luckily, there is the official guide for this [TF Gradient Clipping How?](https://www.tensorflow.org/api_guides/python/train#Gradient_Clipping). In breif, you get the gradient values from the optimizer manually by calling [compute_gradients](https://www.tensorflow.org/api_docs/python/tf/train/Optimizer#compute_gradients), then manipulate the gradient values with [clip_by_value](https://www.tensorflow.org/api_docs/python/tf/clip_by_value). Lastly, you need to put back the modified gradients into the optimizer by calling [apply_gradients](https://www.tensorflow.org/api_docs/python/tf/train/Optimizer#apply_gradients)\n",
    "\n",
    "<img src=\"./gradient_clipping.png\" style=\"width:700px;\"/>\n",
    "<div style=\"text-align:center;\">Fig 4. Gradient Clipping</div>\n",
    "<br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /anaconda/envs/test/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use the retry module or similar alternatives.\n"
     ]
    }
   ],
   "source": [
    "save_path = 'checkpoints/dev'\n",
    "(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = load_preprocess()\n",
    "max_target_sentence_length = max([len(sentence) for sentence in source_int_text])\n",
    "\n",
    "train_graph = tf.Graph()\n",
    "with train_graph.as_default():\n",
    "    input_data, targets, target_sequence_length, max_target_sequence_length = enc_dec_model_inputs()\n",
    "    lr, keep_prob = hyperparam_inputs()\n",
    "    \n",
    "    train_logits, inference_logits = seq2seq_model(tf.reverse(input_data, [-1]),\n",
    "                                                   targets,\n",
    "                                                   keep_prob,\n",
    "                                                   batch_size,\n",
    "                                                   target_sequence_length,\n",
    "                                                   max_target_sequence_length,\n",
    "                                                   len(source_vocab_to_int),\n",
    "                                                   len(target_vocab_to_int),\n",
    "                                                   encoding_embedding_size,\n",
    "                                                   decoding_embedding_size,\n",
    "                                                   rnn_size,\n",
    "                                                   num_layers,\n",
    "                                                   target_vocab_to_int)\n",
    "    \n",
    "    training_logits = tf.identity(train_logits.rnn_output, name='logits')\n",
    "    inference_logits = tf.identity(inference_logits.sample_id, name='predictions')\n",
    "\n",
    "    # https://www.tensorflow.org/api_docs/python/tf/sequence_mask\n",
    "    # - Returns a mask tensor representing the first N positions of each cell.\n",
    "    masks = tf.sequence_mask(target_sequence_length, max_target_sequence_length, dtype=tf.float32, name='masks')\n",
    "\n",
    "    with tf.name_scope(\"optimization\"):\n",
    "        # Loss function - weighted softmax cross entropy\n",
    "        cost = tf.contrib.seq2seq.sequence_loss(\n",
    "            training_logits,\n",
    "            targets,\n",
    "            masks)\n",
    "\n",
    "        # Optimizer\n",
    "        optimizer = tf.train.AdamOptimizer(lr)\n",
    "\n",
    "        # Gradient Clipping\n",
    "        gradients = optimizer.compute_gradients(cost)\n",
    "        capped_gradients = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gradients if grad is not None]\n",
    "        train_op = optimizer.apply_gradients(capped_gradients)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Batches and Pad the source and target sequences\n",
    "<br/>\n",
    "<img src=\"./pad_insert.png\" style=\"width:300px;\"/>\n",
    "<div style=\"text-align:center;\">Fig 5. Padding character in empty space of sentences shorter than the longest one in a batch</div>\n",
    "<br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pad_sentence_batch(sentence_batch, pad_int):\n",
    "    \"\"\"Pad sentences with <PAD> so that each sentence of a batch has the same length\"\"\"\n",
    "    max_sentence = max([len(sentence) for sentence in sentence_batch])\n",
    "    return [sentence + [pad_int] * (max_sentence - len(sentence)) for sentence in sentence_batch]\n",
    "\n",
    "\n",
    "def get_batches(sources, targets, batch_size, source_pad_int, target_pad_int):\n",
    "    \"\"\"Batch targets, sources, and the lengths of their sentences together\"\"\"\n",
    "    for batch_i in range(0, len(sources)//batch_size):\n",
    "        start_i = batch_i * batch_size\n",
    "\n",
    "        # Slice the right amount for the batch\n",
    "        sources_batch = sources[start_i:start_i + batch_size]\n",
    "        targets_batch = targets[start_i:start_i + batch_size]\n",
    "\n",
    "        # Pad\n",
    "        pad_sources_batch = np.array(pad_sentence_batch(sources_batch, source_pad_int))\n",
    "        pad_targets_batch = np.array(pad_sentence_batch(targets_batch, target_pad_int))\n",
    "\n",
    "        # Need the lengths for the _lengths parameters\n",
    "        pad_targets_lengths = []\n",
    "        for target in pad_targets_batch:\n",
    "            pad_targets_lengths.append(len(target))\n",
    "\n",
    "        pad_source_lengths = []\n",
    "        for source in pad_sources_batch:\n",
    "            pad_source_lengths.append(len(source))\n",
    "\n",
    "        yield pad_sources_batch, pad_targets_batch, pad_source_lengths, pad_targets_lengths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train\n",
    "\n",
    "`get_accuracy`\n",
    "- compare the lengths of target(label) and logits(prediction)\n",
    "- add(pad) 0s at the end of the ones having the shorter length\n",
    "  - `[(0,0),(0,max_seq - target.shape[1])]` indicates the 2D array. The first (0,0) means no padding for the first dimension. The second (0, ...) means there is no pads in front of the second dimension but pads at the end. And pad as many times as ... .\n",
    "- above process is to makes two entities to have the same shape (length)\n",
    "- finally, returns the average of where the target and logits have the same value (1)\n",
    "\n",
    "[numpy pad function](https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.pad.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch   0 Batch  300/1077 - Train Accuracy: 0.4293, Validation Accuracy: 0.5064, Loss: 1.9365\n",
      "Epoch   0 Batch  600/1077 - Train Accuracy: 0.4974, Validation Accuracy: 0.5241, Loss: 1.1117\n",
      "Epoch   0 Batch  900/1077 - Train Accuracy: 0.5410, Validation Accuracy: 0.5614, Loss: 0.9078\n",
      "Epoch   1 Batch  300/1077 - Train Accuracy: 0.5958, Validation Accuracy: 0.6044, Loss: 0.6807\n",
      "Epoch   1 Batch  600/1077 - Train Accuracy: 0.6786, Validation Accuracy: 0.6346, Loss: 0.5282\n",
      "Epoch   1 Batch  900/1077 - Train Accuracy: 0.7066, Validation Accuracy: 0.6779, Loss: 0.5059\n",
      "Epoch   2 Batch  300/1077 - Train Accuracy: 0.7512, Validation Accuracy: 0.7383, Loss: 0.4164\n",
      "Epoch   2 Batch  600/1077 - Train Accuracy: 0.7894, Validation Accuracy: 0.7784, Loss: 0.3555\n",
      "Epoch   2 Batch  900/1077 - Train Accuracy: 0.8164, Validation Accuracy: 0.7869, Loss: 0.3256\n",
      "Epoch   3 Batch  300/1077 - Train Accuracy: 0.8651, Validation Accuracy: 0.8224, Loss: 0.2444\n",
      "Epoch   3 Batch  600/1077 - Train Accuracy: 0.8519, Validation Accuracy: 0.8171, Loss: 0.2170\n",
      "Epoch   3 Batch  900/1077 - Train Accuracy: 0.8676, Validation Accuracy: 0.8509, Loss: 0.2241\n",
      "Epoch   4 Batch  300/1077 - Train Accuracy: 0.9219, Validation Accuracy: 0.8750, Loss: 0.1508\n",
      "Epoch   4 Batch  600/1077 - Train Accuracy: 0.9022, Validation Accuracy: 0.8984, Loss: 0.1493\n",
      "Epoch   4 Batch  900/1077 - Train Accuracy: 0.9105, Validation Accuracy: 0.8803, Loss: 0.1448\n",
      "Epoch   5 Batch  300/1077 - Train Accuracy: 0.9359, Validation Accuracy: 0.9091, Loss: 0.1008\n",
      "Epoch   5 Batch  600/1077 - Train Accuracy: 0.9364, Validation Accuracy: 0.9130, Loss: 0.1137\n",
      "Epoch   5 Batch  900/1077 - Train Accuracy: 0.9449, Validation Accuracy: 0.9205, Loss: 0.1105\n",
      "Epoch   6 Batch  300/1077 - Train Accuracy: 0.9548, Validation Accuracy: 0.9375, Loss: 0.0767\n",
      "Epoch   6 Batch  600/1077 - Train Accuracy: 0.9472, Validation Accuracy: 0.9347, Loss: 0.0794\n",
      "Epoch   6 Batch  900/1077 - Train Accuracy: 0.9441, Validation Accuracy: 0.9446, Loss: 0.0855\n",
      "Epoch   7 Batch  300/1077 - Train Accuracy: 0.9692, Validation Accuracy: 0.9364, Loss: 0.0530\n",
      "Epoch   7 Batch  600/1077 - Train Accuracy: 0.9483, Validation Accuracy: 0.9322, Loss: 0.0719\n",
      "Epoch   7 Batch  900/1077 - Train Accuracy: 0.9535, Validation Accuracy: 0.9513, Loss: 0.0616\n",
      "Epoch   8 Batch  300/1077 - Train Accuracy: 0.9667, Validation Accuracy: 0.9542, Loss: 0.0461\n",
      "Epoch   8 Batch  600/1077 - Train Accuracy: 0.9528, Validation Accuracy: 0.9506, Loss: 0.0595\n",
      "Epoch   8 Batch  900/1077 - Train Accuracy: 0.9656, Validation Accuracy: 0.9489, Loss: 0.0506\n",
      "Epoch   9 Batch  300/1077 - Train Accuracy: 0.9720, Validation Accuracy: 0.9563, Loss: 0.0383\n",
      "Epoch   9 Batch  600/1077 - Train Accuracy: 0.9624, Validation Accuracy: 0.9645, Loss: 0.0488\n",
      "Epoch   9 Batch  900/1077 - Train Accuracy: 0.9727, Validation Accuracy: 0.9638, Loss: 0.0511\n",
      "Epoch  10 Batch  300/1077 - Train Accuracy: 0.9753, Validation Accuracy: 0.9631, Loss: 0.0343\n",
      "Epoch  10 Batch  600/1077 - Train Accuracy: 0.9494, Validation Accuracy: 0.9748, Loss: 0.0482\n",
      "Epoch  10 Batch  900/1077 - Train Accuracy: 0.9746, Validation Accuracy: 0.9688, Loss: 0.0416\n",
      "Epoch  11 Batch  300/1077 - Train Accuracy: 0.9778, Validation Accuracy: 0.9737, Loss: 0.0258\n",
      "Epoch  11 Batch  600/1077 - Train Accuracy: 0.9513, Validation Accuracy: 0.9680, Loss: 0.0353\n",
      "Epoch  11 Batch  900/1077 - Train Accuracy: 0.9602, Validation Accuracy: 0.9712, Loss: 0.0419\n",
      "Epoch  12 Batch  300/1077 - Train Accuracy: 0.9774, Validation Accuracy: 0.9656, Loss: 0.0252\n",
      "Epoch  12 Batch  600/1077 - Train Accuracy: 0.9710, Validation Accuracy: 0.9698, Loss: 0.0413\n",
      "Epoch  12 Batch  900/1077 - Train Accuracy: 0.9770, Validation Accuracy: 0.9684, Loss: 0.0352\n",
      "Model Trained and Saved\n"
     ]
    }
   ],
   "source": [
    "def get_accuracy(target, logits):\n",
    "    \"\"\"\n",
    "    Calculate accuracy\n",
    "    \"\"\"\n",
    "    max_seq = max(target.shape[1], logits.shape[1])\n",
    "    if max_seq - target.shape[1]:\n",
    "        target = np.pad(\n",
    "            target,\n",
    "            [(0,0),(0,max_seq - target.shape[1])],\n",
    "            'constant')\n",
    "    if max_seq - logits.shape[1]:\n",
    "        logits = np.pad(\n",
    "            logits,\n",
    "            [(0,0),(0,max_seq - logits.shape[1])],\n",
    "            'constant')\n",
    "\n",
    "    return np.mean(np.equal(target, logits))\n",
    "\n",
    "# Split data to training and validation sets\n",
    "train_source = source_int_text[batch_size:]\n",
    "train_target = target_int_text[batch_size:]\n",
    "valid_source = source_int_text[:batch_size]\n",
    "valid_target = target_int_text[:batch_size]\n",
    "(valid_sources_batch, valid_targets_batch, valid_sources_lengths, valid_targets_lengths ) = next(get_batches(valid_source,\n",
    "                                                                                                             valid_target,\n",
    "                                                                                                             batch_size,\n",
    "                                                                                                             source_vocab_to_int['<PAD>'],\n",
    "                                                                                                             target_vocab_to_int['<PAD>']))                                                                                                  \n",
    "with tf.Session(graph=train_graph) as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "\n",
    "    for epoch_i in range(epochs):\n",
    "        for batch_i, (source_batch, target_batch, sources_lengths, targets_lengths) in enumerate(\n",
    "                get_batches(train_source, train_target, batch_size,\n",
    "                            source_vocab_to_int['<PAD>'],\n",
    "                            target_vocab_to_int['<PAD>'])):\n",
    "\n",
    "            _, loss = sess.run(\n",
    "                [train_op, cost],\n",
    "                {input_data: source_batch,\n",
    "                 targets: target_batch,\n",
    "                 lr: learning_rate,\n",
    "                 target_sequence_length: targets_lengths,\n",
    "                 keep_prob: keep_probability})\n",
    "\n",
    "\n",
    "            if batch_i % display_step == 0 and batch_i > 0:\n",
    "                batch_train_logits = sess.run(\n",
    "                    inference_logits,\n",
    "                    {input_data: source_batch,\n",
    "                     target_sequence_length: targets_lengths,\n",
    "                     keep_prob: 1.0})\n",
    "\n",
    "                batch_valid_logits = sess.run(\n",
    "                    inference_logits,\n",
    "                    {input_data: valid_sources_batch,\n",
    "                     target_sequence_length: valid_targets_lengths,\n",
    "                     keep_prob: 1.0})\n",
    "\n",
    "                train_acc = get_accuracy(target_batch, batch_train_logits)\n",
    "                valid_acc = get_accuracy(valid_targets_batch, batch_valid_logits)\n",
    "\n",
    "                print('Epoch {:>3} Batch {:>4}/{} - Train Accuracy: {:>6.4f}, Validation Accuracy: {:>6.4f}, Loss: {:>6.4f}'\n",
    "                      .format(epoch_i, batch_i, len(source_int_text) // batch_size, train_acc, valid_acc, loss))\n",
    "\n",
    "    # Save Model\n",
    "    saver = tf.train.Saver()\n",
    "    saver.save(sess, save_path)\n",
    "    print('Model Trained and Saved')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Parameters\n",
    "Save the `batch_size` and `save_path` parameters for inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_params(params):\n",
    "    with open('params.p', 'wb') as out_file:\n",
    "        pickle.dump(params, out_file)\n",
    "\n",
    "\n",
    "def load_params():\n",
    "    with open('params.p', mode='rb') as in_file:\n",
    "        return pickle.load(in_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save parameters for checkpoint\n",
    "save_params(save_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Checkpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import problem_unittests as tests\n",
    "\n",
    "_, (source_vocab_to_int, target_vocab_to_int), (source_int_to_vocab, target_int_to_vocab) = load_preprocess()\n",
    "load_path = load_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Translate\n",
    "This will translate `translate_sentence` from English to French."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from checkpoints/dev\n",
      "Input\n",
      "  Word Ids:      [158, 189, 82, 152, 206, 176, 101]\n",
      "  English Words: ['he', 'saw', 'a', 'old', 'yellow', 'truck', '.']\n",
      "\n",
      "Prediction\n",
      "  Word Ids:      [18, 212, 182, 26, 220, 48, 317, 126, 1]\n",
      "  French Words: il a vu un nouveau camion jaune . <EOS>\n"
     ]
    }
   ],
   "source": [
    "def sentence_to_seq(sentence, vocab_to_int):\n",
    "    results = []\n",
    "    for word in sentence.split(\" \"):\n",
    "        if word in vocab_to_int:\n",
    "            results.append(vocab_to_int[word])\n",
    "        else:\n",
    "            results.append(vocab_to_int['<UNK>'])\n",
    "            \n",
    "    return results\n",
    "\n",
    "translate_sentence = 'he saw a old yellow truck .'\n",
    "\n",
    "translate_sentence = sentence_to_seq(translate_sentence, source_vocab_to_int)\n",
    "\n",
    "loaded_graph = tf.Graph()\n",
    "with tf.Session(graph=loaded_graph) as sess:\n",
    "    # Load saved model\n",
    "    loader = tf.train.import_meta_graph(load_path + '.meta')\n",
    "    loader.restore(sess, load_path)\n",
    "\n",
    "    input_data = loaded_graph.get_tensor_by_name('input:0')\n",
    "    logits = loaded_graph.get_tensor_by_name('predictions:0')\n",
    "    target_sequence_length = loaded_graph.get_tensor_by_name('target_sequence_length:0')\n",
    "    keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')\n",
    "\n",
    "    translate_logits = sess.run(logits, {input_data: [translate_sentence]*batch_size,\n",
    "                                         target_sequence_length: [len(translate_sentence)*2]*batch_size,\n",
    "                                         keep_prob: 1.0})[0]\n",
    "\n",
    "print('Input')\n",
    "print('  Word Ids:      {}'.format([i for i in translate_sentence]))\n",
    "print('  English Words: {}'.format([source_int_to_vocab[i] for i in translate_sentence]))\n",
    "\n",
    "print('\\nPrediction')\n",
    "print('  Word Ids:      {}'.format([i for i in translate_logits]))\n",
    "print('  French Words: {}'.format(\" \".join([target_int_to_vocab[i] for i in translate_logits])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "test",
   "language": "python",
   "name": "test"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}